My Own Private Echelon

[pphaneuf]

I wrote about communities recently. I was saying how they helped me become better, a bit like some people have said that using Google (or the Internet in general) was like having their IQ increased by 10 points.

So, logically, it follows that I want more of it! But, how, where?

It's pretty clear that I'm not going to actually become smarter by reading stuff on the Internet. What I can get is good intelligence.

Turns out that there's a lot of interesting things that aren't directly written by people, but that are by-products of what they do, such as the music they listen to, the web pages they read, photos they take or look at, source code that they commit, how they link to one another in their journals, etc...

And what's more, a lot of this information is available in RSS or Atom feeds. For example, a number of feeds available about me:

• My bookmarks
  
• My photos
  
• Comments on my photos
  
• Further comments on photos I commented on
  
• Musical recommendations people made to me
  
• Recent tracks I've listened to
  
• Commits I've made to open source projects
  
• Events on my calendar

Some of this is interesting to me (like musical recommendations or the further comments on photos I commented on), some of it is interesting to other people (like my bookmarks or my photos).

That's not to mention also feeds for things like bug trackers, per-project commits, wiki recent changes, and what-not, which are very useful to me. As a pusher at NITI, I sure would have liked to have feeds for PDR runs or autobuilder results, say. Maybe just the failed ones, but still.

All these feeds can yield tremendous amounts of information, and since you're picking them, most likely useful information, there need to be some way to sift through it as efficiently as possible. Thankfully, it turns out the intelligence community (like the NSA and the CIA) have been using RSS and Atom to feed their analysts data in a manageable way. And I have a feeling that when it comes to dealing with a metric fuckload of transiently useful bits of information, these people could have a useful clue or two...

Comments

[sfllaw.livejournal.com]

The amount of information I process every day that comes from web feeds is far too high. So I think there's definitely an upper bound to how much intelligence you can usefully process.

[pphaneuf.livejournal.com]

Yeah, but the amount of information I can process without a good feed reader, just polling various web sites, is extremely limited in comparison.

There's two options with that newfound power: get the same amount of intelligence in a tiny fraction of the time, or ramp it up until it takes the same amount of time.

And there's a number of extremely low-traffic feeds that would just be ridiculous to try polling by hand, like my last.fm recommendations or the comments on my photos. I'm mostly thinking about stuff that I poll (or used to poll), and getting rid of them as much as possible.

But I think there's some sort of best-practices yet to be found, both in producing and consuming feeds...

[sfllaw.livejournal.com]

Oh, I agree that it's good at aggregating the information sources you do get. But eventually, there comes to be a point where the signal is drowned out by noise. I find low-traffic feeds almost disappear in the torrent that is my Friends list.

[pphaneuf.livejournal.com]

Hence not using my friends list. I have those feeds in a group, so I can easily skip over them, mark them all as read or other such brutalities when I want to. The most egregiously useless feeds, I simply drop (but not from the "friends" list on LJ, as I use those as "people on LJ that I have met in person", made possible by my near-zero usage of the friends-lock feature).

[sfllaw.livejournal.com]

I think that a better user-interface doesn't really amortize the amount of raw information that flows in. If you're willing to drop packets on the floor, that's fine, but you're going to get huge gaps after a while.

[pphaneuf.livejournal.com]

Well, not being subscribed at all is a bit like dropping them on the floor continually. And really, I do not have nearly as many feeds as I have LJ friends (and, conversely, there are some feeds I read but which I have not friended, like jwz).

The way I have things set up, I have simply set priorities on what is important for me to read, and when I have a lot of free time, I get to read everything, but when I'm busy, I cut off at the bottom (instead of everywhere).

[caspianxi.livejournal.com]

Funny you should mention this. Our Label has been looking at ways to capitalize on some of this data (as you said.. It's useful stuff). I've also been playing with the idea of aggregation of it, and transferral into some other form (boiling it down). It would be interesting to see how close to the real person the picture of the aggregated data would give us.

[pphaneuf.livejournal.com]

You know of Planet, right? One of the thing it does is create an aggregated feed.

There's also the FOAF (which is a file format) aspect of it, which is a bit like a meta-feed (what feeds do I read).

As for your music label, I'd say that one of my big wish, as a music fan, would be for some sort of feed to tell me when my favourite bands are coming to town. But it's kind of tricky, because I want an intersection of my favourite bands and the venues in my town.

I've seen some bands (like Nick Cave and the Bad Seeds, for example) have feeds for their shows, but it's kind of damning when you see all those shows far away. Also, most bands don't have feeds like that, so I think I'd get more mileage out of a feed for local shows, but then, Garou and Roch Voisine can just go fuck themselves. But the trick is, there's a big grey area of bands that do not have feeds at all (so I could learn of it through a local feed), and places that aren't exactly my town but that I'd be willing to go to for some bands (I'd do a Montreal-Toronto type of ride for Einstürzende Neubauten, no question asked).

Tricky problem, that's for sure...

But your label could already go and do the first step by publishing feeds for each of your artists. Do not make just one global feed, that's just rude and pushing some artists on to the fans of the other ones. Atom has a pretty cool architecture, in that it was designed for easily bundling together various entries (it has the concept of an "entry" as a self-sufficient document, unlike all the others), so you could have one show per entry, one entry per file, scattered over a directory for each band, and it would be very easy (through the incredible technology of "cat"!) to create aggregated feeds of various things with very decent performance (no database other than the filesystem). Or you could just leave the aggregation to the people's feed readers. ;-)

In any case, I'd be curious to hear more about your ideas, it's something I'm highly interested in.

[pphaneuf.livejournal.com]

Here's a link to the FOAF project, which pretty squarely addresses the subject of this post.

Also, as I visited last.fm (which is pretty awesome in general, I sure like it a whole fuckload more than MySpace!) a few minutes ago, I saw they were announcing some new features to their web site, including a location-based calendar. I'm not entirely sure of what they're doing, but there's some good potential to do good, there, with the data they have on me, they could suggest shows that I might like, even though I do not know the artists, etc...

I also have a feeling there might be some potential for adding geocoding to Atom entries, through an extension, and thus be able to filter by location. That'd be pretty cool.

http://www.georss.org/
http://www.worldkit.org/doc/rss.php

On the other hand:

http://geoweb.blog.com/387412/