As time went on, I started paying attention to the “Most Popular” feeds on the homepage, and even managed to get on there myself a few times. It was a good way to find interesting techy links. Then, when the friends network facility was introduced I started following the links that my friends were saving as well. All was well.

However, I never really saved as many of the links that I found interesting as I could have done. There were plenty of occasions when I remembered telling people about particular pages because they were useful, but not being able to find them again myself. Oh, how I wished I’d added them to del.icio.us.

The problem is that I’m lazy. So, I was really happy when Packratius peeked its nose up to the cage bars. The idea behind Packratius is that it’ll keep an eye on your twitter activity and automagically add any links you tweet to del.icio.us (by default with a via:packrati.us tag). It’s deceptively simple, and something that deeply interested me – I immediately signed up.

A good idea gone overwhelming

As it turns out, so did quite a few of my friends. Overnight, my del.icio.us network feed went from being a carefully curated list of links to a mishmash of Carrots Glazed with Cum, gowalla updates and twitpics.

Now, to its credit, Packratius has options to allow you to not auto-delish links tweeted by any twitter clients that you define. This allows you to not auto tweet twitpics and the like. Of course, telling it to stop tweeting those things from your own twitter account is great, but it doesn’t stop you from seeing those updates from other people (/me looks pointedly at Simon Jobling). So, I needed to find a way to split these auto-delished URLs off from the main list.

As it turns out, whilst del.icio.us’ searching abilities are pretty powerful, you can’t do a simply negative search. So whilst you could say “Show me everything in my network feed tagged with ‘kittens’ but not ‘via:packrati.us’” you can’t say “Show me everything in my network feed not tagged with ‘via:packrati.us’”. Thankfully I came up with a solution to this – I created a Yahoo! Pipe called “del.icio.us network feed minus packratius“. (Catchy, huh?)

It’s a fairly simple pipe, with the following aims:

• Provide a feed of your del.icio.us network with items tagged with via:packrati.us removed.
• Easily customisable to show different peoples’ network feeds.

If you’ve been having similar problems with Packratius overwhelming your del.icio.us network feed, then give it a go. I know James Broad has been finding Packratius overwhelming – it was his tweet that inspired me to finally write this post up, in the hope of benefitting a few more people.

How does the pipe work?

Like I said, the pipe’s fairly simple – most of the complexity comes from generating the URL to load a given user’s del.icio.us network feed.

Once we’ve obtained the network feed, the only thing we need to do is discard all the feed items that contain via:packrati.us as a tag. Handily, del.icio.us provides us this information in its feeds:

<item>
  …
  <link>

http://wwp.greenwichmeantime.com/time-zone/africa/south-africa/

  </link>
  <description>
    South Africa is in Southern Africa, 
    at the southern tip of the continent of Africa.
    http:// bit.ly/am78yK
    #statingthebleedingobvious
  </description>
  <category domain="http://delicious.com/simey_j/">
    statingthebleedingobvious
  </category>
  <category domain="http://delicious.com/simey_j/">
    via:packrati.us
  </category>
</item>

In the feed item above (cut down for space), we can see that as well as via:packrati.us, a statingthebleedingobvious tag has been automatically generated from the hashtag in the original tweet. This is a great feature of Packratius, but it did mean that I ended up having to do more than a very simple filter to filter out the via:packrati.us tagged feed items.

Instead what I did was create a Filter block in pipes with rules that looked at the first 6 tags on each feed item (the idea being that there isn’t enough space in a tweet for more than 6 or so tags). If any of them matched via:packrati.us then they’re discarded from the feed.

The filter I created looked something a little like this:

Filter

Block items that match any of the following:

• item.category.0.content is via:packrati.us
• item.category.1.content is via:packrati.us
• item.category.2.content is via:packrati.us
• item.category.3.content is via:packrati.us
• item.category.4.content is via:packrati.us
• item.category.5.content is via:packrati.us
• item.category.content is via:packrati.us

Those first six filter rules cover the instance of more than one tag being applied to the item, and the final filter covers the intances where only a single tag exists. Simple.

And that’s all there is to it. Of course, writing this Pipe wouldn’t be necessary if del.icio.us allowed us to do a simple negative search. Still, it’s good that it’s nice and easy to generate this new feed using Pipes.

Hopefully it’ll be useful to a couple of you.

Just after I left Yahoo!, I was asked by YDN if I’d mind still presenting the talk I was planning to do about Yahoo! Pipes. Being the wonderful sort of chap I am, I happily agreed. So, after a couple of months away from the company, on Tuesday I rolled up to give my presentation. I’ve got to say, it felt a bit weird giving a talk for a company I no longer worked for. But, it seemed to go okay. People asked interesting questions, and they seemed to enjoy me talking about how to make Pipes, and sharing a few of the ones that I’d previously made with them.

The idea behind the talk was that whilst Pipes is a wonderful tool, it does have a fairly steep learning curve, and many people are dissuaded from using it before they’ve managed to do anything because of that. When I started using Pipes two years ago, I wished I had someone who already used it to give me a few pointers, so by giving this talk I hope I helped a few people out in the way that I wish I had been two years previously.

All the pipes used within the talk are linked within the slide deck, and if you want to look at any of my other pipes, you can find them at pipes.yahoo.com/neilcrosby.

This is unfortunate, since twitter does provide an in_reply_to_status_id parameter for tweets, which is set when one tweet is specifically written in reply to another. If we were able to query this parameter directly then the Twitter Conversation Yahoo! Pipe that I’ve written would be unnecessary and I wouldn’t have had to spend a couple of hours writing it. But I did have to write it, and that’s okay because it gave me a much better understanding of how to use Pipes.

In order to track twitter conversations (between non-private users), I did the following:

1. Find the status_id of the tweet that started the conversation.

   This can found at the end of the tweet’s URL.

2. Make a request to the twitter API to find out who wrote that tweet.

   This isn’t strictly necessary – the user could provide this too. However, by working this out for them the user only has to provide one piece of information.

3. Make a request to twitter search for any tweets written since the original tweet that contain “@username”, where ‘username’ is the username of the original tweeter.

4. For each search result make a request to the twitter API to find out if that tweet has its in_reply_to_status_id parameter parameter set.

   Yup, that’s an API request for each of conceivably a lot of tweets. If the twitter search API returned this data as part of its result set that I could remove this call and be a much happier man.

5. Discard any search results which do not have in_reply_to_status_id set to the status_id of the original tweet.

6. At this point, I originally wanted to recurse the pipe and run through this process again for all the tweets I’d just found via the search so as to get a richer conversation experience.

   Unfortunately, due to the massive number of HTTP requests required, Pipes understandably started timing out.

7. Finally, add the original tweet to the list of tweets if required and order by time, starting with the original tweet.

There’s a fair bit going on there, and things get a bit complicated in the Pipe due to the way various properties had to be manipulated. It all ends up working pretty well though.

This Pipe becomes more useful once you add a quick little bookmarklet to it. Drag my Twitter Conversation bookmarklet to your browser’s bookmarks folder, and then click on it any time you’re on a tweet page that you think will start an interesting conversation. You’ll immediately be taken to a Pipes page which contains the conversation. Then, you’ll be happy.

I’d been wanting to get round to writing this post for a while now, but was finally pushed into getting it written by a statement Paul Boag made on twitter:

Paul Boag: wishes that when somebody asks a question on Twitter you could easily see if anybody else has answered it.

This was quickly replied to by Dominic Baggott, who had seen my Pipe when I mentioned it a couple of weeks previously on twitter:

@boagworld You could check out @NeilCrosby’s conversation pipe built for pretty much that purpose. http://bit.ly/bCkvL

Clearly there’s a want out there for people to be able to see these conversation threads.

The fact that I’ve been able to fairly easily hack together the proof of concept in Pipes shows that it wouldn’t be the most difficult thing in the world for twitter to implement themselves. Even if twitter just “flipped a switch” and turned on output of in_reply_to_status_id in search API results then this hack would become a whole lot more palatable due to the massive reduction in HTTP requests this would mean. It might even mean that Pipes would be able to recurse into the conversations, which would be fantastic.

UPDATE: Darren Foreman has just shown me pbtweet, which also surfaces conversations in twitter, as well as a few other things, and is also apparently based on Pipes.