This is unfortunate, since twitter does provide an in_reply_to_status_id parameter for tweets, which is set when one tweet is specifically written in reply to another. If we were able to query this parameter directly then the Twitter Conversation Yahoo! Pipe that I’ve written would be unnecessary and I wouldn’t have had to spend a couple of hours writing it. But I did have to write it, and that’s okay because it gave me a much better understanding of how to use Pipes.

In order to track twitter conversations (between non-private users), I did the following:

1. Find the status_id of the tweet that started the conversation.

   This can found at the end of the tweet’s URL.

2. Make a request to the twitter API to find out who wrote that tweet.

   This isn’t strictly necessary – the user could provide this too. However, by working this out for them the user only has to provide one piece of information.

3. Make a request to twitter search for any tweets written since the original tweet that contain “@username”, where ‘username’ is the username of the original tweeter.

4. For each search result make a request to the twitter API to find out if that tweet has its in_reply_to_status_id parameter parameter set.

   Yup, that’s an API request for each of conceivably a lot of tweets. If the twitter search API returned this data as part of its result set that I could remove this call and be a much happier man.

5. Discard any search results which do not have in_reply_to_status_id set to the status_id of the original tweet.

6. At this point, I originally wanted to recurse the pipe and run through this process again for all the tweets I’d just found via the search so as to get a richer conversation experience.

   Unfortunately, due to the massive number of HTTP requests required, Pipes understandably started timing out.

7. Finally, add the original tweet to the list of tweets if required and order by time, starting with the original tweet.

There’s a fair bit going on there, and things get a bit complicated in the Pipe due to the way various properties had to be manipulated. It all ends up working pretty well though.

This Pipe becomes more useful once you add a quick little bookmarklet to it. Drag my Twitter Conversation bookmarklet to your browser’s bookmarks folder, and then click on it any time you’re on a tweet page that you think will start an interesting conversation. You’ll immediately be taken to a Pipes page which contains the conversation. Then, you’ll be happy.

I’d been wanting to get round to writing this post for a while now, but was finally pushed into getting it written by a statement Paul Boag made on twitter:

Paul Boag: wishes that when somebody asks a question on Twitter you could easily see if anybody else has answered it.

This was quickly replied to by Dominic Baggott, who had seen my Pipe when I mentioned it a couple of weeks previously on twitter:

@boagworld You could check out @NeilCrosby’s conversation pipe built for pretty much that purpose. http://bit.ly/bCkvL

Clearly there’s a want out there for people to be able to see these conversation threads.

The fact that I’ve been able to fairly easily hack together the proof of concept in Pipes shows that it wouldn’t be the most difficult thing in the world for twitter to implement themselves. Even if twitter just “flipped a switch” and turned on output of in_reply_to_status_id in search API results then this hack would become a whole lot more palatable due to the massive reduction in HTTP requests this would mean. It might even mean that Pipes would be able to recurse into the conversations, which would be fantastic.

UPDATE: Darren Foreman has just shown me pbtweet, which also surfaces conversations in twitter, as well as a few other things, and is also apparently based on Pipes.