Up until now I’ve used a collection of Smart Playlists which take into account my ratings, when I last listened to songs and a whole bunch of other information. At the last count, I had over 70 Smart Playlists feeding into each other to generate the playlists which finally get Synced over to my iPhone.

But there was something missing. Whilst I could generate playlists based on the mid-90s simplicity of the “Genre” tag, that tag is by design only able to identify one piece of information. So I wouldn’t be able to find, for example, all tracks in my collection that are “rock”, “funny” and performed by a “female”. And even if I could, I wouldn’t want to go and tag 40,000 tracks by hand. That way madness lies.

It turns out there’s a fairly simple solution to this – last.fm allows users to tag any and all tracks it knows about, and it keeps track of how many times each track has been tagged with each tag. So, it’s entirely possible to grab the top 15 tags for each track in my collection, add that data to my tracks somehow, and then query that.

The somehow turns out to be pretty simple too – the ID3 “comment” field is there for the taking, and by design is expected to be larger than any other field. Excellent. So that’s the “where to store” sorted, now how about “how to store so that the data is queryable”?

For this I took a trick out of Brian Suda’s book. A couple of years back I took a look at how he’d written his microformats parser using XSLT to be able to check for classes an element may or may not have – the trick being to wrap strings of classes with a space at the beginning and end so that every class always had spaces surrounding it. I used the same trick for storing my tag data, also wrapping it in a ‘lfm’ square bracketed enclosure to separate it from any other data in the comment.

So, the data I saved in the comment looked a little something like this:

[lfm: tag1 tag2 tag23 tag3 ]

You’re probably starting to see why wrapping in spaces is important now. Because iTunes’ Smart Playlist system can only perform simple string matching (Apple doesn’t seem to like regexes – AppleScript can’t do them natively either), you need some way of targetting the beginning and end of tags. Without those spaces, a query of “Comments contains 'tag2'” would return you both tag2 and tag23, which no-one wants. By requesting “Comments contains ' tag2 '” instead (note the extra spaces) you end up just being given tag2.

All this thinking is packaged up into a couple of scripts (though obviously take the latest code if you want to actually use it yourself) I wrote a couple of days ago that takes a given playlist from your iTunes library, asks last.fm for tags for each of the songs in that playlist, and then adds that data to the library using AppleScript.

Problems?

There are, unfortunately, a couple of issues with the script at the moment. First off, because last.fm allows anything and everything to be given as a tag, they can contain characters that need to be normalised out. Right now I’m doing that very simply (spaces to dashes mostly), but I should revisit the code and harden that normalisation – I’ve seen tagging bail out a couple of times because of unexpected data.

Next up, the fact that last.fm allows tags to be incredibly long coupled with the fact that under ID3 v2.x the comments field can only be 256 characters long means that it’s possible to run into issues with tagsets being too long to fit into the comments field. Add that to the fact that some songs in my collection legitimately contain comments already and that I want to add tags non-destructively, and we’re ending up with not much space for tags. So, an improvement that I’m planning on making to the script is to make sure that tags try to take up no more than the space that’s legitimately available to them.

The final issue that I’ve been having lives with last.fm’s track.getTopTags API method. Unfortunately at present the track.getTopTags API method does not perform any auto-correction of track names, which means that if you try to get the tags for something that does get auto-corrected by the system then you end up with the tags for the uncorrected track name. A subtle distinction, but it probably means that we end up with unmaintained tags that haven’t been touched for the year or so since auto-correction was turned on. A possible solution here if last.fm choose not to “fix” this behaviour (it could be argued that this is correct behaviour) would be to make a request out to Musicbrainz to get an ID for the track and then pass that into track.getTopTags instead. But that seems like a whole bunch of extra work for a not massive gain.

The final Smart Playlist

So, what about that Smart Playlist? After all, that’s the thing that all this work was done for.

In iTunes, select File, New Smart Playlist, and then enter the following:

Match all of the following rules:
    Comments contains ' female '
    Comments contains ' rock ' ...
    any of the following rules:
        Comments contains ' comedy '
        Comments contains ' funny '

This cunning playlist uses a Smart Playlist feature I wasn’t aware of until yesterday – rule groups. Up until now, I’d been pulling in other Smart Playlists as sources to perform the same job.

To create a new rule group in a Smart Playlist, simply click an ellipsis button in the Smart Playlist interface, rather than clicking on the plus button.

Essentially what these rule groups give you is the ability to nest rulesets, and generate very complex playlists that don’t have half their rules hidden in external playlists. In the playlist above we’re simply asking for all tracks tagged with “female”, “rock” and either “funny” or “comedy”. So essentially “female comedy rock”. At present this playlist returns me no items from my collection, but not everything’s been tagged yet. I’m hopeful.

Well, colour me surprised.

I’ve been using Spotify for a little while now, but only ever to listen to particular songs that I wanted to hear whilst away from my full media library or to listen to artist radio. I’d never been enticed by playlists in Spotify in general, mainly because I only ever used it for little bits and pieces here and there. Finding out that I could use it to get people to suggest music to me was a revelation.

So, I tried a little experiment. This morning I tweeted a link to my collaborative playlist, asking people to add to it. My aim was to press play at that point and just keep listening to the playlist all day, with no self intervention. I trusted people to add music that they thought was good, with a little bit of adding amusingly bad songs for the hell of it. Sure enough, Aqua made it in there with “Barbie Girl”, as did Celine Dion with “My Heart Will Go On”. But, I listened to them all. Well, all of them except for Enya, but you have to draw the line somewhere, don’t you?

During the course of the day I listened to many artists I’d forgotten about, as well as plenty I didn’t know at all. One of the nice things that came out of this experiment was finding a load of new songs that I liked, and thanks to George Brocklehurst’s Lastify add-on for Spotify I was able to easily mark them as “loved” on Last.fm so that I could come back and listen to them again later.

By the time I finished working I’d only managed to make it halfway down the playlist that had been created. So, tomorrow I’ll be carrying on where I left of and listening to some more new to me music.

There are a couple of things that I wish were possible with Spotify’s Collaborative Playlists though. First off, I wish I could see who it was who added certain songs, so I could gently mock them for their choices. Following on from that it would be nice if you could lock down a playlist so that only certain users could add to it, or block certain user entirely. Not that I had any problems today, but it’s still a worry that someone might go crazy and just add fifteen hours of Rick Astley to a playlist.

Why not give it a go yourself? Just create a playlist in Spotify, right click on its name and set it as collaborative, and then give people the URL. You’ll probably be pleasantly surprised by what you get added.

For those of you who aren’t sure, a Genius Playlist is a playlist of music that all “works together” that’s created from one seed track of the user’s choice. Essentially, it’s a shortcut to creating playlists of stuff that suits how you’re feeling at that moment, rather than you having to go to the trouble of creating the playlist yourself.

Whilst I was initially sceptical of how well this would work (I’ve worked in this problem space before, so I know it isn’t awesomely easy), since iTunes gained this feature in v8 I’ve become more and more reliant on it. My reliance has been so great that I’ve expected the feature in all the other music based services I use. For example, when I started using Spotify, one of the first things I did without thinking was to start playing a track that I wanted to listen to and then look for the “genius” button to see what else Spotify would try and play for me. It upset me that this feature wasn’t there. Sure, Spotify and last.fm allow you to play radio stations based on a certain genre, artist or tag, but that misses a piece of perceived control that starting with a chosen song implies.

So, for last.fm’s hack day I decided to hack together a genius playlist creator. After all, last.fm’s API supplies the track.getSimilar method, which pretty much gives all the data you need for a genius playlist.

There were a few hurdles to get through though. First up was playlist creation and playback itself. It turns out that in last.fm-world you can only create actual playlists if you’re a paid subscriber to the service. That’s not a dealbreaker for me, so I paid my £1.50 for a month’s subscription to see how it went. Unfortunately, at that point I realised that whilst I could now create playlists, I could only play them back if they had at least 45 songs in them by at least 15 different artists. Oh, and I could only play them back in shuffle mode. On the website. Not in the desktop app or on the iPhone. Somewhat sub-optimal.

So, I went back to the drawing board. If I couldn’t do playlists how I wanted in last.fm, how about if I found the data I wanted in last.fm and then used Spotify to play it? Unfortunately, Spotify doesn’t yet have an API, so this also proved impossible.

Scratch Plan B then. Thankfully, I did have a Plan 9 available to me – last.fm’s ability to autostart tracks on the website by adding ?autostart to the end of any song page URL. Using this ability, I reckoned, I would be able to create a page containing a list of songs that could then be played in a browser window by loading up the individual song pages in a hidden iframe. Simplicity itsef, and as if by magic it actually worked!

There was, of course, more magic that went into creating Last Genius. Not all songs on the site are available to stream, and of those that are not all of them are full length tracks. Because I was playing the songs in iframes I had no way of knowing when a song had actually ended. I had to make sure that you didn’t end up hearing the same song twice in a playlist. All these things and more were hacked around during the hack day, and in the end I had something that actually worked.

The end result was a bookmarklet that you can stick into your browser toolbar and click to create a genius playlist any time you want. Just make sure you’re on a Last.fm song page and a playlist will be created for you. If you’re not on a song page, lets just say that the system will try and find you some music that it reckons you’ll like.

Last Genius bookmarklet – drag onto your toolbar.

All I ask is that you remember that this is a hack. It’s only tested in Firefox 3 on the Mac, because during a hack day you don’t have time to make everything work everywhere (it’ll probably work elsewhere, I just haven’t tested yet). Because Last Genius has to make quite a few calls back to the mothership to set itself up, it might also take a little while to start up. Just be patient. For what it’s worth, I have plans to improve this over Christmas.

Because this is a toy I want for myself I plan to spend a bit more time making it nicer. With that in mind, if you have any ideas for things that will make it better, please let me know. Likewise, the source code is up on GitHub, so please go fork it and make improvements. Anything awesome I’ll very happily merge back into the trunk.

It turns out that you can increase the number of tracks returned by the feed simply by adding a limit=somenumber parameter to the URL. So, for me to grab the last 20 songs that I’ve scrobbled, I would ask for:

http://ws.audioscrobbler.com/1.0/user/NeilCrosby/recenttracks.rss?limit=20

I use this as a way of getting a decent sampling of my recent tracks to use on NeilCrosby.com, as I found the default 10 was too limiting since my “Stream of Consciousness” stream normally covers a few days worth of information, whereas my last 10 tracks would only cover an hour or so of data.