Reflecting on Roses

It’s been a while since my last post on this blog, but as you can imagine, it’s all been go for team Nouse tech.

The week leading up to Roses was a big hard job for me and the rest of the development team, but one we accomplished well, matching the amazing work put in by the sports team, the photo team, Anna Bucks and the YSTV team, and everyone else who helped out, who as well as running the coverage and live blog were putting together a newspaper at the same time.

Everything went very smoothly technically, which was my main concern, especially with launching such a big new feature as the streaming live blog. I’m fairly sure I scared everyone in the Nouse office on Thursday night as I was shouting “fuck you” at my laptop repeatedly, normally whilst debugging in IE6 (approximately 15% of our readers still read in IE6, presumably due to computing services not upgrading the machines on campus), but it all seemed to work pretty well at the end of the day.

The drive behind launching the streaming live blog was two fold, firstly to reduce the load on Apache over the weekend and secondly to get the news out faster than anyone else. During testing, the news would slide in on the connected clients before the “Post successful” screen even loaded on our back end.

If you missed it, we also launched an image slideshow, more dynamic than the simple “picture of the day” we used for Elections, that hooked into the wonderful Flickr API to get the photos the photographers uploaded immediately. Mike Tomasello was responsible for that one.

So, back to live blogging, the crown in the jewel as it were. Now I’m going to get a bit technical, but there’s diagrams so hopefully not too technical (a rough version of this was in the office whiteboard, and Henry seemed to understand it with copious amounts of hand waving).

The bit of technology that made it all possible was Meteor Server. Basically, it serves a web page that appears to take ages to load – but web browsers will deal with web pages as they come in, not wait for it to load, so when you connect to the live blog, this web page opens in a hidden part of the page and then whenever Meteor receives a new message from us, it then sends it to all the connected browsers as a little bit more of the page. And it’s even more clever, because if this doesn’t work in your browser, or behind your connection, it “polls”, which means instead of having something open constantly, every 5 seconds it reconnects and asks if there’s any more data to receive. So at worst you’ll have a 5 second delay.

This is the software Twitterfall use to power their stuff, so big thanks to them for pointing me in the right direction and giving me some helpful tidbits.

Setting up Meteor was a bit challenging, least of all because it’s documentation isn’t great, but there also were bugs with it – chiefly the security settings it tried to set are invalid for .co.uk domains (like Nouse), but the bug didn’t prove to be too hard to track down and fix, and being good open source consumers, we gave the patch back to the community.

So, I wrote a WordPress plugin to link this all together (as well as pushing to Twitter), which works roughly as the diagram shows:

Once the message (actually an encoded JSON string) is received by the browser, the Meteor client library passes it to our event handler, written by Ali ‘Fantasy Football’ Clark, which does the magic animation stuff.

Then, of course, Richard Mitchell suggested we should have comment streaming. Another late night later, liveblogging.php catches comments as they’re posted (if they’re from an authenticated user, or when they get approved by a moderator otherwise) and then streams them out using Meteor (but on a different channel) and the event handler posts them in the appropriate place.

It’s not perfect, there’s still a couple of bugs to work out (the ‘Your comment is awaiting moderation’ message will stay on your screen even after moderation – but that’s not too difficult to solve), and sending a comment still requires a POST request – I’d prefer to have it AJAX’d, but the core of it is there.

So that’s it! That’s our secret behind live blogging. The plugin will be released as open source as soon as I’ve abstracted some Nouse specific stuff away.

Roses was a fantastic experience, it’s a shame I couldn’t get out and see any of it (I was the one responsible for updating the official score on roses2009.com, whose API fed the Nouse site, as well as being fed in part by the Nouse site itself), but as score central was YSTV, I did get to see whatever they caught on camera. I also got to see the reactions of LU Tube (Lancaster’s TV station) who were sharing the studio as our points just went up, and up, and up.