1. First, write code that you know works.

   Any moron can write code that works. This is called hacking, and it’s a pavlovian-conditioned pleasure. “Hey, it works. Can I have another biscuit now?” Sometimes, code that works is copied from other code that works. That’s ok, we’ll deal with it later.

   How do you know it works? You’ll need tests, of course. Do not go on to the next step until you have tested code that works.

   The hard bits are yet to come. You are far from finished.

2. Now, remove any duplication.

   This is the hardest part of programming and takes years of experience to get passably good at. The programmers I think of as naturally talented have the best noses for sniffing out duplication. Experience gives you a range of techniques by which to remove it.

   Do not remove duplication if it stops the code working. Sometimes duplication is easy to spot, othertimes it’s more subtle; sometimes it’s easy to remove, sometimes it’s very tricky.

   This is where familiarity with the idioms and features of your chosen language is important.

   This is the hardest part. If you want a name for this stage, you can call it abstraction or, if you like, design.

3. Now, make the code clearly express your intentions.

   Removing duplication has already helped you here. But there’s likely further to go. Well-named methods, variables, classes. Spend some time here. Aim to concisely express, not to literally describe.

   Please don’t think that this is where you comment your code. Prefer clearly written code over comments. Readable, beautiful, clear code is your aim. Trust that the how and why of the code will take care of itself.

   The tricky bit here is that clarity is subjective. But, if you rename and restructure so that a programmer of your level can read the code and understand it, you’re in the right mindset.

   You’re likely an average programmer. Your own understanding is a good benchmark.

   Don’t introduce duplication and keep the code working. We’ve been there already, and we don’t want to slip backwards.

4. Remove redundant code and classes.

   Just a final chance to tighten the nuts. And if you’ve got this far, you’re probably already in good shape, but this is where you have a chance to polish.

   The tests you wrote in the first stage keep you in check here. Don’t regress: keep the code clear and free of duplication.

If you do these things, in this order, you will, with practice, produce better code and become a better programmer. There’s no magic here. Give it a try.

Last week I was asked how to handle running a shell command from Ruby, with the added requirement that if the command took longer than a certain time it should time out.

Running commands from ruby is easy, and you should watch the video of my Ruby Manor talk for some of the gory details.

But, is there a simple way to time-out a shell script using ruby? Well, you could probably write some heady threaded code that does it. But don’t. This is what the shell is for.

The usual Unix caveat applies: in the seventies, under the influence of prog rock, speed and the Californian sun, a bearded fellow already wrote the exact tool you want. And that tool has no doubt shipped with every Unix ever since. And everyone but you uses it daily.

But, this time, I couldn’t find that tool. There must be one. Anyone know of it?

Anyway, this is what I did:

#!/bin/bash

# timeout
# Usage: timeout timeout_in_seconds command [args]
# Example: timeout 120 wget http://www.slowserver.com 

timeout=$1
command=$2
command_args=${@:3} 

$command $command_args &
pid=$!
( sleep $timeout && kill -9 $pid ) &> /dev/null

wait $pid &> /dev/null
exitcode=$?
exit $exitcode

We launch the main command in the background; then, we launch a second command that sleeps for the requested timeout before attempting to kill the first one; then, we wait for the first command to finish (either of its own accord or thanks to a kill signal), returning its exit code as our own.

It’s brutish and dumb, but for most cases this simple approach works.

Be careful if your machine is going through PIDs like billy-o -you could end up sending the TERM signal to the wrong process -, but in general use that shouldn’t be a problem.

You could make it safer by storing start time and PID and making sure they both match, or something similar. Left as an exercise to the reader, as us lazy folk say.

A few bash things you may not have seen before: the special parameters !, @ and ?, wait and bash parameter substring expansion.

$! expands to the PID of the most recently executed background command, and $? expands to the exit status of the most recently executed foreground command.

“$@” is equivalent to “$1″ “$2″… I use the ${parameter:offset} notation to grab a slice that array.

wait is a useful shell built-in that takes any PID, waits for it to finish and returns its exit code.

Much more detail, of course, lurks in the bash man page – I still find I look up these cryptic characters or copy them from an existing script.

There’s a standard tool that does this better, right?

Being latterly obsessed with rights and culture, I’m certain that that lovely room was one big rights violation, from the use of trademarked characters to the showing of a film without licence.

I’m of course guessing here, but I’d bet that there was no legal permission sought. Should we really care about that? I can’t bring myself to.

When things become part of our culture, they should transcend the petty rights-squabbling of corporations and their lawyers.

In my moral calculus, the rights of kids to be surrounded by and participate in familiar, comforting culture trump the rights of corporations to squeeze money from ideas they bought from artists.

We should need pretty strong evidence that future culture creation is threatened to mess with that equation. I don’t think that evidence is anywhere like as strong as rights people contend.

In the digital world, it’s looking like enforcement and detection is going to be easier, even if the so-called crimes are the same.

Imagine a version of Adobe Photoshop that won’t let you draw a picture of Mickey Mouse. Somewhere a rights lawyer just exploded with that possibility. Will the next version of Garageband let you record cover songs?

And you can bet it’ll happen. My paranoia reckons within 5 years it’ll be seriously proposed by someone, if it hasn’t been already.

Why are we letting this wholesale denial of folk-culture happen? Remind me again.

This is why I oppose what the BBC is doing with its content protection.

The humane solution is not to regulate these architectures, but to remove them.

But, first up, another architecture of control: train ticket barriers.

A couple of months ago, I went to Paddington to meet my wife and child off the train. I wanted to help them off the train, with pushchair, luggage and all. But the guard at the ticket barrier prevented me from so doing.

It wasn’t his fault. He was just doing his job, that’s just the way things work, no-one is allowed through. Can you just wave me through, I want to help my wife off the train with the pushchair? “No, I can’t do that”.

Not much harm done, but real human moments missed – waving hello, waving goodbye, being there to carry someone’s bags. All in the name of fare-protection. All for my benefit.

I don’t want to overstate what happened. Martin and his family had a much worse experience than I did.

But, these are not systems that encourage kindness. These are systems that create bureaucrats and pettiness, and all manner of knock-on grief.

Typically, these systems are put in for the benefit of the majority. They’re well-meaning. That makes it much harder to argue against them. You’ll be characterised as not having taken seriously a certain issue, in this case fare-dodging, or as trying to promote a minority’s interest at the expense of the common-sense majority case.

No cost-benefit analysis covers these human stories – what cost do you attach to waving good-bye to your family as the train pulls out, or to helping an elderly relative find their seat?

But I’m convinced, these are things that we should be accounting for. Our progress is better judged by its humanity than by the technological prowess of the Oyster card.

Incidentally, I was reminded of another, much worse aspect of this system, reading this article about Kids Company:

Some of the solutions may strike others as not so much flexible as perverse. For example, when Kids Company clients kept being arrested for assaulting London Transport staff, Batmanghelidjh’s solution was to issue them with free travel passes. But the arrests ended almost overnight.

So, to the BBC’s content protection shenanigans. Opposing these doesn’t make me a pirate, any more than I’m siding with fare-dodgers above.

Again, rational, well-meaning folk think that exercising levers of control (in this case, on copying culture) will provide a net benefit to the majority. And, again, there are real human, and loving, stories to consider.

Here’s one: my wife’s cousins moved to Italy when they were young. My wife’s family set up an shuttle service of video tapes. Every week they’d video a selection of children’s TV and send the tapes to Italy; the next week they’d get the tapes back and record another selection to send – they couldn’t afford too many tapes.

This was kindness and contact and real. But, under the cost-benefit analysis, it’s piracy, too.

My Dad would wheel out a cine-projector at my birthday parties and play copied Woody Woodpecker cartoons to my invited friends.

Again, an awful piratical move, but human and real.

Video-tapes are awful inhuman, revenue-destroying, copying machines. But they’re also the Will Hay films my grandparents taped from the TV and watched again and again and again. On a twenty-year-old TV that needed no bureaucratic permission to watch them.

Anyway, you get my drift. None of these things show up in cost-benefit analyses of copy-protection. But they must. We should demand that these stories are taken account of. In fact, I would demand that for public service they should be the starting point of decision-making.

Even if the current architecture is mild in effect, it will spawn bureaucratic pettiness; it will serve as an example for anyone who wants to ratchet up the control dial in future; it will ultimately hurt us all, in a million petty little ways that just won’t show up on the balance sheets.

I can’t believe that the people asserting this control don’t have their own, personal examples of sharing culture. Of bootlegging songs from the top 40, or albums from their friends, on to cassette tapes. Of watching a copied video tape so many times that it was a snowstorm of noise. Of being given a mix-tape by a loved one.

The BBC is complicit in rolling out an architecture not based on kindness or on human stories, but on bureaucratic cost-benefit analyses. And we must resist that. We should consider all the kindness these systems kill.

If you have any stories of how copying content was a kindness, please leave them in the comments below. I’d love to be able to submit them the next time Ofcom has a consultation.

I was so pleased with the hardback version of my twitter book that I knew I had to make another book. And quickly.

At the Warblecamp hackday last Saturday/Sunday I set about making book from one of the twitter events I’d been following. The Twitter Calculus Wars is a fun project where three Algebra students from Michigan replayed the famous Newton/Leibniz rivalry on twitter.

I took their excellent work and turned it, along with some footnotes from wikipedia, into a 60 page book. You can download the final PDF here.

That was on Monday. The first printed copy of it arrived today (Friday). That’s just four days – some excellent service from lulu.com. It looks like it was printed in Eastbourne, whereas the hardback I’d previously had printed came from the states and took a bit longer.

Here it is:

I’m very, very happy with it. It feels like a book. I had worried that it wouldn’t – that it’d be too contrived, or there’d be some terrible flaw in the design that I hadn’t noticed, or that it just wouldn’t work in some way. But, it feels like a book.

The fonts work well, and I’ve learnt lots about the differences between screen and print. Some things that are difficult on the screen, work well in print – in this case, I was worried that using different fonts for footnotes and body copy would confuse. And on screen, it does somewhat; in print, it actually works well.

One other thing I learnt is that next time I’m making a twitter book, I’ll probably need to choose a font that has a hash (#) character. I had to cheat throughout and use the section mark (§) instead.

It’s a long while since I made something that I’m really proud of and that I don’t feel is compromised in some way. I’m getting back a lot of the confidence I’ve lacked recently, I think.

Anyhow, it’s easy and fun to get a book printed, and it’s pretty cheap too. It cost me $6 or thereabouts to get the copy printed and another $10 in super-express-do-whatever-it-takes-to-get-it-to-me-now postage. There are cheaper postage options available, if you’re more patient than I am.

The techniques I used were the same as with my twitter_book project. Firt, ruby scripts to pull tweets from twitter (and also wikipedia in this case) and emit some xml. Then, an xsl stylesheet to convert that xml into something I could feed into Apache fop. The resulting pdf was uploaded straight into lulu.com.

All of this makes me want to make another one. And quickly. Keeping my eyes open for ideas…

I’m still not sure what I think about the exercise.

Twitter’s the closest thing I have to a diary. I’ve never tweeted specifically in order to make a daily record, but that’s the form I chose for the book.

And, I’ve got my own, personal reasons for wanting a record of the last couple of years.

So, this book – it’s 393 pages, give or take. And it has just short of 5,000 of my tweets in it. It cost me £20 to get printed.

It certainly does work as a device to take me back to certain times and certain places.

I like the way the references to webpages, events, and even people are already somewhat cryptic and heading towards being lost altogether. I do wonder what I’ll think of it in 5, 10, 15 years time. I wonder how much I’ll remember.

In form, it reminds me of a play or script, and there are certainly dramatis personae. But I’m the only one who speaks, and there’s little drama to the thing.

Sometimes, though, in the midst of my ramblings are some magic tweets – our last night in Berlin; the 12-week scan; Leonard coming home from hospital.

Browsing through it feels like going through an old box of photos.

This version is something short of beautiful. But it’s a start. I’m going to literally put it on the shelf for a month or two and then come back to it.

In the meantime, if you want to make your own, you can try to run the scripts I wrote. I got this copy printed in Blackwells on Charing Cross Road, where they have a special in-store bookmaking machine. I’ve got another one on order from lulu.com in a slightly different format.

Using the simple webservice I’ve written, it’s a bit easier to use scraperwiki data in YQL queries and to mash up scraperwiki data with other YQL sources.

YQL

YQL presents a uniform query interface, modelled on SQL, for various web APIs. You can run queries like select * from flickr.photos.recent to get a list, in json or in xml, of recent flickr photos.

Data can be mashed together from multiple tables/urls, like so:

    select * from search.web where query in
        (select title from rss
            where url="http://rss.news.yahoo.com/rss/topstories"
            | truncate(count=1))
    limit 1

[run that in the YQL console]

By publishing a chunk of XML and javascript by way of an adaptor non-Yahoo! services can be queried through YQL.

There are already adaptors published for many web APIs. YQL calls these things ‘datatables’. And you can use them like this:

    USE "http://myserver.com/mytables.xml" AS mytable;
    SELECT * FROM mytable WHERE...

With me so far?

Scraperwiki

Scraperwiki is a new, still-in-beta service for building web-scrapers and sharing the data they scrape. More scrapers are appearing daily and the site provides a useful API for querying the data created.

Using scraperwiki in YQL

I’ve created a service that automatically generates YQL datatable definitions from scraperwiki scrapers. You can find the definition for any scraperwiki scraper at a url that looks like this: http://swikiyql.heroku.com/SCRAPERWIKI_SHORT_NAME.xml

You’ll need a scraperwiki API key – you can sign up for one on their site. But once you’ve done that you should be able to run YQL queries like this

    use 'http://swikiyql.heroku.com/wikipedia-2010-uk-election-candidates.xml'
        AS candidates;
    SELECT * FROM candidates
        WHERE party='UKIP'
        AND sw_api_key='YOUR_SW_API_KEY';

which queries the data from this scraperwiki scrape. And using this, you should be able to mash together different scraperwiki scrapes, or mash scrapes with any other YQL tables.

That’s it really – the source code for the swikiyql (pronounced swikiyql) is on github and there are probably lots of things wrong with it, but it works for me.

Now, go, mash things up!

Christmas tree decoration seen in Canary Wharf shopping centre reads: CCTV in operation here.

This has to be a prank, right? There’s no way, even in the Orwellian Albtraum of Canary Wharf, that someone would have decided this an appropriate Christmas decoration for the shopping centre’s christmas trees.

Alas, I fear it’s real and intentional: what you see here is the red version of the sign on the red-lit trees. Similar colour-coordinated signs were on the blue trees too. In fact, this sign was on every damned Christmas tree in the whole of the damned centre.

It’s possibly the most obnoxious, awful thing I’ve ever seen come out of this obnoxious, awful managerial surveillance culture of ours. This defies parody. If it’s a prank, it’s an excellent one. But…

You know, I hate the polythene-thin sentiment of Christmas, in so many ways. But even I think we’re worse off if Christmas means goodwill to all men, and remember, we’re watching you people.

This is what I came up with.

The script takes one argument – the term to search on – and creates a sqlite3 database in the current directory containing all the tweets that the twitter search API returns for that term.

It requires datamapper and twitter_search gems to be installed.

Have at it, make it better, bend it to your twitter-slurping needs.

You’ll need a room, account, etc. in Campfire; you’ll need to compile this version of dumb-frotz; and you’ll need a zcode file to play – plenty of those at the Interactive Fiction Database.

I only wrote about this because @lazyatom called me a genius for doing it. Which was nice.

Have fun.