Entries from August 2006 ↓

My colleague, James Mead, has just released a new version of his mocha stubbing and mocking library for ruby. This one features rspec compatability and auto-verification of assertions. Go and get it and start ridding your rails code of fixtures right now.

Excellent progress on the ruby on rails weblog just now moving the security hole issue forward in an unemotive, measured, factual and constructive way.

We’re thanked for our patience, but still no sorry? Only kidding…

Well done chaps, much better.

So the Rails chaps have had their first crisis. And, they’ve not handled it too badly – believe me I’ve done worse – but the response still left a little to be desired. So, rather than join in with the angry commenters on their blog posts (with whom I broadly agree) I thought it better to give them some pointers that I’ve learnt from years of mishandling crises. I plan to write some more about how to structure a crisis response process shortly, but in the meantime here are some general guidelines:

What do we know, and how do we know it?
The big mistake made yesterday was that the rails crew, especially DHH, starting communicating what they thought and not what they knew. Panic makes you do this, arrogance makes you do this, I’ll let you decide what was going on there.

From the original blog post:

UPDATE: This problem affects 0.13, 0.14, 1.0, and 1.1.x. So here’s a happy opportunity to upgrade if you still haven’t.

This turned out not to be true. I can imagine the scene in the rails secret headquarters. ‘Shit. What versions does this affect?’ ‘Um, all of them I guess’. Looks like no-one asked ‘how do we know that?’

There’s no harm in being absolutely straight up about this – tell people how you know what’s going on. So what the post should have said is something like: ‘we’ve confirmed this problem exists on 1.1.4 and are looking at other versions as we speak’. If someone had asked ‘what do we know and how do we know it?’ the debacle would have been avoided – and I wouldn’t have spent 5 hours that I don’t have to spare last night trying to contain a problem that didn’t exist.

Listen to the quiet voices
I’ve noticed that this is a particular problem that affects developers and the more arrogant a developer, the more dangerous they are in a crisis. Choose who you want to head crisis teams appropriately – the loudest voices are the worst in these situations.

Be serious and don’t even think of selling – now’s the time to apologise
So, it’s a matter of style, I suppose, but serious problems require a measure of seriousness from those solving them. And, don’t even try to turn it into a sales opportunity. It doesn’t wash:

We’re still hard at work on Rails 1.2, which features all the new dandy REST stuff and more
So here’s Rails 1.1.5!
So here’s a happy opportunity to upgrade if you still haven’t.

And for god’s sake: Elton John may think that sorry’s the hardest word, but you need to say it – it’s just polite.

Explain the impact
Rails is getting wider adoption and being used in revenue- and business-critical applications.

There is always a risk associated with upgrading and no serious development shop should take it lightly.

Yesterday’s announcement had no realistic impact assessment: what kind of security vulnerability was exposed, how do I judge whether I need to get on the phone and rudely wake-up our developers. No excuse for this one, the ‘hiding from black-hats’ argument just doesn’t wash I’m afraid.

Test your fixes and keep the panic inside

UPDATE 2: We’ve fixed the zlib buffer problems for people on Windows. Redownload the gem and everything should be dandy.

Uploading a broken gem was inexcusable – haste is a the most serious of errors in these situations. Make sure that the sense of panic that you probably feel isn’t passed on to the people you’re trying to help.

Listen to more country music
Kenny Rogers got it right when he said: ‘there’ll be time enough for counting, when the dealing’s done’.

(Hmm. I’m not sure the last point is something I’d entirely recommend).