The Dreamhost $7.5 million Oops

This morning I was checking my email and noticed a billing email from Dreamhost (my web host) and I thought it was sort of weird. Turns out they had a slight snafu with their billing software and they basically billed their entire user base for the entire year of 2008.

They have since started reversing all of the charges for their users and posted the following on the Dreamhost Blog detailing what happened:

DreamHost Blog » Um, Whoops.

Now, I’m a little annoyed by this but not so much as to drop everything and leave. Dreamhost has had it’s ups and downs but for the most part it provides a good service for a fantastic price. What I’m more interested in is the reason they’ve given for the billing error in the first place. Now, I’m interested because I write software for embedded systems for a living and we have to be very conscious of the code we write as it’s very difficult (and super expensive) to fix issues once the product has gone to the customer.

Basically, their billing system was written in such a way as to allow future dates to be run through the system. Why? Someone decided (or more specifically, failed to decide) that the system should be able to bill for any date, past, present or future:

Well guess what… NO! We will NEVER want to rebill as though today were a day that hasn’t happened yet! But instead, somebody along the line (Sage? Me? Somebody else?) figured, “What’s the harm in keeping it flexible?”

Flexible? This is not a flexible design, it is a massive failure waiting to happen. Now, given the fact that Dreamhost can potentially bill for any date in the future, they are damn lucky on this one as it may have been as easy to have another typo – December 31, 2070. Could you imagine how much money Dreamhost would have been attempting to pull out of their customers? If they had a $7.5 million billing error for an 11.5 month goof, what would a >60 year billing error look like? How about around $465 million? For me personally, I could have been dinged with a credit card charge of over $7000 if that had happened. I’m not sure Dreamhost would survive the backlash (and likely lawsuits) of a billing error of that size.

So, from the coding point of view, what can we learn from this? Well, one thing is bounds checking. You always need to ensure that the data you’re about to process is “reasonable”. What’s reasonable? Well, for a billing system, I’d say 30 days in the future is a reasonable maximum. (Ideally, 0 days in the future is what it should be but maybe you want to allow it to happen in some specific situations.)

What about ensuring that your system has some sort of safety checks to validate the user input:

You’ve specified December 31, 2008 which is a future date, are you sure you wish to continue with this operation?

Super simple even if it’s annoying to see. Heck, I’d guess that 99.999% of the time the operator would not see such a warning at all even if it did exist.

3 thoughts on “The Dreamhost $7.5 million Oops

  1. Mark W

    The explanation on Josh’s blog is at best, mistaken, and at worst, a lie. People did not simply get billed erroneously for the entire year of 2008. I was billed TWICE for all of 2008–almost. Two random months (neither at the beginning nor end of the year) were billed only once, while all the rest were billed twice.

    This is not explained by a single erroneous run of the billing script with bad parameters. What actually happened? It doesn’t add up. He’s covering his ass about SOMETHING.

    Reply
  2. Lady

    I see your point. Sure, it would be a good idea to have the program issue a little warning if one was to accidently make a typo such as they did, though it would probably be missed. Even the best people get to the point of seeing the same screen over and over again and it starts to look the same after a while–even if it isn’t.

    Reply
  3. Pingback: Dreamhost - The place to be in 2008! at Ivany.org

Leave a Reply