It's HighScalability Time (Good luck today):

• A Friday the 13th Postmorterama:
  • James Hamilton with some high powered perspective on the report for the Fukushima Nuclear Accident. Apparently they haven't heard of the blameless post-mortem. Lots of interesting stuff, but this is a potentially disaster saving general lesson learned: operators can’t figure out what is happening or take appropriate action without detailed visibility into the state of the system.
  • Evernote with a nicely detailed note on a recent outage. A kernel panic happened while upgrading two new “shard” servers with 3x as much RAM, SSDs instead of 15krpm disks, bonded networking, and an updated kernel. They had to revert and shite loves to happen when other shite happens.
  • Heroku with their postmortem on what happened when AWS went down. They lost 30% of their instances across 3 AZs in the US-East region. Rich detail on the impact of the AWS, but not much on what they can do about it in the future, probably because there's not much to do unless you want to take the multi-region hit.
  • Forget the money, follow the lack of power. Saleforce, like Amazon, suffered an outage because of a power failure. Why don't these expensive backup power systems seem to work?
  Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...