It's HighScalability Time (Good luck today):
- A Friday the 13th Postmorterama:
- James Hamilton with some high powered perspective on the report for the Fukushima Nuclear Accident. Apparently they haven't heard of the blameless post-mortem. Lots of interesting stuff, but this is a potentially disaster saving general lesson learned: operators can’t figure out what is happening or take appropriate action without detailed visibility into the state of the system.
- Evernote with a nicely detailed note on a recent outage. A kernel panic happened while upgrading two new “shard” servers with 3x as much RAM, SSDs instead of 15krpm disks, bonded networking, and an updated kernel. They had to revert and shite loves to happen when other shite happens.
- Heroku with their postmortem on what happened when AWS went down. They lost 30% of their instances across 3 AZs in the US-East region. Rich detail on the impact of the AWS, but not much on what they can do about it in the future, probably because there's not much to do unless you want to take the multi-region hit.
- Forget the money, follow the lack of power. Saleforce, like Amazon, suffered an outage because of a power failure. Why don't these expensive backup power systems seem to work?
- CloudBees’ Postmortem on Two Recent Outages: AWS and Leap Second Linux Bug: customers with a database running in a healthy zone weren’t impacted by this outage. No databases were lost.
- Scaling lessons learned at Dropbox: theme is robustify. Run with extra load, create custom stats, master the shell for analytics, verbose logging is good, test weak points, test by running code, keep it consistent, and many more. Nice article.
- @b6n: @jaykreps "All consistent state is local." -- Protip O'Neill
- Urban Airship on how they used HBase to construct a high write mobile backend to support hundreds of millions of devices and a frontend API that sustains thousands of requests per second: HBase offers operational ease and a low latency, high throughput system with known scalability characteristics.
- Bitly on Debugging a Specialized Database Cache. They use simplelevelydb to implement their view network history feature. Make reads fast by doing the hard work at insertion time. Valgrind to find memory leaks. Malloc overhead was very high so redesigned memory usage. Graphs and metrics uncovered a lot of bugs.
- Social and engineering are two words that don't usually go together in a good way. That's what makes this effort by Facebook to add social to their tools quite interesting. Phabricator, for example, is their internal code-review tool, that was extended using Open Graph to publish a single Open Graph action against a "diff" object. when changes are created, accepted, closed, or requests. Annoying no doubt, but an interesting use. A social dimension was added to a large number of other tools like Pixelcloud, Scuba, SIOG, StayFitB, and Pokemon. StayFitB tracks workouts using Open Graph actions each time an employee badges in to the fitness center.
- They are coming. They are coming in 2014 says HP. Memristors that is...and yes...memristors could still change everything.
- Ricky Ho with another great article, this time it's an expansive description of the Couchbase Architecture. Some key features: load balancing, async writes, append-only updates, automatic compaction, map function generated query indexes.
- HBase Log Splitting. Jimmy Xiang with a detailed explanation of how log splitting is used to "recover lost updates from region server failures."
- Architecting Data Center Networks in the era of Big Data and Cloud by Brad Hedlund: This session for data center architects will explore the transition from traditional scale-up chassis based Layer 2 centric networking, to the next generation of scale-out Layer 3 Leaf/Spine CLOS based fabrics of fixed switches.
- Scalr's Sebastian Stadil and Julien Rialan with a wonderful interview on FLOSS Weekly 217. You may recognize Sebastian as the founder of the Cloud Meetup in Sillycon Valley. Today's fun fact: Scalr's software was used to help discover the Higgs.
- Scalability and Reliability in the Cloud by Greg Thompson. Nice slide deck covering scalability factors, vertical vs horizontal, stateless applications, connection management, segmenting traffic, segmenting responsbility, clustering, and much more.
- Now this is testing: NBC, Google, Stage ‘War Games’ To Prepare for Olympic Disruptions. Let's hope those emergency power generators cooperate.
- Don't get a divorce Mommy and Daddy memcached, I hate it when you argue.
- Benefits of High-Frequency Monitoring. John Bergmans with an excellent summary of an Open Space Session. Major finding: High-Frequency Monitoring Can Enable Short Response Times to Anomalies.
- Reagents: Expressing and Composing Fine-grained Concurrency: We introduce reagents, a set of combinators for concisely expressing concurrency algorithms. Reagents scale as well as their hand-coded counterparts, while providing the composability existing libraries lack.
- SML: Data Streams - STATISTICS 241B, COMPUTER SCIENCE 281B. Learn all about: Random subset, Heavy hitter detection, Randomized statistics, and Realtime analytics.
- Organizing Functional Code for Parallel Execution; or, foldl and foldr Considered Slightly Harmful
- Swarm Intelligence: A whole new way to think about business. If you think you are a drone now, just wait...
- Presentation: Progressive Architectures at the Royal Bank of Scotland: Its architecture introduces a non-allocating queue specialized for multiple producers and single consumers.
- ThemisMR: An I/O-Efficient MapReduce: a MapReduce implementation that reads and writes data records to disk exactly twice, which is the minimum amount possible for data sets that cannot fit in memory.
This weeks musical selection: