Stuff The Internet Says On Scalability For January 14, 2011

 Submitted for your reading pleasure...

• On the new year Twitter set a record with 6,939 Tweets Per Second (TPS). Cool video visualizing New Year's Eve Tweet data across the world. 
• Marko Rodriguez in Memoirs of a Graph Addict: Despair to Redemption tells a stirring tale of how graph programming saved the world from certain destruction by realizing Aritstotle's dream of an eudaimonia-driven society. Could a relational database do that?
• The never never ending battle of good versus evil has nothing on programmers arguing about bracket policies or sync vs async programming models. In this node.js thread, I love async, but I can't code like this, the battle continues. In the end programmers desire async, but leave the bar with sync.
• Quotable Quotes
  • @AmyDeLong: Walked into a starbucks and overheard 3 separate discussions all on scalability. #firstworldproblems #onlyinsf
  • @chvest: You may not need "high" scalability, but you should still consider your growth rates and prepare.

Click to read more ...

Google Megastore - 3 Billion Writes and 20 Billion Read Transactions Daily

A giant step into the fully distributed future has been taken by the Google App Engine team with the release of their High Replication Datastore. The HRD is targeted at mission critical applications that require data replicated to at least three datacenters, full ACID semantics for entity groups, and lower consistency guarantees across entity groups.

This is a major accomplishment. Few organizations can implement a true multi-datacenter datastore. Other than SimpleDB, how many other publicly accessible database services can operate out of multiple datacenters? Now that capability can be had by anyone. But there is a price, literally and otherwise. Because the HRD uses three times the resources as Google App Engine's Master/Slave datastatore, it will cost three times as much. And because it is a distributed database, with all that implies in the CAP sense, developers will have to be very careful in how they architect their applications because as costs increased, reliability increased, complexity has increased, and performance has decreased. This is why HRD is targeted ay mission critical applications, you gotta want it, otherwise the Master/Slave datastore makes a lot more sense.

The technical details behind the HRD are described in this paper, Megastore: Providing Scalable, Highly Available Storage for Interactive Services. This is a wonderfully written and accessible paper, chocked full of useful and interesting details. James Hamilton wrote an excellent summary of the paper in Google Megastore: The Data Engine Behind GAE. There are also a few useful threads in Google Groups that go into some more details about how it works, costs, and performance (the original announcement, performance comparison).

Some Megastore highlights:

Click to read more ...

Riak's Bitcask - A Log-Structured Hash Table for Fast Key/Value Data

How would you implement a key-value storage system if you were starting from scratch? The approach Basho settled on with Bitcask, their new backend for Riak, is an interesting combination of using RAM to store a hash map of file pointers to values and a log-structured file system for efficient writes.  In this excellent Changelog interview, some folks from Basho describe Bitcask in more detail.

The essential Bitcask:

Click to read more ...

BankSimple Mini-Architecture - Using a Next Generation Toolchain

I know people are always interested in what others are using to build their systems. Alex Payne, CTO of the new startup BankSimple, gives us a quick hit on their toolchain choices in this Quora thread. BankSimple positions itself as a customer-focused alternative to online banking. You may remember Alex from the early days of Twitter. Alex was always helpful to me on Twitter's programmer support list, so I really wish them well. Alex is also a bit of an outside the box thinker, which is reflected in some of their choices:

Click to read more ...

Sponsored Post: Newrelic, Cloudkick, Strata, EA, Joyent, CloudSigma, ManageEngine, Site24x7

Who's Hiring?

• Electronic Art's is hiring a QA Engineer and a Cloud Architect for their Pogo.com team.

Fun and Informative Events

• A new round of Membase meetups have been planned for January 2011 for San Diego, Denver, Seattle, Vancouver and Chicago.
• O'Reilly' Strata Making Data Work Conference on February 1-3, 2011 Santa Clara, CA. Strata is a new conference from O'Reilly, focusing on the business and practice of data.

Cool Products and Services

• Newrelic - What are you doing to ensure the performance of your apps?
• Cloudkick - monitor & manage your servers better with a FREE Cloudkick developer account.
• Join two game developers in a Joyent-sponsored webinar, You’ve Got Game: Planning for the Successful Scale and Performance of Your Cloud-based Game http://bit.ly/eAPt2s
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

Click to read more ...

Stuff The Internet Says On Scalability For January 3, 2010

 Submitted for your reading pleasure...

• Quotable Quotes
  • @hofmanndavid: Performance and scalability anxiety makes developers want to catch the flying butterflies
  • @tivrfoa: "Scalability solutions aren't magic. They involve partitioning, indexing and replication." Twitter engineer
  • Alan Perlis: Fools ignore complexity; pragmatists suffer it; experts avoid it; geniuses remove it.
• CIO update: Post-mortem on the Skype outage. Interesting tale of a cascading collapse in complex, distributed, interactive systems. For more background see the highly illuminating Explaining Supernodes by Dan York.
• RethinkDB and SSD Databases. SSD was not a revolution by Kevin Burton. What’s really shocking to me, is that while SSD and flash storage is very exciting, it wasn’t as revolutionary in 2010 as I would have liked to have seen.
• The case for Datastore-Side-Scripting. Russell Sullivan predicts real-time web applications are going in the direction of being entirely event driven, from client (WebSockets) to web-server (Node.js) to datastore (Redisql). And to complete the even driven chain is datastore-side-scripting.
• Developments that could change everything...

  Click to read more ...

Facebook in 20 Minutes: 2.7M Photos, 10.2M Comments, 4.6M Messages

To celebrate the new year Facebook has shared the results of a little end of the year introspection. It has been a fecund year for Facebook:

• 43,869,800 changed their status to single
• 3,025,791 changed their status to "it's complicated"
• 28,460,516 changed their status to in a relationship
• 5,974,574 changed their status to engaged
• 36,774,801 changes their status to married

If these numbers are simply to large to grasp, it doesn't get any better when you look at happens in a mere 20 minutes:

Click to read more ...

Pinboard.in Architecture - Pay to Play to Keep a System Small

How do you keep a system small enough, while still being successful, that a simple scale-up strategy becomes the preferred architecture? StackOverflow, for example, could stick with a tool chain they were comfortable with because they had a natural brake on how fast they could grow: there are only so many programmers in the world. If this doesn't work for you, here's another natural braking strategy to consider: charge for your service. 

This interesting point, one I hadn't properly considered before, was brought up by Maciej Ceglowski, co-founder of Pinboard.in, in an interview with Leo Laporte and Amber MacArthur on their their net@night show.

Pinboard is a lean, mean, pay for bookmarking machine, a timely replacement for the nearly departed Delicious. And as a self professed anti-social bookmarking site, it emphasizes speed over socializing. Maciej considers Pinboard a personal archive, where you can keep a history of what you are reading: forever. When the demise of Delicious was announced, if Pinboard had been a free site they'd have been down immediately, but being a paid site helped flatten out their growth curve.

Bookmarking sites used to about sharing links with your friends, but Twitter has largely taken over that role. Twitter, however, is infamous for presenting only a small slice of your tweet history. What you really want is a big server sucking down your bookmarks from wherever you might bookmark them, and that's just what Pinboard does.

A few points struck me as particularly cool about Pinboard:

Click to read more ...

Netflix: Continually Test by Failing Servers with Chaos Monkey

In 5 Lessons We’ve Learned Using AWS, Netflix's John Ciancutti says the best way to avoid failure is to fail constantly. In the cloud it's expected instances can fail at any time, so you always have to be prepared. In the real world we prepare by running drills. Remember all those exciting fire drills? It's not just fire drills of course. The military, football teams, fire fighters, beach rescue, virtually any entity that must react quickly and efficiently to disaster hones their responsiveness by running drills.

Netflix aggressively moves this strategy into the cloud by randomly failing servers using a tool they built called Chaos Monkey. The idea is:

If we aren’t constantly testing our ability to succeed despite failure, then it isn’t likely to work when it matters most – in the event of an unexpected outage.

They respond to failures by degrading service, but they always respond:

Click to read more ...

Paper: CRDTs: Consistency without concurrency control

For a great Christmas read forget The Night Before Christmas, a heart warming poem written by Clement Moore for his children, that created the modern idea of Santa Clause we all know and anticipate each Christmas eve. Instead, curl up with a some potent eggnog, nog being any drink made with rum, and read CRDTs: Consistency without concurrency control by Mihai Letia, Nuno Preguiça, and Marc Shapiro, which talks about CRDTs (Commutative Replicated Data Type), a data type whose operations commute when they are concurrent.

From the introduction, which also serves as a nice concise overview of distributed consistency issues:

Click to read more ...