RAFT - In Search of an Understandable Consensus Algorithm

 If like many humans you've found even Paxos Made Simple a bit difficult to understand, you might enjoy RAFT as described in In Search of an Understandable Consensus Algorithm by Stanford's Diego Ongaro and John Ousterhout. The video presentation of the paper is given by John Ousterhout. Both the paper and the video are delightfully accessible.

mcherm has a good summary of the paper:

A consensus algorithm is: a cluster of servers should record a series of records ("log entries") in response to requests from clients of the cluster. (It may also take action based on those entries.) It does so in a way that guarantees that the responses seen by clients of the cluster will be consistent EVEN in the face of servers crashing in unpredictable ways (but not loosing data that was synched to disk), and networks introducing unpredictable delays or communication blockages.

Here's what Raft does. First, it elects a leader, then the leader records the master version of the log, telling other cluster servers what's in that master record and "committing" a log entry then responding to the client of the cluster to acknowledge that entry only when more than half the cluster has recorded a given entry. That works unless the leader crashes or loses communication with too many others; in such a case Raft elects a new leader. The election process is designed to guarantee that any newly elected leader will have (at least) all of the already-committed entries.

We also have a treat in the form of a great roundtable discussion of the topic via a Think Distributed hangout, featuring several folks from Basho, Peter Bailis, and Diego Ongaro. 

Perhaps the most interesting part of the talk came late in the discussion when Peter commented that he was astounded that an academic paper already has so many open source implementations. RAFT already has 40 or so different implementations in many different languages. 

The key that others can learn from is: understandability. Most academic papers are opaque, to put it generously. Diego talks about this saying:

Click to read more ...

Sponsored Post: BlueStripe, Apple, Surge, Change, Booking, Rackspace, aiCache, Aerospike, ScaleOut, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• We help Change the world every day. It’s really that simple. There aren’t a lot of technology jobs out there where your work will make it possible for a 5 year old hear or change the Boy Scouts' policy on LGBT teens. http://www.change.org/hiring/21, http://www.change.org/hiring/178


• We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.


• Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here.
  • Siri Software Engineer. Play a part in the next revolution in human-computer interaction. Contribute to a product that is redefining mobile computing. Create groundbreaking technology for large scale systems, spoken language, big data, and artificial intelligence. And work with the people who created the intelligent assistant that helps millions of people get things done — just by asking. To apply please visit This URL.
  • Siri Software Engineer. Play a part in the next revolution in human-computer interaction. Contribute to a product that is redefining mobile computing. Create groundbreaking technology for large scale systems, spoken language, big data, and artificial intelligence. And work with the people who created the intelligent assistant that helps millions of people get things done — just by asking. To apply please visit This URL.
  • Siri Software Engineer. Play a part in the next revolution in human-computer interaction. Contribute to a product that is redefining mobile computing. Create groundbreaking technology for large scale systems, spoken language, big data, and artificial intelligence. And work with the people who created the intelligent assistant that helps millions of people get things done — just by asking. To apply please visit This URL.
  • Siri Software Engineer. Play a part in the next revolution in human-computer interaction. Contribute to a product that is redefining mobile computing. Create groundbreaking technology for large scale systems, spoken language, big data, and artificial intelligence. And work with the people who created the intelligent assistant that helps millions of people get things done — just by asking. To apply please visit This URL.
  • Software Engineer - Messaging Services. An exciting opportunity for a Software Engineer to join Apple's Messaging Services team. We build the cloud systems that power some of the busiest applications in the world. You'll have the opportunity to explore a wide range of technologies, developing the server software that is driving the future of messaging and mobile services. To apply please visit this URL. 
  • Software Engineer - Messaging Services. An exciting opportunity for a Software Engineer to join Apple's Messaging Services team. We build the cloud systems that power some of the busiest applications in the world. You'll have the opportunity to explore a wide range of technologies, developing the server software that is driving the future of messaging and mobile services. To apply please visit this URL. 
  • iCloud Documents Server Engineer - C++. The iCloud team is looking for a C++ engineer with a strong background in web services development. The successful candidate has demonstrated deep experience in building high-performing systems that are scalable and extensible. This is a terrific position for an engineer interested in developing the next generation of cloud support for iOS and OS X. To apply please visit this URL.
  • Sr Software Engineer-iCloud. An exciting opportunity for a Software Engineer to join Apple's Messaging Services team. We build the cloud systems that power some of the busiest applications in the world, including iMessage, FaceTime and Apple Push Notifications. To apply please visit this URL.
  • SW Engineering Apps Manager. Be a founding member of Apple's newly-minted iCloud Infrastructure Engineering Team! We are designing, building, and supporting new, critical infrastructural systems and frameworks which provide services like structured and unstructured storage, request routing, search queueing, security, and much more. These form the platform upon which many iCloud backend systems will be built. To apply please visit this URL.
  • iCloud Software Engineer. Explore the far reaches of the possible by joining the team building the future of cloud services at Apple!  Consider joining a small team writing the software which forms the foundation for some of our most exciting iCloud products and services. To apply please visit this URL.
  • Sr. Software Engineer - iCloud. iCloud is looking for a talented software engineer who can help us make iCloud even better. Do you love designing & architecting highly scalable, distributed web services? Does the idea of performance tuning Java applications make your heart leap? To apply please visit this URL.
  • Software Engineer - iCloud. iCloud is looking for a talented software engineer who can help us make iCloud even better. Do you love designing & architecting highly scalable, distributed web services? Does the idea of performance tuning Java applications make your heart leap? To apply please visit this URL.
  • Sr. Software Engineer. The Messaging Services team is looking for a talented software engineer to help us develop Apple's messaging platform. In this role, you'll build the server stacks for iMessage, FaceTime, Apple Push Notifications, and other systems. To apply please visit this URL.
  • Sr Software Engineer-Messaging Services. The Messaging Services team is looking for a talented software engineer to help us develop Apple's messaging platform. In this role, you'll build the server stacks for iMessage, FaceTime, Apple Push Notifications, and other systems. To apply please visit this URL.


• LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online. 


• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• Surge - The Scalability & Performance Conference, presented by OmniTI, Sept. 12th-13th, features speakers from Joyent, Fastly, Dyn, Netflix, Linkedin and Amazon. Special, High Scalability Reader Rate: $50 off registration--through Sept. 10! Book hotel and get $50 off, from OmniTI. 

Cool Products and Services

• BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.


• AppDynamics is an easy-to-use application performance management solution that offers code-level insight into Java, .NET and PHP applications. Get the free trial.


• The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.


• aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required . http://aicache.com/


• ScaleOut Software. In-Memory Data Grids for the Enterprise. Download a Free Trial.


• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.


• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.


• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.


• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

Sponsored Post: Apple, Surge, Change, Booking, Rackspace, aiCache, Aerospike, ScaleOut, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• We help Change the world every day. It’s really that simple. There aren’t a lot of technology jobs out there where your work will make it possible for a 5 year old hear or change the Boy Scouts' policy on LGBT teens. http://www.change.org/hiring/21, http://www.change.org/hiring/178


• We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.


• Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here.
  • Software Engineer - Messaging Services. An exciting opportunity for a Software Engineer to join Apple's Messaging Services team. We build the cloud systems that power some of the busiest applications in the world. You'll have the opportunity to explore a wide range of technologies, developing the server software that is driving the future of messaging and mobile services. To apply please visit this URL. 
  • iCloud Documents Server Engineer - C++. The iCloud team is looking for a C++ engineer with a strong background in web services development. The successful candidate has demonstrated deep experience in building high-performing systems that are scalable and extensible. This is a terrific position for an engineer interested in developing the next generation of cloud support for iOS and OS X. To apply please visit this URL.
  • Sr Software Engineer-iCloud. An exciting opportunity for a Software Engineer to join Apple's Messaging Services team. We build the cloud systems that power some of the busiest applications in the world, including iMessage, FaceTime and Apple Push Notifications. To apply please visit this URL.
  • SW Engineering Apps Manager. Be a founding member of Apple's newly-minted iCloud Infrastructure Engineering Team! We are designing, building, and supporting new, critical infrastructural systems and frameworks which provide services like structured and unstructured storage, request routing, search queueing, security, and much more. These form the platform upon which many iCloud backend systems will be built. To apply please visit this URL.
  • iCloud Software Engineer. Explore the far reaches of the possible by joining the team building the future of cloud services at Apple!  Consider joining a small team writing the software which forms the foundation for some of our most exciting iCloud products and services. To apply please visit this URL.
  • Sr. Software Engineer - iCloud. iCloud is looking for a talented software engineer who can help us make iCloud even better. Do you love designing & architecting highly scalable, distributed web services? Does the idea of performance tuning Java applications make your heart leap? To apply please visit this URL.
  • Software Engineer - iCloud. iCloud is looking for a talented software engineer who can help us make iCloud even better. Do you love designing & architecting highly scalable, distributed web services? Does the idea of performance tuning Java applications make your heart leap? To apply please visit this URL.
  • Sr. Software Engineer. The Messaging Services team is looking for a talented software engineer to help us develop Apple's messaging platform. In this role, you'll build the server stacks for iMessage, FaceTime, Apple Push Notifications, and other systems. To apply please visit this URL.
  • Sr Software Engineer-Messaging Services. The Messaging Services team is looking for a talented software engineer to help us develop Apple's messaging platform. In this role, you'll build the server stacks for iMessage, FaceTime, Apple Push Notifications, and other systems. To apply please visit this URL.


• LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online. 


• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• Surge - The Scalability & Performance Conference, presented by OmniTI, Sept. 12th-13th, features speakers from Joyent, Fastly, Dyn, Netflix, Linkedin and Amazon. Special, High Scalability Reader Rate: $50 off registration--through Sept. 10! Book hotel and get $50 off, from OmniTI. 

Cool Products and Services

• AppDynamics is an easy-to-use application performance management solution that offers code-level insight into Java, .NET and PHP applications. Get the free trial.


• Install Aerospike Community Edition in 10 Minutes! To help you get up and running with our Community Edition as quickly as possible, we've created a tutorial video on AerospikeTV that provides guidance on installing your free database. Download our FREE Database and see how easy it is to install!


• The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.


• aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required . http://aicache.com/


• New Benchmark shows Aerospike nearly 10x Faster than the Competition. Thumbtack Technology YCSB Benchmark shows Aerospike nearly 10x faster than Cassandra, Couchbase and Mongodb. Read it now!


• ScaleOut Software. In-Memory Data Grids for the Enterprise. Download a Free Trial.


• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.


• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.


• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.


• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...

We're on a Break

Hey, just letting you know I'll be on vacation starting today and I'll be back in a few weeks. I won't be posting anything new, so we'll all have a refreshing break. Feel free to cheat with other blogs and social networks. 

Stuff The Internet Says On Scalability For July 19, 2013

Hey, it's HighScalability time:

(Still not a transporter: Looping at 685 mph)

• 898 exabytes: US storage, 1/3 global total


• Create your own trust nobody point-to-point private cloud. Dan Brown shows how step-by-step in How I Created My Own Personal Cloud Using BitTorrent Sync, Owncloud, and Raspberry Pi. BitTorrent Sync is used to copy large files around. Raspberry Pi is a cheap low power always on device with BitTorrent Sync installed. Owncloud is an open source cloud that provides a web interface for file access files from anywhere.


• This is different. Funding a startup using Airbnb as a source of start-up capital. It beats getting a part-time job and one of your guests might even be a VC.


• This is not different. Old industries clawing and digging in, using the tools of power to beat back competition. Steve Blank details a familiar story in Strangling Innovation: Tesla versus “Rent Seekers”. The thing is nobody really wants to compete, they want to make money, which has a completely different metaphysics.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

How do you create a 100th Monkey software development culture?

Someone reading my now ancient C++ coding standard recommendation for using doxygen to automatically generate documentation from source code, asked a great question:

I've often considered using doxygen, I always ask myself - is this really useful? Would I use it if I were new to a project? Would programmers working on the project use it?

I'll rephrase their question to more conveniently express a point I've thought a lot about: Why do companies put so little effort into automating their own development process to make development easier?

It's like the hair stylist whose own hair looks like someone cut it using a late night infomercial vacuum cleaner attachment. Or it's like the interior decorator whose own house looks like a monk's cell.

Software organizations rarely build software to make developing software easier. Why is that?

Because there are three ways changes are made in an organization:

Click to read more ...

Steve Ballmer Says Microsoft has Over 1 Million Servers - What Does that Really Mean?

James Hamilton in Counting Servers is Hard has an awesome breakdown of what one million plus servers really means in terms of resource usage. The summary from his calculations are eye popping:

• Facilities: 15 to 30 large datacenters
• Capital expense: $4.25 Billion
• Total power: 300MW
• Power Consumption: 2.6TWh annually

The power consumption is about the same as used by Nicaragua and the capital cost is about a third of what Americans spent on video games in 2012. Now that's web scale. 

Ask HS: What's Wrong with Twitter, Why Isn't One Machine Enough?

Can anyone convincingly explain why properties sporting traffic statistics that may seem in-line with with the capabilities of a single big-iron machine need so many machines in their architecture?

This is a common reaction to architecture profiles on High Scalability: I could do all that on a few machines so they must be doing something really stupid. 

Lo and behold this same reaction also occurred to the article The Architecture Twitter Uses to Deal with 150M Active Users. On Hacker News papsosouid voiced what a lot of people may have been thinking:

I really question the current trend of creating big, complex, fragile architectures to "be able to scale". These numbers are a great example of why, the entire thing could run on a single server, in a very straight forward setup. When you are creating a cluster for scalability, and it has less CPU, RAM and IO than a single server, what are you gaining? They are only doing 6k writes a second for crying out loud.

This is a surprisingly hard reaction to counter convincingly, but nostrademons has a triple great response:

They create big, complex, fragile architectures because they started with simple, off-the-shelf architectures that completely fell over at scale.

I dunno how long you've been on HN, but around 2007-2008 there were a bunch of HighScalability articles about Twitter's architecture. Back then it was a pretty standard Rails app where when a Tweet came in, it would do an insert into a (replicated) MySQL database, then at read time it would look up your followers (which I think was cached in memcached) and issue a SELECT for each of their recent tweets (possibly also with some caching). Twitter was down about half the time with the Fail Whale, and there was continuous armchair architects about "Why can't they just do this simple solution and fix it?" The simple solution most often proposed was write-time fanout, basically what this article describes.

Do the math on what a single-server Twitter would require. 150M active users * 800 tweets saved/user * 300 bytes for a tweet = 36T of tweet data. Then you have 300K QPS for timelines, and let's estimate the average user follows 100 people. Say that you represent a user as a pointer to their tweet queue. So when a pageview comes in, you do 100 random-access reads. It's 100 ns per read, you're doing 300K * 100 = 30M reads, and so already you're falling behind by a factor of 3:1. And that's without any computation spent on business logic, generating HTML, sending SMSes, pushing to the firehose, archiving tweets, preventing DOSses, logging, mixing in sponsored tweets, or any of the other activities that Twitter does.

(BTW, estimation interview questions like "How many gas stations are in the U.S?" are routinely mocked on HN, but this comment is a great example why they're important. I just spent 15 minutes taking some numbers from an article and then making reasonable-but-generous estimates of numbers I don't know, to show that a proposed architectural solution won't work. That's opposed to maybe 15 man-months building it. That sort of problem shows up all the time in actual software engineering.)

And the thread goes on with a lot of enlightening details. (Just as an aside, in an interview the question "How many gas stations are in the US" is worse than useless. If someone asked for a Twitter back-of-the-napkin analysis like nostrademons produced, now we are getting somewhere.)

Do you have an answer? Are these kind of architectures evidence of incompetence or is there a method to the madness?

Stuff The Internet Says On Scalability For July 12, 2013

Hey, it's HighScalability time:

(The Shard Scaling 6)

• 3500+: number of species found underneath miles of ice; 360 TB/disc data capacity: 5D optical memory in glass; 50% Of Internet Traffic: Comes From Only 35 Sites/Services; 60 billion: potential alien planets that could give Snowden asylum
• Quotable Quotes:
  • @ieure: "How concert app Loudie doubled its users in two weeks" The other co-founder finally signed in.
  • @michalbe: "Daddy, how is software made?" "Well, when a programmer loves an idea very much they stay up all night and then push to github the next day"
  • @TomDSweeney: Alan Kay: “The past 30 years have been completely mundane. It’s all been scaling (of old technology) and Angry Birds”
  • Rick Branson: Adopt a technology by understanding what it's best at and letting it do that first, then expand…
  • K. Eric Drexler: System-level design seeks flexible models and accepts complexity, accepts imprecision and buries it in margins of safety, recognizes unpredictability and seeks to avoid it, embraces competing options and seeks to add more, all in a world in which questions have a range of satisfactory answers and testing can establish a narrow yet valuable truth: “This one works.”


• In a recent Twitter Architecture it was observed how "Much can be implied by the implicit social contract when bidirectional follows don’t exist." To see this idea in action here are some very cool influencer maps. Identified are 6 types of social networks that are an overlay on Twitter's follower model: polarized networks, in-group networks, brand/public topic networks, bazaar networks, broadcast networks, and support networks.


• OK, this is funny: Nikola Tesla Pitching Silicon Valley VCs. It's a vision thing.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Click to read more ...

Sponsored Post: NoSQL Now!, Booking, Apple, Two Sigma, RAMP, Blurocket, Incapsula, Surge, Rackspace, aiCache, Aerospike, ScaleOut, New Relic, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?

• We need awesome people @ Booking.com - We want YOU! Come design next generation interfaces, solve critical scalability problems, and hack on one of the largest Perl codebases. Please apply online.


• An exciting opportunity for a Software Engineer to join Apple's Messaging Services team. We build the cloud systems that power some of the busiest applications in the world. You'll have the opportunity to explore a wide range of technologies, developing the server software that is driving the future of messaging and mobile services. To apply please visit this URL. 


• Two Sigma is building our next generation research environment, and we're looking for a functional programmer with a passion for distributed computing. We're scaling machine learning and operations research to tens of thousands of CPUs. Please send qualifications to buildstuff@twosigma.com.


• Blurocket is looking for smart and fun people to build its next generation ecommerce platform.  If creating scalable services is in your DNA, let us know! (Salary $250k+). Apply over at StackOverflow.


• LogicMonitor is looking for a Front End developer to have a huge impact, be valued, realize their dreams, and help us realize ours. We are looking for someone to own the code that delivers the design and usability of LogicMonitor's enterprise SaaS application(s). Please apply online. 


• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• The NoSQL Now! Conference is the largest vendor-neutral forum on NoSQL technologies, and teaches how to build faster, more scalable and more agile data-centric solutions. Earlybird discounts are available until July 19. Free Big Data + NoSQL Summer School is also available to qualified customers. Learn more at www.NoSQLNow.com.


• Surge - The Scalability & Performance Conference, presented by OmniTI, Sept. 12th-13th, features speakers from Joyent, Fastly, Dyn, Netflix, Linkedin and Amazon. Special, High Scalability Reader Rate: $50 off registration--through Sept. 10! Book hotel and get $50 off, from OmniTI. FREE Puppet training w/registration, Sept 11; space is limited.


• Visit Budapest this July and learn from those who have been in the trenches of building highly scalable systems at RAMP.

Cool Products and Services

• AppDynamics is an easy-to-use application performance management solution that offers code-level insight into Java, .NET and PHP applications. Get the free trial.


• With a 5-minute setup, Incapsula provides any website with protection against all known and emerging threats (Including spam, SQL Injections, DDoS and more), and acceleration through a global CDN, caching and optimization. See the power of Incapsula on your website with our Free trial.


• Install Aerospike Community Edition in 10 Minutes! To help you get up and running with our Community Edition as quickly as possible, we've created a tutorial video on AerospikeTV that provides guidance on installing your free database. Download our FREE Database and see how easy it is to install!


• The Rackspace Cloud Application Programming Interface (API)  has changed the game allowing customers to easily modify their cloud configuration with just a few lines of code. The API is a powerful tool and something everyone should know about, regardless of your level of technical ability.


• aiScaler, aiProtect, aiMobile integrated solutions for Dynamic Site Acceleration, Denial of Service Protection and Simplifying Mobile Content. Free instant trial, no sign-up required . http://aicache.com/


• New Benchmark shows Aerospike nearly 10x Faster than the Competition. Thumbtack Technology YCSB Benchmark shows Aerospike nearly 10x faster than Cassandra, Couchbase and Mongodb. Read it now!


• ScaleOut Software. In-Memory Data Grids for the Enterprise. Download a Free Trial.


• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.


• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.


• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.


• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Click to read more ...