75% Chance of Scale - Leveraging the New Scaleogenic Environment for Growth

"I'll never need to scale so why bother? We aren't Twitter or Facebook or Google after all." This is the most common email I get, a question in the form of a thinly disguised rationalization for not having to worry about scaling. And in these days of giant transformer-like machines they are probably right. But what if there are Barry Bonds enhancing type forces at work that argue for the chances of your needing to scale being higher than you think?

And if that happens, how will you cross the scalability chasm? Will you want to completely change your architecture or evolve it from a tool-chain that was meant to scale from the start? Architecturally, that's the question you have to answer. Today's tool-chains are making it possible to grow a system from small to large without needing to implement complete architectural phase changes at various scale inflection points, but that's a different topic. We're trying to think about why you may actually need to scale, that is the question.

Tumblr is a good example of a product that grew beyond expectation because they managed both to execute and harness powerful growth factors. Tumblr is a "light" blogging service that probably didn't think they were Twitter or Facebook or Google either, but need to scale they did. From Tumblr:

Frankly, keeping up with growth has presented more work than our small team was prepared for — with traffic now climbing more than 500M pageviews each month. But we are determined and focused on bringing our infrastructure well ahead of capacity as quickly as possible. We’ve nearly quadrupled our engineering team this month alone, and continue to distribute and enhance our architecture to be more resilient to failures like today’s.

500 million pageviews a month is an amazing accomplishment. And that was just the start. Techchrunch reported Tumblr, as of July 2010, had over 6 million users and 1.5 billion pageviews a month, and is now growing by 300 million pageviews per month and 25,000 new users a day. Really stunning growth. And seemingly quite unexpected.

Why did Tumblr grow so fast? An article in ReadWriteWeb identifies several factors: 

1. First mover advantage. Tumblr invented an entirely new market segment. Just as often this doesn't bring success, but it can.
2. Social. Celebrities and big media adopted Tumblr and Tumblr had better social networking features than their competition. Fred Wilson shares an experience with his daughter when she came home from college on thursday night and showed me all of her friend's Tumblrs. A Tumblr is self expression. Jessica's looks different than Emily's, mine and the Gotham Gal's. That's powerful. And that is what I think is driving Tumblr's popularity. Self expression matters.
3. Performance. One commentor and this Yotta article both pointed to Tumblr's superior performance, being twice as fast as the competition, as a driver for their success.

When Tumblr started in 2007 was there any reason to expect it to take off? I don't think so. My impression of Tumblr at the time was that it did stuff other services already did, which is exactly why I don't make predictions! Tumblr took off by executing and harnessing drivers that lead to fast growth. 

Could this happen to you too? We usually don't think so, but applications are being built today in a scaleogenic environment that tends to encourage growth. Scaleogenic is a play on the word obesogenic, which is a term for factors tending to make individuals fat, creating an environment that promotes gaining weight.  We have a genetic propensity towards obesity as a survival mechanism, but that's just part of the story, the larger part is how our environment has changed to be one where cheap, high-caloric food and is always available and we need very little exercise to live. Combined, these trends equal a Biggest Loser class obesity problem.

While not quite as powerful a force, we are seeing applications being deployed inside a scaleogenic environment, where a combination of factors can make an a application need to scale and scale quickly. Part of this environment has only recently been created in the form of app stores, social networks, and the explosion of mobile devices. Another part of this environment is quite self-consciously constructed, it has to do with applications using viral loops, gameification, social networking, and other techniques to create and leverage rapid growth factors. 

Do any of these factors mean you will need to scale? No. But they do increase the chances and increased chances means taking this part of your architecture seriously could pay off. Let's take a look at some of the components of our new scaleogenic environment in more detail.

The Scaleogenic Environment

• Social. Social changes everything. Zynga and Playfish needed to scale to hundreds of millions of users quickly because they figured out how to harness the power of social networks in their game design. We are seeing this growth strategy used by nearly all categories of software now. 
• Mobile. Paul Scott's tweet neatly summarizes the impact of mobile: Holy crap, so when I launch the mobile client for this thing, they expect 100x more users to join. Best I get a scalability plan together.
• App stores. Distribution has totally changed. Your app can be placed immediately in-front of millions of people. The key to sales is to remove barriers to purchasing. Make it as easy as possible. App stores have done for buying software what credit cards did for hard goods. Evernote's experience releasing on the Mac app store offers another good case study for both the power of mobile and the power of app stores. Over 70% of Evernote’s new users came from mobile app stores in the last year.  In the first week of the Mac App Store bout 320,000 people downloaded Evernote of which 120,000 were new customers. I remember shlepping shrink wrapped software around from conference to conference. You didn't have to worry about fast growth then, it was simply too rare an occurrence. Now it's not.
• Fame. One of my most read posts had absolutely nothing to do with me. I offhandedly used "Kevin Rose" in a title and they came. It took me a while to figure out why. Deals are now being done with celebrities to tweet products, become friends, and so on. Make this happen and you too can harness the shallowness of the general public, a quality that seems both infinite in supply and perfectly sustainable. Go green. 
• Population. There are always more people and things being added to the potential user base.
• Crowdsourcing. Letting users "help you" by adding their own data to your system can quickly turn into a shockingly massive load. Flickr with 3,000 photos uploaded every minute. Facebook adds 12 terabytes per day. Twitter adds 7 terabytes a day. 
• Gamification.  Use of game play mechanics for non-game applications. The idea is that by leveraging the same design tactics that have made games so popular, other applications can become more successful with users. StackOverflow is one such example from the deep nerd world, but these ideas are much more general and powerful and if harnessed correctly, could lead to huge levels of adoption.
• Freemium business model. This has become a popular business model, but when you give away the milk a lot of people will suddenly want milkshakes, and that requires the ability to scale quickly at a very low cost.

We are seeing a perfect storm of conditions for creating a scaleogenic environment that requires thinking about building scalable applications from the start:

• Large pools of users are now being aggregated by social networks and mobile devices of all types.
• Ever easier means of application distribution and purchasing through App stores to access those large pools of users.
• Completely new models of application construction, like viral loops and gamification, that drive user adoption and expansion inside those large pools of easier to access users.

This is a bold new world, even when compared to just a few users ago. If you are planning to tap into any of these huge new forces (and why wouldn't you?), then if you can also execute, you will experience an increasing chance of scale.

Related Articles

• As Downtime Mounts, Tumblr Adds Data Center by Rich Miller.
• What Should I Do? Choosing SQL, NoSQL or Both for Scalable Web Applications 
• Fun is the Future: Mastering Gamification by Gabe Zichermann.
• The Implications Of Punctuated Scalabilium For Website Architecture
• Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
• Why Are Facebook, Digg, And Twitter So Hard To Scale?