Let's Welcome our Neo-Feudal Overlords
Monday, January 25, 2010 at 8:23AM 
This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud.
There's a pattern, already begun, that has accelerated by the need for applications to scale and increase complexity, the end result of which will be that applications give up their independence and enter a kind of feudal relationship with their platform provider.
To understand how this process works, like a glacier slowly and inevitably carving out a deep river valley, here's the type of question I get quite a lot:
I've learned PHP and MySQL and I've built a web app that I HOPE will receive traffic comparable to eBay's with a similar database structure. I see all these different opinions and different techniques and languages being recommended and it's so confusing. All I want is perhaps one book or one website that focuses on PHP and MySQL and building a large database web app like eBay's. Does something like this exist?
I'm always at a loss for words with these questions. What can I possibly say? The gulf between a LAMP application and eBay is so huge that there's no way to even start explaining it. Yet, what they want is reasonable. They just want to make a site that scales to meet their dreams. This barrier will be reached more and more as people try to make a living out on the digital frontier. They won't be able to do it on their own. So I'm forced to recommend that they find the protection of a King and take a look at Google App Engine, EC2, or one of the other platforms. There's no practical alternative at the moment.
The King Owns the Level 3 Platform You Need
What people will be forced to find is what Marc Andreessen calls a Level 3 Platform. A platform is a system that can be programmed and therefore customized by outside developers. A Level 1 Platform is the API. Example: Twitter. Applications using Twitter's API are completely separate from Twitter yet use Twitter's data. A Level 2 Platfrom is a Plug-in API. Example: Facebook. Facebook allows applications to embed themselves into Facebook's UI, but the apps are still separate from Facebook. A Level 3 Platform is a Runtime Environment. Example: Salesforce, Google App Engine, Amazon, Azure, and Heroku. This next step is where application merge into the platform. Developer application code is uploaded and actually runs inside the platform.
As a programmer, leveraging a Level 3 Platform makes perfect sense. As the need to scale becomes greater and greater it becomes harder and harder to do, so we look for a little help from the very few companies that will have the infrastructure to make it happen. Quite understandable. When we can't do something, we naturally seek out a partner. We outsource it to those who promise to do it for us. Applications have started to become mashups where the hard parts are written in terms of a platform that does all those hard parts.
The relationship here is asymmetrical. Once adopted, a platform becomes hard to leave which adds a political dimension to the evolution of future application architectures. This result may not be readily apparent because we are still so close to the era of small disconnected systems: application writers are becoming 'vassals' to their feudal platform 'lords', a kind of neo-feudalism that bares a striking relationship to the previous agriculture based feudalism.
This somewhat imprecise feudal analogy goes like this: the platform provider (the King) provides services, land, and protection to their platform users (vassals). The vassals pledge to use the King's resources to produce money (crops) using computers (serfs). In exchange, the King gets a slice of the production.
This sounds all melodramatic, but I'm just using feudalism as metaphor to make clear how the power relationships are aligned. As a developer you won't have much power. Not only are you bound by the platform and mashup APIs used to write applications, but you are also bound by a need to pick a partner in order to planet-scale.
I'm not even implying there is anything nefarious or even necessarily conscious about this process. It's simply a natural outgrowth of the forces of scale, complexity, and danger. While I don't think there's an evil intention involved, we should still be aware of where we might unthinkingly end up.
Datacenters are the New Land
In the feudal era having land meant everything. Only land owners could grow crops for food, for trade, and for money. Land was wealth. When the feudal era collapsed industry and trade became the new means for acquiring wealth. The nobility became land rich and money poor. It was more profitable to own shares in the East India Company than it was to be born an Earl. A radical transformation for culture that had lasted hundreds of years.
To stretch the analogy further, in terms of building applications we have been in the hunter gather era, we are entering the feudal era of great land owners, and we are contemplating what might constitute the next industrial revolution. Datacenters have become the equivalent of land, castles, and the new centers of wealth.
This is why we see a consistent stream of new datacenters being built. If you want to farm you need land. If you want to run code you need compute resources. Land in the datacenter is made from compute resources: SANs, CPUs, switches, memory, networks, and so on. It's not just about hardware though, it's also about services like: elasticity, monitoring, DNS, failover, security, queuing, email, database, storage, geographically disperse operations, backups, documents, searching, and so on. Perfect soil in which to plant, tend, and grow applications.
Datacenters are the New Castles
The fact that the King gets a slice of production is obvious. But one of the key duties of the King in the feudal system is to take up the mantle of protector. It may not seem obvious how datacenter owners become our protector, but as attacks escalate this may become one the most coveted services of all. Datacenters are like castles spread out across the landscape, offering safety and control of their domain.
The digital world is vicious. A DDoS attack can take down even the largest of companies and no small land holder can even begin to defend themselves. But if you enter Google's domain Google will protect you. Google can stop a DDoS attack. They have the technology and the people to make the problem go away. You don't even have to think about it.
Here's my personal story of woe as an illustration. One day my email stopped. The inbox was as empty as a cubicle farm on Thanksgiving. Another day went by, no email. Then another day went by, still no email! Figuring I couldn't have become that unpopular, I contacted my email provider.
They informed me my email was shut down because there was a DDoS attack on my domain. Damn! Who would do such a thing and why? Their standard response is to wait it out. Attackers usually give up soon.
So we waited. We waited six days. Six days without mail and without a letup on the attackers. Persistent little black hats. It seems the resources dedicated to their assault were so inconsequential they that just left the attack going.
For my email provider that attack was anything but inconsequential. They kept my email turned off to protect their system. They simply didn't have the expertise, the equipment, or the resources to deal with these attacks.
It's a little like homesteaders on the frontier being attacked by marauders. On the frontier of the US when homesteaders came under attack and could no longer protect themselves, they had to move to the nearest fort for protection.
That's exactly what I did. I abandoned my log cabin and moved into fort Google for the duration. I made Google the email provider for my domain. As soon as I made the move email started flowing again. Yay, my digital corpse has been resuscitated! Google was not cowed by these attackers. Google took the attack, ate it up, and spit it back out without even blinking. My savior!
Meanwhile back at the frontier my previous email provider said the attack stopped after 10 days and I that could go back. Well, I didn't go back. Why would I? I need protection from a big strong Google with all the hardware and processes and experience. I needed the protection of the King and his castle. It really, really, really kills me to have to say that, but that's what a King can do for you.
The castle walls in the datacenter aren't made of stone, though there is usually a hardened security perimeter, they are more in the form of security appliances, DDoS monitoring systems, SSL appliances, deep packet inspection appliances, and constant operational vigilance. Soldiers are always walking the walls and guarding the gates.
Security isn't the only area that requires protection. How about massive power failures, disk failures and server failures? These are all areas where a large, complex, well tended infrastructure, like in a datacenter, provides protection.
As the world becomes more connected and dangers become more pronounced, these trends will only accelerate. In good times you can homestead in the wilderness, but when the bandits attack and you can no longer bring your goods to market, you seek the protection of the King.
The King has Subtle Forms of Persuasion
Originally I considered the driving forces of scale and complexity as sufficient to compel developers to end up in the arms of platform provider. It's a very organic and understandable process.
Lately there have been some darker ideas explaining why there will be a convergence at the datacenter. First is Google Makes A Bid To Control The Internet by Rob Diana. Second is Tim O'Reilly's The War for the Web. What are they afraid of?
Rob Diana's hints at Google's power to drive technology selections by pounding its big hammer: pagerank. The logic of his post goes:
- Google is considering penalizing the pagerank of sites that aren't fast enough,
- Google's search is the market leader, so Google is he who can not be ignored.
- Google is making a series of initiatives (SPDY, Chrome, GO, DNS, URL Shortener) to increase speed.
- The only plausible way a majority of sites to comply with the new speed requirements will be to adopt Google's stack.
- Google's stack becomes the Internet for a large portion of users.
For my thesis it doesn't really matter if this process is part of some master plan or is derived from well meaning steps independently taken. By linking pagerank to performance and providing all the intermediary steps necessary to become performant, the result is likely to drive developers to a platform for refuge. Smaller hosts and less performant stacks (like LAMP) will be difficult to justify when pagerank, and thus revenues, are dependent on speed. Google is certainly not the only platform option, but people will go to a platform people as an easy prepackaged solution to a tough to solve problem.
Tim O'Reilly is his article talks about how natural monopolies have formed on the web and how companies may use monopoly power to reach into other areas:
It could be that everyone will figure out how to play nicely with each other, and we'll see a continuation of the interoperable web model we've enjoyed for the past two decades. But I'm betting that things are going to get ugly. We're heading into a war for control of the web. And in the end, it's more than that, it's a war against the web as an interoperable platform. Instead, we're facing the prospect of Facebook as the platform, Apple as the platform, Google as the platform, Amazon as the platform, where big companies slug it out until one is king of the hill. And it's time for developers to take a stand. If you don't want a repeat of the PC era, place your bets now on open systems. Don't wait till it's too late.
The point here is as one commenter says, certain websites have such a strong gravitational field they suck in everything around them and get ever larger. Exciting capabilities like maps are a big platform adoption driver, but I think the biggest baddest black hole of them all will end up needing to planet-scale.
An article by ZDNet's Dion Hinchcliffe, Cloud computing and the return of the platform wars, seems to back up Tim's fears. Hinchcliffe thinks there will be a protracted platform war with a fairly predictable and oft-repeated cycle of events for which a small number of large winners are likely to emerge victorious. He has a cool diagram showing the process: 
Hinchcliffe observes the results of this war will end in a reversal of recent trends: The world of software has recently, at least up until now, been moving slowly and steadily towards an increasingly commoditized, virtualized, and open sourced future. Cloud computing in its present form does appear to herald a return to the classical days of big vendor computing and all the baggage (good and bad) that it implies along with some unique twists of its own. Exactly the result Tim warns about.
How this cycle becomes important is summarized by the idea of path dependence: the degree to which initial design choices condition later technological development. This is very real effect, otherwise we wouldn't still be using the deficient by design QWERTY keyboard. Once initial platform choices have been made, the shape of the future is largely determined, regardless of the quality of those early decisions or the quality of later alternatives.
I'm Not a Number, I'm a Free Man
The GooglePlex isn't the only King out there. Others are trying. We have FacebookPlex, AmazonPlex, MicrosoftPlex, and there will be others. As a serf, the key question for me going forward is: how can we create a FreePlex where a free people can carve out a patch of land and make a living?
If you would like to read the rest of the article please take a look at Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud.
Reader Comments (8)
The difference, if I may play with your metaphor a bit, is that stone walls and swords were never as easy to duplicate as information. We peasants might not be able to duplicate the barons' and dukes' resource level, but we can build exactly the same walls and moats and other fortifications that they have nonetheless because those are the form of information we call software. "Data centers as land" is an interesting metaphor worth exploring, but "data centers as castles" only works to the extent that castles traditionally contained the various locations and institutions important to trade - market squares and counting houses, banks and guilds. The castle walls existed to protect those, not the peasants who were often locked outside of the walls to die by the thousands during an attack. The "new feudalism" manifests in the form of creating and maintaining a class of lords, not in the forms that their property might take.
The last point I agree with, it's seeing the pattern through time which can be tricky. That information levels the battle field is harder to see. From my example, compare the ability of a shared hosting service to handle DDOS to Google's ability to handle the same attacks. They are a small group carrying spears and Google can deploy an endless sea of mounted knights . If it was just software then wouldn't they be on the same field? It's software, resources, attention, expertise, and the ability project and maintain a standing army to meet a persistent threat. Systems of castles were build to dominate, manage, and extract value from a region. Datacenters IMHO take the same level of organization and resources for very similar ends.
"linking pagerank to performance"
True. speed is one indication of quality.
My guess, Google runs stress tests, concurrent with measures of the impact.
Pagerank as a measure of resistance to fatigue.
I like the King analogy. But putting Google and Amazon in the same bin for "lock-in" is wrong. Google gives a level 3 platform, but Amazon (if you just use EC2) gives you an infrastructure.
But with Amazon, 99% of your application will be just your application. Even if you use fancy features like RDS, SQS and Load balancing, you are really outsourcing parts of your application. Yes, this creates lock-in, but it's self-imposed. You use them because they save time. If you're worried about lock-in, you write your app not to use them (or don't use them in the first place).
With Google App engine, you can only run in Google. (un-scalable open-source clones not withstanding). No amount of software changes will make it easy to migrate away.
Anonymouse, I agree completely that your app on Amazon will be primarily your architecture, the issue though is to scale your application must run on an Amazon type platform and that's where you are locked-in at some fundamental existential level. That's the lock-in that must be broken.
Just build the LAMP app, and worry about scaling later. If you have enough demand to need to scale to eBay's size, the odds are you won't be short on cash at that time.
In the meanwhile, there are certainly intermediate steps; caching comes to mind.
Dean in the full article I'm projecting out to the time when we have true world wide applications that involves a significant portion of the human population plus the interaction with billions of sensors and smart devices. In our future world most everyone will be eBay by default. Look at BuddyPoke today, 65 millions installs, and that's with relatively few users. Simply LAMPing it won't work.
This article sounds like a preach. As already was said write an app that gets traction then search for a solution to scale it. BuddyPoke is one of tens of thousands. Probably majority of the rest can run from a home server with DSL connection.
BTW, Google, Facebook, Yahoo and many others started with LAMP (or parts of it) and still use it.