Sponsored Post: Zoosk, Aerospike, Server Stack, Wiredrive, NY Times, CouchConf, FiftyThree, Percona, ElasticHosts, ScaleOut, New Relic, NetDNA, GigaSpaces, AiCache, Logic Monitor, AppDynamics, CloudSigma

Who's Hiring?

• Zoosk is looking for a Data Center Operations Manager to scale our rapidly growing business. Are you ready to push the boundaries of technology driving data center operations for a top consumer web product? Join our team www.zoosk.com/careers. 
• Wiredrive is looking for a SENIOR WEB APPLICATION SYSTEMS ADMINISTRATOR and a TEST AUTOMATION ENGINEER to join our agile infrustructure team. For full job descriptions please see http://wdrv.it/QA6iTw
• The New York Times is seeking a developer focused on infrastructure to join its newsroom development team. Read the full description here and send resumes to chadas@nytimes.com.
• FiftyThree, the company behind the award-winning iPad app Paper, is looking for a {Backend || DevOps} Engineer to help us build our next great product: a service to "bring ideas together". http://www.fiftythree.com/jobs
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• December 3-4, 2012 Percona Live London 2012 Millennium Gloucester Hotel and Conference Centre, London, UK http://www.percona.com/live/london-2012/
• CouchConf is a one-day, three track event is for any developer who wants to take a 
• deeper dive into Couchbase NoSQL technology, 
learn where it’s headed and build really cool stuff.

Cool Products and Services

• Aerospike: Two Trillion Transactions per month...100 million stored user profiles...25% of all video ads processed on the internet - mere realities of success for Aerospike customers. Industry leaders reveal their secrets! 
• ServerStack offers the industry's most scalable managed servers for high traffic/bandwidth websites backed by unlimited 24/7 network, server and application support.
• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. Test aiCache acceleration for free.  No sign-up required. http://aicache.com/deploy
• ElasticHosts launches white-label cloud reseller program offering 30% revenue share on fully rebranded cloud hosting.
• ScaleOut Software. In-memorry Data Grids for the Enterprise. Download a Free Trial.
• Follow the Cloudify blog to learn more about our open source PaaS stack – latest integration recipes, builds, features, and other cool stuff.  Visit the GigaSpaces blog to learn how to take your application to the next level of scalability and performance.
• NetDNA, a Tier-1 GlobalContent Delivery Network, offers a Dual-CDN strategy which allows companies to utilize a redundant infrastructure while leveraging the advantages of multiple CDNs to reduce costs.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Utility style high performance cloud servers in the US and Europe delivered on all 10GigE networking. Run any OS, take advantage of SSD storage and tailored infrastructure options.

For a longer description of each sponsor, please read more below...

Click to read more ...

Gone Fishin': Hilarious Video: Relational Database Vs NoSQL Fanbois

This is an all time favorite post. Even though I've seen this video a hundred times I still can't help but laugh...

This is so funny I laughed until I cried! Definitely NSFW. OMG it's hilarious, but it's also not a bad overview of the issues. Especially loved: You read the latest post on HighScalability.com and think you are a f*cking Google and architect and parrot slogans like Web Scale and Sharding but you have no idea what the f*ck you are talking about. There are so many more gems like that.

Thanks to Alex Popescu for posting this on MongoDB is Web Scale. Whoever made this deserves a Webby.

Gone Fishin': 10 Ways to Take your Site from One to One Million Users by Kevin Rose

This is the post that got me kicked off my original shared hosting service and prompted the move to SquareSpace. I couldn't figure out why so many people were reading this article. But they kept on coming. The site went down and I was told to vamoose.  It finally dawned on me nobody actually cared about the article, it was the name Kevin Rose that was magic. Learned a good lesson about publishing biz...

At the Future of Web Apps conference Kevin Rose (Digg, Pownce, Wefollow) gave a cool presentation on the top 10 down and dirty ways you can grow your web app. He took the questions he's most often asked and turned it into a very informative talk.

This isn't the typical kind of scalability we cover on this site. There aren't any infrastructure and operations tips. But the reason we care about scalability is to support users and Kevin has a lot of good techniques to help your user base bloom.

Here's a summary of the 10 ways to grow your consumer web application:

Click to read more ...

Gone Fishin': Building Super Scalable Systems: Blade Runner Meets Autonomic Computing In The Ambient Cloud

All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see...

"But it is not complicated. [There's] just a lot of it." 
-- Richard Feynman on how the immense variety of the world arises from simple rules.

Contents:

1. Have We Reached the End of Scaling?
2. Applications Become Black Boxes Using Markets to Scale and Control Costs
3. Let's Welcome our Neo-Feudal Overlords
4. The Economic Argument for the Ambient Cloud
5. What Will Kill the Cloud?
6. The Amazing Collective Compute Power of the Ambient Cloud
7. Using the Ambient Cloud as an Application Runtime
8. Applications as Virtual States
9. Conclusion

We have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.

Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.

Tomorrow the numbers foreshadow a new Cambrian explosion of connectivity that will look as different as the image of a bare lifeless earth looks to us today. We will have 10 billion people, we will have trillions of things, and we will have a great multitude of social networks densely interconnecting all these people to people, things to things, and people to things.

How can we possibly build planet scalable systems to handle this massive growth if building much smaller applications currently stresses architectural best practices past breaking? We can't. We aren't anywhere close to building applications at this scale, except for perhaps Google and a few others, and there's no way you and I can reproduce what they are doing. Companies are scrambling to raise hundreds of millions of dollars in order to build even more datacenters. As the world becomes more and more global and more and more connected, handling the load may require building applications 4 or 5 orders of magnitude larger than any current system. The cost for an infrastructure capable of supporting planet-scale applications could be in the 10 trillion dollar range (very roughly estimated at $100 million a data center times 10K).

If you aren't Google, or a very few other companies, how can you possibly compete? For a glimmer of a possible direction that may not require a kingdom's worth of resources, please take a look at this short video:

Click to read more ...

Are we seeing the renaissance of enterprises in the cloud?

A series of recent surveys on the subject seems to indicate that this is indeed the case:

Research conducted by HP found that the majority of businesses in the EMEA region are planning to move their mission-critical apps to the cloud. Of the 940 respondents, 80 percent revealed plans to move mission-critical apps at some point over the next two to five years.

A more recent survey, by research firm MeriTalk and sponsored by VMware and EMC (NYSE:EMC), showed that one-third of respondents say they plan to move some mission-critical applications to the cloud in the next year. Within two years, the IT managers said they will move 26 percent of their mission-critical apps to the cloud, and in five years, they expect 44 percent of their mission-critical apps to run in the cloud.

The Challenge - How to Bring Hundreds of Enterprise Apps to the Cloud

The reality is that cloud economics only start making sense when there are true workloads that utilize the cloud infrastructure.

If the large majority of your apps fall outside of this category, then you’re not going to benefit much from the cloud. In fact, you’re probably going to lose money, rather than save money.

The Current Approach

• Focus on building IaaS - Current cloud strategies of many enterprises has been centered on making the infrastructure cloud ready. This basically means ensuring that they are able to spawn machines more easily than they were before. A quick look at many initiatives of this nature shows that there is still only a small portion of enterprises whose applications run on such new systems.
• Build a new PaaS - PaaS has been taught as the answer to run apps on the cloud. The reality however, is that most of the existing PaaS solutions only cater to new apps and quite often the small, and “non” mission-critical share of our enterprise applications, which still leaves the majority of our enterprise workload outside of our cloud infrastructure.
• App Migration as a One Off Project - The other approach for migrating applications to the cloud has been to select a small group of applications, and then migrate these one by one to the cloud. Quite often the thought behind this approach has been that application migration is a one-off project. The reality is that applications are more of a living organism – things fail, are moved, or need to be added and removed over time. Therefore it’s not enough to move apps to the cloud using some sort of virtualization technique, it’s critical that the way they’re run and maintained will also fit the dynamic nature of the cloud.

Why is This not Going to Work?

Simple math shows that if you apply this model to the rest of your apps, it’s probably going to take years of effort to migrate all your apps to the cloud. The cost of doing so is going to be extremely high, not to mention the time to market issue which can be even an even greater risk in the end, as it will reflect on cost of operation, profit margins and even the ability to survive in this an extremely competitive market, if it is too long.

What's missing?

What we’re missing is a simple and systematic way to brings all these hundreds and thousands of apps to the cloud.

Moving Enterprise Workloads to the Cloud at a Massive Scale

Instead of thinking of cloud migration as a one-off thing, we need to think of cloud migration on a massive scale.

Thinking in such terms drives a fairly different approach.

In this post, I outlined what i believe should be the main principles for moving enterprise application at such a scale.

Read full post: http://www.cloudifysource.org/2012/10/30/moving_enterprise_workloads_to_the_cloud_on_a_massive_scale.html

Cost Analysis: TripAdvisor and Pinterest costs on the AWS cloud

This is a guest post by Ali Khajeh-Hosseini, Technical Lead at PlanForCloud.com.  

I read a recent blog post about TripAdvisor's experiment with AWS where they attempted to process 700K HTTP requests per minute on a replica of their live site. There was also an interesting blog post on Pinterest's massive growth on AWS. These blogs highlighted exactly the types of questions we're interested in, mainly:

1. How much would it cost to deploy System X on Cloud Y? e.g. how much would it cost to host TripAdvisor on the AWS US-East cloud?
2. Would it be cheaper to use deployment option X or Y? e.g. would it be cheaper to use reserved instances, different types of instances, different cloud providers...
3. What happens to costs when the system grows? e.g. Pinterest has around 410TB of data on S3, what if that keeps growing at a rate of 25% every month, like it has been in the last 10 months?

I created a couple of deployments in PlanForCloud to explore these questions and the results are very interesting - they show how important it is to get an idea of costs and budget for cloud spend prior to using the cloud:

Click to read more ...

Gone Fishin': LiveJournal Architecture

This was the first architecture profile on HighScalability. IMHO LiveJournal was really the start of the openness on how to build stuff at scale, setting the whole industry off with an excellent role model. They wrote about their architecture, they open sourced their tools, they showed that success wasn't based on keeping secrets, and they set forth principles still followed by our rather amazing industry. No other industry is so open and cooperative, with their eyes cast so far forward, intent on building cool stuff. When all around seems dark it would be good to keep this little bit of light in mind...

A fascinating and detailed story of how LiveJournal evolved their system to scale. LiveJournal was an early player in the free blog service race and faced issues from quickly adding a large number of users. Blog posts come fast and furious which causes a lot of writes and writes are particularly hard to scale. Understanding how LiveJournal faced their scaling problems will help any aspiring website builder.

Site: http://www.livejournal.com/

Information Sources

Click to read more ...

Sponsored Post: Zoosk, Aerospike, Server Stack, Wiredrive, NY Times, CouchConf, FiftyThree, Percona, ElasticHosts, ScaleOut, New Relic, NetDNA, GigaSpaces, AiCache, Logic Monitor, AppDynamics, CloudSigma

Who's Hiring?

• Wiredrive is looking for a SENIOR WEB APPLICATION SYSTEMS ADMINISTRATOR and a TEST AUTOMATION ENGINEER to join our agile infrustructure team. For full job descriptions please see http://wdrv.it/QA6iTw
• The New York Times is seeking a developer focused on infrastructure to join its newsroom development team. Read the full description here and send resumes to chadas@nytimes.com.
• FiftyThree, the company behind the award-winning iPad app Paper, is looking for a {Backend || DevOps} Engineer to help us build our next great product: a service to "bring ideas together". http://www.fiftythree.com/jobs
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• December 3-4, 2012 Percona Live London 2012 Millennium Gloucester Hotel and Conference Centre, London, UK http://www.percona.com/live/london-2012/
• CouchConf is a one-day, three track event is for any developer who wants to take a 
• deeper dive into Couchbase NoSQL technology, 
learn where it’s headed and build really cool stuff.

Cool Products and Services

• Aerospike: Two Trillion Transactions per month...100 million stored user profiles...25% of all video ads processed on the internet - mere realities of success for Aerospike customers. Industry leaders reveal their secrets! 
• ServerStack offers the industry's most scalable managed servers for high traffic/bandwidth websites backed by unlimited 24/7 network, server and application support.
• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. Test aiCache acceleration for free.  No sign-up required. http://aicache.com/deploy
• ElasticHosts launches white-label cloud reseller program offering 30% revenue share on fully rebranded cloud hosting.
• ScaleOut Software. In-memorry Data Grids for the Enterprise. Download a Free Trial.
• Follow the Cloudify blog to learn more about our open source PaaS stack – latest integration recipes, builds, features, and other cool stuff.  Visit the GigaSpaces blog to learn how to take your application to the next level of scalability and performance.
• NetDNA, a Tier-1 GlobalContent Delivery Network, offers a Dual-CDN strategy which allows companies to utilize a redundant infrastructure while leveraging the advantages of multiple CDNs to reduce costs.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Utility style high performance cloud servers in the US and Europe delivered on all 10GigE networking. Run any OS, take advantage of SSD storage and tailored infrastructure options.

For a longer description of each sponsor, please read more below...

Click to read more ...

Gone Fishin': Welcome to High Scalability

As a little trip down memory lane this was the first post on High Scalability, an unbelievable five years ago. The purpose of the site as laid out here has not really changed at all. The goal is to help people do their jobs better and I think more times than not that's what has happened...

We started High Scalability to help you build successful scalable websites. This site tries to bring together all the lore, art, science, practice, and experience of building scalable websites into one place so you can learn how to build your system with confidence. Hopefully this site will move you further and faster along the learning curve of success. Please Start Here

.

Gone Fishin' Two

Well, not exactly Fishin', I'll be on vacation starting today and I'll be back late November. I won't be posting anything new, so we'll all have a break. Disappointing, I know, but fear not, I will be posting some oldies for your re-enjoyment.

And If you've ever wanted to write an article for HighScalability, this would be a great time :-) I especially need help on writing Stuff the Internet Says on Scalability as I will be reading the Interwebs on a much reduced schedule. Shock! Horror! So if the spirit moves you, please write something.

My connectivity in Italy will probably be good, so I will check in and approve articles on a regular basis. Ciao...