Gone Fishin': Tumblr Architecture - 15 Billion Page Views A Month And Harder To Scale Than Twitter

With over 15 billion page views a month Tumblr has become an insanely popular blogging platform. Users may like Tumblr for its simplicity, its beauty, its strong focus on user experience, or its friendly and engaged community, but like it they do.

Growing at over 30% a month has not been without challenges. Some reliability problems among them. It helps to realize that Tumblr operates at surprisingly huge scales: 500 million page views a day, a peak rate of ~40k requests per second, ~3TB of new data to store a day, all running on 1000+ servers.

One of the common patterns across successful startups is the perilous chasm crossing from startup to wildly successful startup. Finding people, evolving infrastructures, servicing old infrastructures, while handling huge month over month increases in traffic, all with only four engineers, means you have to make difficult choices about what to work on. This was Tumblr’s situation. Now with twenty engineers there’s enough energy to work on issues and develop some very interesting solutions.

Tumblr started as a fairly typical large LAMP application. The direction they are moving in now is towards a distributed services model built around Scala, HBase, Redis, Kafka, Finagle,  and an intriguing cell based architecture for powering their Dashboard. Effort is now going into fixing short term problems in their PHP application, pulling things out, and doing it right using services.

The theme at Tumblr is transition at massive scale. Transition from a LAMP stack to a somewhat bleeding edge stack. Transition from a small startup team to a fully armed and ready development team churning out new features and infrastructure. To help us understand how Tumblr is living this theme is startup veteran Blake Matheny, Distributed Systems Engineer at Tumblr. Here’s what Blake has to say about the House of Tumblr:

Site:  http://www.tumblr.com/

Click to read more ...

Gone Fishin': Justin.Tv's Live Video Broadcasting Architecture

This is one of my favorite posts for a couple of reasons. I think it gives a lot of useful information in an interesting space. And Kyle Vogt was just a real pleasure to talk to. He was very helpful and forthcoming, which makes the whole experience better for everyone.

The future is live. The future is real-time. The future is now. That's the hype anyway. And as it has a habit of doing, the hype is slowly becoming reality. We are seeing live searches, live tweets, live location, live reality augmentation, live crab (fresh and local), and live event publishing. One of the most challenging of all live technologies is that of live video broadcasting. Imagine a world in which everyone becomes a broadcaster and a consumer of video streams, all in real-time (< 250 msec latency), all so you can talk and interact directly without feeling like you are in the middle of a time shift war. The resources and the engineering needed to make this happened must be substantial. How do you do that?

To find out I talked to Kyle Vogt, Justin.tv Founder and VP of Engineering. Justin.tv certainly has the numbers. Their 30 million unique monthly visitors even outshine YouTube in the video upload game, reportedly uploading nearly 30 hours per minute of video compared to YouTube's 23. I asked for an interview after listening to an interview with Justin Kan, another Founder of the eponymously named Justin.tv. Justin talked about how live video was fundamentally different than YouTube's batch video approach, where all the video is stored on disk and replayed later on demand. Live video can't be made by pushing video faster, it takes a completely differently architecture. Since the YouTube Architecture article is the most popular article ever on this site, I thought people might also enjoy learning about live side of the video world. Kyle was unbelievably generous with his time and insight into how Justin.tv makes all this live video magic happen, going way beyond the call, providing a tremendous number of juicy details. Anyone building a system can learn something from how they run their business. I can't thank Kyle enough for putting up with my never ending prodding.

Click to read more ...

Sponsored Post: Zoosk, Aerospike, Server Stack, Wiredrive, NY Times, CouchConf, FiftyThree, Percona, ElasticHosts, ScaleOut, New Relic, NetDNA, GigaSpaces, AiCache, Logic Monitor, AppDynamics, CloudSigma

Who's Hiring?

• Zoosk is looking for a Data Center Operations Manager to scale our rapidly growing business. Are you ready to push the boundaries of technology driving data center operations for a top consumer web product? Join our team www.zoosk.com/careers. 
• Wiredrive is looking for a SENIOR WEB APPLICATION SYSTEMS ADMINISTRATOR and a TEST AUTOMATION ENGINEER to join our agile infrustructure team. For full job descriptions please see http://wdrv.it/QA6iTw
• The New York Times is seeking a developer focused on infrastructure to join its newsroom development team. Read the full description here and send resumes to chadas@nytimes.com.
• FiftyThree, the company behind the award-winning iPad app Paper, is looking for a {Backend || DevOps} Engineer to help us build our next great product: a service to "bring ideas together". http://www.fiftythree.com/jobs
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• December 3-4, 2012 Percona Live London 2012 Millennium Gloucester Hotel and Conference Centre, London, UK http://www.percona.com/live/london-2012/
• CouchConf is a one-day, three track event is for any developer who wants to take a 
• deeper dive into Couchbase NoSQL technology, 
learn where it’s headed and build really cool stuff.

Cool Products and Services

• Aerospike: Two Trillion Transactions per month...100 million stored user profiles...25% of all video ads processed on the internet - mere realities of success for Aerospike customers. Industry leaders reveal their secrets! 
• ServerStack offers the industry's most scalable managed servers for high traffic/bandwidth websites backed by unlimited 24/7 network, server and application support.
• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. Test aiCache acceleration for free.  No sign-up required. http://aicache.com/deploy
• ElasticHosts launches white-label cloud reseller program offering 30% revenue share on fully rebranded cloud hosting.
• ScaleOut Software. In-memorry Data Grids for the Enterprise. Download a Free Trial.
• Follow the Cloudify blog to learn more about our open source PaaS stack – latest integration recipes, builds, features, and other cool stuff.  Visit the GigaSpaces blog to learn how to take your application to the next level of scalability and performance.
• NetDNA, a Tier-1 GlobalContent Delivery Network, offers a Dual-CDN strategy which allows companies to utilize a redundant infrastructure while leveraging the advantages of multiple CDNs to reduce costs.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Utility style high performance cloud servers in the US and Europe delivered on all 10GigE networking. Run any OS, take advantage of SSD storage and tailored infrastructure options.

For a longer description of each sponsor, please read more below...

Click to read more ...

Gone Fishin': Hilarious Video: Relational Database Vs NoSQL Fanbois

This is an all time favorite post. Even though I've seen this video a hundred times I still can't help but laugh...

This is so funny I laughed until I cried! Definitely NSFW. OMG it's hilarious, but it's also not a bad overview of the issues. Especially loved: You read the latest post on HighScalability.com and think you are a f*cking Google and architect and parrot slogans like Web Scale and Sharding but you have no idea what the f*ck you are talking about. There are so many more gems like that.

Thanks to Alex Popescu for posting this on MongoDB is Web Scale. Whoever made this deserves a Webby.

Gone Fishin': 10 Ways to Take your Site from One to One Million Users by Kevin Rose

This is the post that got me kicked off my original shared hosting service and prompted the move to SquareSpace. I couldn't figure out why so many people were reading this article. But they kept on coming. The site went down and I was told to vamoose.  It finally dawned on me nobody actually cared about the article, it was the name Kevin Rose that was magic. Learned a good lesson about publishing biz...

At the Future of Web Apps conference Kevin Rose (Digg, Pownce, Wefollow) gave a cool presentation on the top 10 down and dirty ways you can grow your web app. He took the questions he's most often asked and turned it into a very informative talk.

This isn't the typical kind of scalability we cover on this site. There aren't any infrastructure and operations tips. But the reason we care about scalability is to support users and Kevin has a lot of good techniques to help your user base bloom.

Here's a summary of the 10 ways to grow your consumer web application:

Click to read more ...

Gone Fishin': Building Super Scalable Systems: Blade Runner Meets Autonomic Computing In The Ambient Cloud

All in all this is still my favorite post and I still think it's an accurate vision of a future. Not everyone agrees, but I guess we'll see...

"But it is not complicated. [There's] just a lot of it." 
-- Richard Feynman on how the immense variety of the world arises from simple rules.

Contents:

1. Have We Reached the End of Scaling?
2. Applications Become Black Boxes Using Markets to Scale and Control Costs
3. Let's Welcome our Neo-Feudal Overlords
4. The Economic Argument for the Ambient Cloud
5. What Will Kill the Cloud?
6. The Amazing Collective Compute Power of the Ambient Cloud
7. Using the Ambient Cloud as an Application Runtime
8. Applications as Virtual States
9. Conclusion

We have not yet begun to scale. The world is still fundamentally disconnected and for all our wisdom we are still in the earliest days of learning how to build truly large planet-scaling applications.

Today 350 million users on Facebook is a lot of users and five million followers on Twitter is a lot of followers. This may seem like a lot now, but consider we have no planet wide applications yet. None.

Tomorrow the numbers foreshadow a new Cambrian explosion of connectivity that will look as different as the image of a bare lifeless earth looks to us today. We will have 10 billion people, we will have trillions of things, and we will have a great multitude of social networks densely interconnecting all these people to people, things to things, and people to things.

How can we possibly build planet scalable systems to handle this massive growth if building much smaller applications currently stresses architectural best practices past breaking? We can't. We aren't anywhere close to building applications at this scale, except for perhaps Google and a few others, and there's no way you and I can reproduce what they are doing. Companies are scrambling to raise hundreds of millions of dollars in order to build even more datacenters. As the world becomes more and more global and more and more connected, handling the load may require building applications 4 or 5 orders of magnitude larger than any current system. The cost for an infrastructure capable of supporting planet-scale applications could be in the 10 trillion dollar range (very roughly estimated at $100 million a data center times 10K).

If you aren't Google, or a very few other companies, how can you possibly compete? For a glimmer of a possible direction that may not require a kingdom's worth of resources, please take a look at this short video:

Click to read more ...

Cost Analysis: TripAdvisor and Pinterest costs on the AWS cloud

This is a guest post by Ali Khajeh-Hosseini, Technical Lead at PlanForCloud.com.  

I read a recent blog post about TripAdvisor's experiment with AWS where they attempted to process 700K HTTP requests per minute on a replica of their live site. There was also an interesting blog post on Pinterest's massive growth on AWS. These blogs highlighted exactly the types of questions we're interested in, mainly:

1. How much would it cost to deploy System X on Cloud Y? e.g. how much would it cost to host TripAdvisor on the AWS US-East cloud?
2. Would it be cheaper to use deployment option X or Y? e.g. would it be cheaper to use reserved instances, different types of instances, different cloud providers...
3. What happens to costs when the system grows? e.g. Pinterest has around 410TB of data on S3, what if that keeps growing at a rate of 25% every month, like it has been in the last 10 months?

I created a couple of deployments in PlanForCloud to explore these questions and the results are very interesting - they show how important it is to get an idea of costs and budget for cloud spend prior to using the cloud:

Click to read more ...

Gone Fishin': LiveJournal Architecture

This was the first architecture profile on HighScalability. IMHO LiveJournal was really the start of the openness on how to build stuff at scale, setting the whole industry off with an excellent role model. They wrote about their architecture, they open sourced their tools, they showed that success wasn't based on keeping secrets, and they set forth principles still followed by our rather amazing industry. No other industry is so open and cooperative, with their eyes cast so far forward, intent on building cool stuff. When all around seems dark it would be good to keep this little bit of light in mind...

A fascinating and detailed story of how LiveJournal evolved their system to scale. LiveJournal was an early player in the free blog service race and faced issues from quickly adding a large number of users. Blog posts come fast and furious which causes a lot of writes and writes are particularly hard to scale. Understanding how LiveJournal faced their scaling problems will help any aspiring website builder.

Site: http://www.livejournal.com/

Information Sources

Click to read more ...

Sponsored Post: Zoosk, Aerospike, Server Stack, Wiredrive, NY Times, CouchConf, FiftyThree, Percona, ElasticHosts, ScaleOut, New Relic, NetDNA, GigaSpaces, AiCache, Logic Monitor, AppDynamics, CloudSigma

Who's Hiring?

• Wiredrive is looking for a SENIOR WEB APPLICATION SYSTEMS ADMINISTRATOR and a TEST AUTOMATION ENGINEER to join our agile infrustructure team. For full job descriptions please see http://wdrv.it/QA6iTw
• The New York Times is seeking a developer focused on infrastructure to join its newsroom development team. Read the full description here and send resumes to chadas@nytimes.com.
• FiftyThree, the company behind the award-winning iPad app Paper, is looking for a {Backend || DevOps} Engineer to help us build our next great product: a service to "bring ideas together". http://www.fiftythree.com/jobs
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs

Fun and Informative Events

• December 3-4, 2012 Percona Live London 2012 Millennium Gloucester Hotel and Conference Centre, London, UK http://www.percona.com/live/london-2012/
• CouchConf is a one-day, three track event is for any developer who wants to take a 
• deeper dive into Couchbase NoSQL technology, 
learn where it’s headed and build really cool stuff.

Cool Products and Services

• Aerospike: Two Trillion Transactions per month...100 million stored user profiles...25% of all video ads processed on the internet - mere realities of success for Aerospike customers. Industry leaders reveal their secrets! 
• ServerStack offers the industry's most scalable managed servers for high traffic/bandwidth websites backed by unlimited 24/7 network, server and application support.
• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. Test aiCache acceleration for free.  No sign-up required. http://aicache.com/deploy
• ElasticHosts launches white-label cloud reseller program offering 30% revenue share on fully rebranded cloud hosting.
• ScaleOut Software. In-memorry Data Grids for the Enterprise. Download a Free Trial.
• Follow the Cloudify blog to learn more about our open source PaaS stack – latest integration recipes, builds, features, and other cool stuff.  Visit the GigaSpaces blog to learn how to take your application to the next level of scalability and performance.
• NetDNA, a Tier-1 GlobalContent Delivery Network, offers a Dual-CDN strategy which allows companies to utilize a redundant infrastructure while leveraging the advantages of multiple CDNs to reduce costs.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Utility style high performance cloud servers in the US and Europe delivered on all 10GigE networking. Run any OS, take advantage of SSD storage and tailored infrastructure options.

For a longer description of each sponsor, please read more below...

Click to read more ...

Gone Fishin': Welcome to High Scalability

As a little trip down memory lane this was the first post on High Scalability, an unbelievable five years ago. The purpose of the site as laid out here has not really changed at all. The goal is to help people do their jobs better and I think more times than not that's what has happened...

We started High Scalability to help you build successful scalable websites. This site tries to bring together all the lore, art, science, practice, and experience of building scalable websites into one place so you can learn how to build your system with confidence. Hopefully this site will move you further and faster along the learning curve of success. Please Start Here

.