Wednesday
Feb132008
What's your scalability plan?
Wednesday, February 13, 2008 at 2:38PM
How do you plan to scale your system as you reach predictable milestones? This topic came up in another venue and it reminded me about a great comment an Anonymous wrote a while ago and I wanted to make sure that comment didn't get lost.
The Anonymous scaling plan was relatively simple and direct:
My two cents on what I'm using to start a website from scratch using a single server for now. Later, I'll scale out horizontally when the need arises.
Phase 1
Single Server, Dual Quad-Core 2.66, 8gb RAM, 500gb Disk Raid 10
OS: Fedora 8. You could go with pretty much any Linux though. I like Fedora 8 best for servers.
Proxy Cache: Varnish - it is way faster than Squid per my own benchmarks. Squid chokes bigtime.
Web Server: Lighttpd - faster than Apache 2 and easier to configure for me.
Object Cache: Memcached. Very scalable.
PHP Cache: APC. Easy to configure and seems to work fine.
Language: PHP 5 - no bloated frameworks, waste of time for me. You spend too much time trying to figure out the framework instead of getting work done.
Database - MySQL 5. I didn't consider Postgres because I've never used it. There are just a lot more tools available for MySQL.
Phase 2
Max Ram out to 64 GB, cache everything
Phase 3
Buy load balancer + 2 more servers for front end Varnish/Memcached/Lighttpd.
Use original server as MySQL database server.
Phase 4
Depending on my load & usage patterns, scale out the database horizontally with an additional server. I don't expect the db to be a bottleneck for my website as only metadata info is stored there. I'll mostly be serving images stored on the file system. Possibly separate Varnish / Memcached / Lighttpd tier into separate tiers if necessary. But I'll carefully evaluate the situation at this point and scale out appropriately and use CDN for static content if necessary.
Phase 5
Max all servers to 64gb of RAM, cache, cache, cache.
Phase 6
If I get this far then I'm a multi-millionaire already so I'll replace all of the above machines with whatever the latest and greatest is at that time and keep scaling out.
The important point is that I know how to scale each layer when/if the need arises. I'll scale the individual machines when necessary and scale horizontally too.
In previous post we also read where ThemBid has a nice simple scalability plan too :
Use Munin to tell when to think about upgrading. When your growth trend will soon cross your resources trend, it's time to do something.
Move MySQL to a separate server. This frees up resources (CPU, disk, memory). What you want to run on this server depend on its capabilities. Maybe run a memcached server on it.
Move to a distributed memory cache using memcached.
Add a MySQL master/slave configuration.
If more webservers are needed us LVS on the front end as a load balancer.
The great insight ThemBid had was to use monitoring to predict when performance is hitting a limit so you can take action before the world ends. Take a look at Monitoring for some options. Some examples of how monitoring is used: Feedburner Architecture, Scaling Early Stage Startups, Thembid Architecture, Flickr Architecture, Ebay Architecture.
Most problems are pretty predictable, especially if you read this site. Have a plan in mind for what you want to do when you grow. You don't have to do all now, but make the path easier by starting in the right direction now. You'll also be much less stressed when all hell breaks loose.
The Anonymous scaling plan was relatively simple and direct:
My two cents on what I'm using to start a website from scratch using a single server for now. Later, I'll scale out horizontally when the need arises.
Phase 1
Phase 2
Phase 3
Phase 4
Depending on my load & usage patterns, scale out the database horizontally with an additional server. I don't expect the db to be a bottleneck for my website as only metadata info is stored there. I'll mostly be serving images stored on the file system. Possibly separate Varnish / Memcached / Lighttpd tier into separate tiers if necessary. But I'll carefully evaluate the situation at this point and scale out appropriately and use CDN for static content if necessary.
Phase 5
Phase 6
The important point is that I know how to scale each layer when/if the need arises. I'll scale the individual machines when necessary and scale horizontally too.
In previous post we also read where ThemBid has a nice simple scalability plan too :
The great insight ThemBid had was to use monitoring to predict when performance is hitting a limit so you can take action before the world ends. Take a look at Monitoring for some options. Some examples of how monitoring is used: Feedburner Architecture, Scaling Early Stage Startups, Thembid Architecture, Flickr Architecture, Ebay Architecture.
Most problems are pretty predictable, especially if you read this site. Have a plan in mind for what you want to do when you grow. You don't have to do all now, but make the path easier by starting in the right direction now. You'll also be much less stressed when all hell breaks loose.
Reader Comments (9)
I second that notion of using monitoring. Adding Munin to our environment was like turning on the lights. We quickly saw that as we hit high points in our traffic pattern, bottlenecks showed up in CPU usage (believe it or not). Sometimes these bottlenecks were happening in the middle of the night, when we weren't at the controls and banging away with top, vmstat, etc.
At Heavy we have written additional plugins to report on application-specific events. One example is an uptime graph for our application code, which resets every time we promote code to the production servers. It gives us good indication to whether new application code is making use of more or fewer resources.
Another I've considered is a graph of how many users are actually logged-into the site, as opposed to anonymous visitors. Logged-in users correlate to higher DB accesses for fresh data, which can really change the operation of our application.
My advice is to seperate out the MySQL server as soon as you can. That's the only server that is easier to beef up than to duplicate. Your PHP servers, or memcache servers or whatever can be commodity priced boxes. Scaling your PHP will always be easier than scaling MySQL because you can just tack on new boxes. Having MySQL on the same server as your PHP is usually asking for trouble as well. I'd move that step up to step 2 (or 1!). I also think you're going to spend more money on 2x 64gb ram servers than you would on 16x 8gb RAM servers, for no discernable difference, except more availability.
Here are what I'd do personally:
Step 1:
- 1 Server
- I'm assuming you're building a PHP/Mysql website.
- PHP 5 + MySQL 5 + maybe memcache, if you want (you might canabalize your memory)
- whatever other monitoring or caching stuff you're into
- This is NOT an efficient setup, but at step 1, you don't need one, because you don't have any traffic
Step 2:
- 1x db server (pretty beefy - whatever you can handle affording until you have to beef it up more)
- 1x memcache server (cpu isn't a big deal, just make sure you've got 3 or 4gb of RAM, more RAM if you serve a lot of different objects - faster RAM is better)
- 2-4x php server (choose one as your origin, and just rsync files to the other(s). if you do file uploads or something like that, make sure that all happens on your origin)
Step 3:
- not really a new step, just add new servers as you think they are needed, but basically keeping the same structure.
- when needed, beef up mysql server
- when needed, add more memcache servers
- when needed, add more php servers
Step 4:
- I'm assuming at this point you're hitting limits on what your 1 mysql server can handle, even with memcache
- if you have db tables that can be sharded off, like stats collection, then do so
- set up a slave db, that's either an actual slave, or just one you copy nightly with lvm snapshots (depends on how timely your data needs to be)
- offload specific read traffic to the slave (search is what we offloaded first)
- continue adding php + memcache servers (we generally have 1 memcache server per 5 php servers)
Step 5:
- continue your slave setup, add more if you need to, and continue to shard by function
- You probably have enough servers now that you can start looking into maybe autoscaling your php with Amazon's EC2 or the like. It depends on how your traffic runs: if it's relatively spikey, then this will help a lot. You just keep up a minimum # of servers during normal hours, and then auto-boot up new ones when your spike events happen. If you have pretty steady traffic at all times, then this isn't really going to help - and will probably hurt, because in general an EC2 server costs more than it's real world counter-part
Step 6:
- I dunno, I'm still on Step 5. :)
I think it would be a good idea to switch to amazon S3 for your images as soon as possible.
S3 is NOT a CDN. At least not yet. If you want somewhere to cache your images and geotarget the serving, you're going to have to get a real CDN. If you just want somewhere to store your images that's going to be decently fast, S3 is fine, but ultimately more expensive than if you hosted them yourself when you get sizable traffic. For small startups though, it's great.
I've recently had to scale an image gallery from 10s of users at a time to 500 cocurrent users.
Over the past few months, these are the steps I've followed. (Disclaimer: It wasn't nearly as clean as this because of money constraints).
Step 1
- MySQL + Apache on the same box
- 4GB ram
Step 2
- MySQL and Apache on different boxes
- 2 Servers: 4GB RAM
Step 3
- MySQL + Lighttpd. Lighttpd acting as a load balancer to delegate calls to PHP fcgi
- 3 servers: 4GB RAM each
- 1 Load Balancer - Lighttpd + PHP
- 1 Database - MySQL
- 1 App Server - PHP
Step 4
I found that I needed to split the static content into its own server, and I needed to handle file-uploads at the origin.
- 1 Load Balancer - Lighttpd + PHP (PHP is only handling a minor load here)
- 1 Database - MySQL
- 1 Static Content Server - Lighttpd is used to server small files (3KB a piece)
- 1 static content holder - Lighttpd is used to server larger files (200KB minimum through 500MB). PHP also handles file-uploads on this server.
- 2 App Servers
Step 5+
I've kept the above set up and am scaling the app servers horizontally.
why max the ram out ? phase 2 should be an analysis of what the bottle neck is.
for our setup we did the following
1 server
1 web / 1 db server
2 web servers (hardware load balanced) 1 db server
3 web servers (hardware load balanced) bigger db server (old db server became the web server)
this setup is fine for the amount of traffic we are managing :)
I'm JT, the original poster of the anonymous plan. I'm using that general platform to scale out a few projects.
Originally it was just a overview of what I was planning for a single site, but I'm going to be hosting multiple sites on the platform now. I just recently went live with http://www.subbmitt.com which is a social news site like digg.com or reddit.com. The difference is that I've stripped down the features to the bare minimum and changed a few things I don't like about digg.com or reddit.com.
First, I don't know about you, but I can never get an article on the front page of digg. So, on subbmitt.com, every article makes it to the front page. This means you get maximum exposure and also you are not wasting your time by submitting articles that will never be seen. Also, Digg has too many features that I never use anyway. I use delicious to bookmark stuff and I use my own email app to forward articles to groups of people, so these features are useless to me and are left out and so are other ones. We'll add features only as users demand them. This helps performance and also allows us to concentrate on other issues. Second, I like the simplicity of Reddit, but I wanted to make it better looking and even easier to use. So, on subbmitt.com, you don't even have to login to submit an article or to post a comment.
subbmitt.com should scale out well since it's just a collection of links & comments stored in the database. Static data is cached by Varnish, Lighttpd is the webserver and APC caches the PHP op code. Not much is currently stored in memcached for this particular site, but will be for other sites or as the usage / functionality changes. The database is a single instance mysql 5, no clustering yet. Things aren't tuned too much yet either and I'll do that as the site usage grows and as usage patterns & performance data are collected. We'll truly know where the bottlenecks are as we get some users. Sometimes little unexpected things can become huge issues. This is another reason we're not adding a bunch of features that probably won't be used yet. Why add something that may cause performance issues if it won't even be used much? Also important is monitoring of the servers. I'm experimenting with OpenNMS, Zabbix & Zenoss to monitor the infrastructure. But, for now though, I'm using Pingdom.com to monitor basic usage of the server (http, smtp, pop, imap, ftp). Pingdom.com is a great service as they monitor your servers from their own servers which are distributed across the world, so you'll see uptime & load times from a global perspective.
The original site is a month or so away from going live and will be added to the same platform. I chose to hold the original project and go forward with subbmitt.com to get a more accurate gauge of performance before I add a site with a more demanding workload.
-JT
subbmitt.com
why is it that you have to go through many servers before reaching the end point?
-----
http://underwaterseaplants.awardspace.com">sea plants
http://underwaterseaplants.awardspace.com/seagrapes.htm">sea grapes...http://underwaterseaplants.awardspace.com/seaweed.htm">seaweed
I search this hardly, thanks :-)
--
Mike
http://www.netvibes.com/halloweencostumes">halloween costumes | http://www.mixx.com/users/halloweencostume">halloween costume