Wednesday
Sep102008
Shard servers -- go big or small?
Wednesday, September 10, 2008 at 1:48PM
Hello everyone,
I'm designing a website/widget that my business partner and I expect to serve millions of hits daily. As such we must shard our database (and we're designing with shards in mind right from the beginning). However, the one thing I haven't been able to figure out from Googling is the best hardware to go with for shards. I'm using exclusively InnoDB tables.
We'll (eventually) be running 3 groups of database servers:
a) Session servers for php sessions. These will have a very high write volume.
b) ID servers. These will match a couple primary indices (such as user ID) to a given shard. These will have an intense read load, plus a moderate amount of writes.
c) Shard servers. These will hold the bulk of the data. These will have a high read load and a lowish write load.
Group A is done as a database instead of using memcached so users aren't logged out if a memcached server goes down. As the write load is high, a pair of high performance master-master servers seems obvious. What's the ideal hardware setup for machines with this role? Maxed RAM and fast disks seem reasonable. Should I bother with RAID > 0 if I have a live backup on the other master? I hear 4 cores is optimal for InnoDB -- recommendations?
Group B. Again, it looks like maxed RAM is recommended here. What about disks? Should I go for 10K or will regular SATA2 drives be okay? RAID 0, 5, 10? Cores? Should I think about slaves to a master-master setup?
Group C. It seems to me these machines can be of any capacity because the data they hold is easily spread between shards. What is the query-per-second per dollar sweet spot when it comes to cores and number of disks? Should I beef these machines up, or stick with low end hardware? Should I still max the RAM?
I have some other thoughts on system setup, too. As the data stored in the PHP sessions won't change frequently (it'll likely remain static for a user's entire visit -- all variable data can be stored in Group C shard servers), I'm thinking of using a memcached setup in front of the database and only pushing writes through to the database when necessary. Your thoughts?
We're also starting this on a minimal budget (of course), so where in the above is it best spent? Keep in mind that I can recycle machines used in Group A & B in Group C as times goes on.
Anyway, I'd love to hear from the expertise of the forum. I've been reading for a long time, and I'll be writing as our project evolves :)
--Mark
I'm designing a website/widget that my business partner and I expect to serve millions of hits daily. As such we must shard our database (and we're designing with shards in mind right from the beginning). However, the one thing I haven't been able to figure out from Googling is the best hardware to go with for shards. I'm using exclusively InnoDB tables.
We'll (eventually) be running 3 groups of database servers:
a) Session servers for php sessions. These will have a very high write volume.
b) ID servers. These will match a couple primary indices (such as user ID) to a given shard. These will have an intense read load, plus a moderate amount of writes.
c) Shard servers. These will hold the bulk of the data. These will have a high read load and a lowish write load.
Group A is done as a database instead of using memcached so users aren't logged out if a memcached server goes down. As the write load is high, a pair of high performance master-master servers seems obvious. What's the ideal hardware setup for machines with this role? Maxed RAM and fast disks seem reasonable. Should I bother with RAID > 0 if I have a live backup on the other master? I hear 4 cores is optimal for InnoDB -- recommendations?
Group B. Again, it looks like maxed RAM is recommended here. What about disks? Should I go for 10K or will regular SATA2 drives be okay? RAID 0, 5, 10? Cores? Should I think about slaves to a master-master setup?
Group C. It seems to me these machines can be of any capacity because the data they hold is easily spread between shards. What is the query-per-second per dollar sweet spot when it comes to cores and number of disks? Should I beef these machines up, or stick with low end hardware? Should I still max the RAM?
I have some other thoughts on system setup, too. As the data stored in the PHP sessions won't change frequently (it'll likely remain static for a user's entire visit -- all variable data can be stored in Group C shard servers), I'm thinking of using a memcached setup in front of the database and only pushing writes through to the database when necessary. Your thoughts?
We're also starting this on a minimal budget (of course), so where in the above is it best spent? Keep in mind that I can recycle machines used in Group A & B in Group C as times goes on.
Anyway, I'd love to hear from the expertise of the forum. I've been reading for a long time, and I'll be writing as our project evolves :)
--Mark
Reader Comments (2)
> Group A is done as a database instead of using memcached so users aren't logged out if a memcached server goes down.
Have you considered client side memcached replication? That would give you reliability and simplicity without having to manage more databases.
> Group B and C
I don't that I would necessarily separate out B & C as they seem to have a similar profile. Since you seem read heavy I would definitely use memcached to offload the database which could reduce your database requirements.
With proper caching at all the usual places and a very capable machine you might be able to serve all your traffic without resorting to sharding. Certainly encap all your database access so you can transparently add sharding later when needed, but I would put my development effort elsewhere until it was shown sharding was necessary.
Wikipedia's setup might be a good guide (http://highscalability.com/paper-wikipedias-site-internals-configuration-code-examples-and-management-issues). They use dual quad-core servers(8 CPU cores per box) with 16GB of RAM and RAID0; master, many replicated slaves, read operations are load balanced over the slaves, write operations go to the master.
> A Have you considered client side memcached replication? That would give you reliability and simplicity without having to manage more databases.
That's a thought. Probably a lot cheaper, too. I just discovered repcached. Have you heard of anyone's experience with it?
> I don't that I would necessarily separate out B & C as they seem to have a similar profile. Since you seem read heavy I would definitely use memcached to offload the database which could reduce your database requirements.
Group B is essentially the index for Group C. I have it grouped differently so Group C stuff doesn't bog down the Group B stuff. Group B isn't easy to shard, while Group C is. There is no reason why the Group B servers can't also serve Group C stuff to start, but I also have it separated so I'm prepared to shard as soon as needed. Thankfully, it doesn't complicate the application design very much.
Traffic wise, we could very easily be dealing with tens of millions of API requests per day in a matter of months. The idea we have is extremely handy and will be very popular once released (can't give more details, sorry). So I'm aiming to have a design capable of handling >10k QPS when the need arises (by adding more hardware).
I read the Wikipedia article. Thanks for the pointer! :) I may indeed follow in their footsteps. It's amazingly difficult to find any guide on what kind of hardware one should use in a distributed setup.