Monday
Nov052007
Strategy: Diagonal Scaling - Don't Forget to Scale Out AND Up
Monday, November 5, 2007 at 2:35PM
All the cool kids advocate scaling out as the secret sauce of scaling. And it is, but don't forget to serve some tasty "scaling up" as a side dish. Scaling up doesn't have to mean buying a jet propelled, liquid cooled, 128 core monster super computer. Scaling up can just mean buying at the high end of the commodity buffet by buying more cores, more memory and using a shared nothing architecture to take advantage of all that power without adding complexity. Scale out when you need to, but big beefy boxes can absorb a lot of load before it's necessary to hit up your data center for more rack space. Here are a few examples of scaling out and up:
John Allspaw, Flickr's operations manager, coined the term diagonal scaling for this strategy. In Making a site faster by removing machines (and a comment on this post) John told how Flickr replaced 67 dual-cpu boxes with 18 dual quad-core machines and recovered almost 4x rack space and reduced costs by about 50 percent.
Fotolog's strategy is to scale up and out. By adding more cache, more RAM, more CPUs, and more efficient CPUs they were able to handle many millions more users with the same number of machines. This was a conscious choice on their part and it worked beautifully.
Wikimedia says scaling out doesn't require using cheap hardware. Wikipedia's database servers these days are 16GB dual or quad core boxes with 6 15,000 RPM SCSI drives in a RAID 0 setup.
Kevin Burton in his Distributed Computing Fallacy #9 post also says scaling out doesn't mean cheap:
Jeremy Cole in Scaling out AND up, a compromise asks for compromise:
We’re seeing machines with eight cores and 32G of memory. If we were to buy eight disks for these boxes it’s really like buying 8 machines with 4G each and one disk. This partially goes into the horizontal vs vertical scale discussion. Is it better to buy one $10k box or 10 $1k boxes? I think it’s neither. Buy 4 $2.5k boxes. The new multicore stuff is super cheap.
Scaling out doesn’t mean using crappy hardware. I think people take the “scale out” model (that they’ve often only read about from outdated conference presentations) to quite an extreme. They think scaling out means using desktop-class, bad hardware, and just buying a ton of them. That model doesn’t work, and it’s hell to maintain in the long term.
Use commodity hardware. You often hear the term “commodity hardware” in reference to scale out. While crappy hardware is also commodity, what this means is that instead of getting stuck on the low-end $40k machine, with thoughts of upgrading to the $250k machine, and maybe later the $1M machine, you use data partitioning and any number of let’s say $5k machines. That doesn’t mean a $1k single-disk crappy machine as said above. What does it mean for the machine to be “commodity”? It means that the components are standardized, common, and the price is set by the market, not by a single corporation. Use commodity machines configured with a good balance of price vs. performance.
Reader Comments (9)
Thanks for posting this. Vertical scaling has its place, and ideally, it sits right alongside horizontal. How about we call it: "Diagonal Scaling" :)
We replaced 67 dual-cpu boxes with 18 dual quad-core machines. In the end, it meant gaining back almost 4x rackspace, and it cost about 50% less: http://www.kitchensoap.com/2007/08/20/making-a-site-faster-by-removing-machines/"> vertical scaling isn't always bad
> Diagonal Scaling
I like it!
I wonder if there's a matrix to measure power against cost. The standard will surely change month to month. But is there a standard method to measure it, and then a benchmark that could be set?
Would you need to aggregate CPU, disk and memory into a single performance number? It might work with memory / $, disk / $, etc.
I suppose it would come down to the current market. So take a range of machines available, from the cheapest to the most expensive. Calculate their respective performance numbers, and then grade accordingly.
Somebody's probably doing it already... :)
Cheers - http://www.callum-macdonald.com/" title="Callum" target="_blank">Callum
PS> Another subject which, when "Re: " is added, exceeds the 64 character limit.
It has nothing to do with scale out vs scale up. The problem is "how do I maximize throughput with the lowest fixed one time cost in hardware and the lowest running cost of power"
Right now, if you look at the market prices, you can see that motherboard pricing scales linearly to 4 Socket. Ram prices scales up linearly if you buy the most efficient price/GB sticks(right now 2GB is the sweet spot, although 4GB is getting into the range). The problem is the CPUs. Both AMD and Intel charges a 4-5 times premium on 4 Socket CPUs. Now, AMD's Opteron gives you a pretty good linear like scaling due to the integrated memory controller. Still the best solution to the question above right now settles in the 2 Socket Server space with 8x 2GB of ram. (45nm Xeon E5400 series and 65nm Quad core Opteron 2300 series) In this space, Xeons have higher performance. But the FB-DIMMs really kill of the performance per watt. You are paying 0.5A for 8 sticks of FB-DIMMs. So the choice right now typically is to get the 5100 san clemente with L5410s or the Nvidia 3600 motherboard with dual Opteron 2346HEs.
To add to the comment above, I forgot the hard drives.
Hard drives are all about IO per watt per dollar. If you are doing database server, those 15K hard drives seems expensive, but they are cheaper per IO. So get yourself a 15K 72GB SAS drive for 200, instead of 3 7200RPM 500GB hard drives. You will be faster with that single 15K SAS drive.
For a 8 core 16GB database system, you need a minimum 8 drive SAS RAID10 to keep the Database server happy.
Web servers can get away with a RAID1 2 drive SAS or 7200RPM SATA drive. But use ReiserFS on the web servers for those small image files.
Well, the 4cores/16gig machines are not anything close to high-end, it's still pretty much commodity — what is it now, about 3k per box? I haven't really heard cool guys who would advocate you should scale out by using an array of mac minis or similar hardware. It basically comes up to an economic equation that includes cost per box, number of boxes, non-linear expenses for switches and cabling, as well as power consumption and managing heat. Personally I do not envy cool guys who make claims about scaling out on particular node configuration without this equation covering their backs.
Just for the record, while I'm sure John Allspaw came up with it independently, he didn't actually coin the term Diagonal Scaling. Sun used the term in a white paper from 2000, titled "Scaling the N-Tier Architecture - Solaris Infrastructure Products and Architecture" and it may go back further than that.
David: Thanks for that. I hadn't seen the term used before, and I couldn't imagine the concept (or even the term) being very original; it should be considered a fundamental of any mature web operation. Even the tech reviewers of my book hadn't heard the term before. :)
I think the reason I was associating these 2 is because I thought: In Allspaw’s model of diagonal scaling, this law is not only undermined, but is infact inverted, also i too, need some http://www.computerhelpguides.com">tips and advice about your computer, the problem needs to be resolved asap,