Monday
Jan072008
How Ruby on Rails Survived a 550k Pageview Digging
Monday, January 7, 2008 at 2:45AM Shanti Braford details how his Ruby on Rails based website survived a 24 hour 550,000+ pageview digg attack. His post cleanly lays out all the juicy setup details, so there's not much I can add.
Hosting costs $370 a month for 1 web server, 1 database server, and sufficient bandwidth. The site is built on RoR, nginx, MySQL, and 7 mongrel servers. He thinks Rails 2.0 has improved performance and credits database avoidance and fragment caching for much of the performance boost.
Keep in mind his system is relatively static, but it's a very interesting and useful experience report.
Reader Comments (10)
I'm more amazed that his system handled the spike on a level he'd never exprienced before without any interference on his part, rather than RoR handling 505k pageviews. Scaling in response to a traffic spike is always a bitch; running a gaming site during E3 was always fun.
Be wary of the traffic comparison he gives at the end though. Twitter only doing 500k uniques a month sounds WAY WAY low. If you've heard of a website, they're doing at least 1 million on month, and I'd put twitter at closer to 5 mill given it's popularity and alexa ranking. Don't be tricked into thinking $327/month will host a good size website.
This article makes a big mistake by conflating unique page views with page views. Twitter may have 500,000 uniques and 25,000,000 page views easily.
550k Pageviews in 24 hours?
That's 22917 per hour.
382 per minute.
6.36 per second.
Sorry, not impressed.
Unless you're stuck with hardware from 5 years ago or your application
requires extraordinary expensive i/o or cpu work then this is just fly squat.
If a "normal webapp" in RoR starts to sweat under such a light load then, dude, change your tools.
Some of our PHP and java garbage (intern hacks, postgres and app on same machine, far from optimized)
handles 30 and more hits/sec with ease, on standard-hardware from a year ago (2 cpus, 2G, 6 spindles).
Anything that chokes under 40 hits/sec per machine is either doing something unusually expensive
or something that you should, by all means, not put on your resume.
Like the previous commenter said... 550K a day is quite nothing. Not too long ago I had to deploy some Java apps doing tons of images + movie processing (extracting some key frame and generate some thumbnails = Java+JNI). The average daily transaction per minute is around 1000 with some spike at 2000 per minute. So it's 1.5 to 3 millions daily hits and it's running on 4 servers.
550 K a day is basically peanuts in a scaling world nowadays... bring it to 10 millions and more and then it become interesting
Just thought I'd take a moment to respond.
First off, it wasn't 550k spread out evenly over 24 hours. About 300k came over the course of the first few hours of being on the front page. Still, it's not earth-shattering, I know.
But people / my clients especially have heard so much RoR FUD that they have no perspective when it comes to scaling. Only the elite, "successful" Web 2.0 startups will ever make it into the traffic stratosphere of 100-200k daily uniques. And this, as we've seen (if you have the right site profile / caching strategies / etc), can be handled for $400 or thereabouts!
What clients really should be asking, is can it scale economically/efficiently/etc?
At $3 CPMs (which I get regularly for sites like these), 550k pageviews / day puts you at ~ $50k per month in revenues. $400 is about 1% of that. What technology company wouldn't want hardware + hosting costs to be 1% of their revenues?
Granted, these kinds of sites aren't Google... but they're not all too different from the coding required for sites like Digg, reddit, etc.
"This article makes a big mistake by conflating unique page views with page views." -- It never conflated that.
To be honest, I'd be surprised if Twitter's web frontend sees this much traffic in a single day (it's API is another story). Since they've never posted raw traffic stats to this effect (recently, at least), it's all conjecture anyway.
It's me, the "not so impressed" guy again.
Well, I'm not sure where this is going, other than stating the obvious: Hardware and hosting is cheap.
If your company pays significantly more than 1% for that then you're in serious trouble either way
(unless your company is in the process of starting up, ofcourse).
I fail to see how this is news because it's been that way for a few years now.
If hardware/hosting was expensive then we wouldn't have seen one-man-shows like del.icio.us and others take off.
The major cost of sustaining online business has not been hosting or hardware for a looong time now.
That's not entirely true...
Hosting shouldn't be a huge expenditure on your part, but bandwith sure can be if you run a video site. A pretty decent sized video website could easily rack up $50-100k/month. If that's 1% of your revenue on that size of a website, then please send me some investment info - I'm in!
And like I said before, Shanti's success shouldn't be cast aside just because his website isn't as big as the ones we've worked on or because we've seen systems work at a similar level. The point is that his system scaled by a huge factor to a level he had not previously dealt with, and it "just worked". I haven't dealt with RoR enough to know whether that's common for it, or if Shanti's system was just well built. Either way, congrats - don't let us get you down.
@Anonymous - cool, you're in the less vocal minority then on the "scaling is cheap" front. =)
I just started consulting full-time, and the #1 question is "can Rails scale?" If it's a non-issue then I'll just tell them that; but the blogosphere seems to love the meme that rails can't scale.
Thanks Brent. Yes, hopefully one day I'll look back on this experience and think "wow, 550k, peanuts!" but until then it's a certainly a personal milestone. Cheers!
Looks like the link in this article might be broken. Would love to read it—can fix? :)
Thanks! I'm a big fan of this site.
Sorry WIllie, I can't manage to find the original article.