Hot Scalability Links for July 17, 2010

And by hot I also mean temperature. Summer has arrived. It's sizzling here in Silicon Valley. Thank you air conditioning!

• Scale the web by appointing a Crawler Czar? Tom Foremski has the idea that Google should open up their index so sites wouldn't have to endure the constant pounding by ravenous crawler bots. Don MacAskill of SmugMug estimates 50% of our web server CPU resources are spent serving crawlers. What a waste. How this would all work with real-time feeds, paid  feeds (Twitter, movies, ...), etc. is unknown, but does it make sense for all that money to be spent on extracting the same data over and over again?
• Tweets of Gold:
  • jamesurquhart: Key to applications is architecture. Key for infrastructure supporting archs is configurability. Configurability==features.
  • tjake:  People who choose their datastore based oh hearsay and not their own evaluation are doomed.
  • b6n: No global lock ever goes unpunished.
  • MichaelSurtees: scalability, systems & process feed each other right?
  • jamesgolick: Statements like: "NoSQL database systems are designed for scalability." make me sad.
  • agastiya: Focus on stability and features first, scalability and manageability second, per-unit performance last of all. This is a quote from Jeff Darcy
    
  • bizarroargv0: I find all this talk of distributed systems a little unsettling. When it comes to data, sometimes you just want a powerful master.
• "You can never think about scale too early," says Facebook VP of Technical Operations Jonathan Heiliger. Others say scaling is a problem that's good to have, don't worry, be happy, but Jonathan thinks planning ahead has some value: "We’ve thought really far ahead but we’ve also punted on really critical things that we needed to do. Now we’re under the gun rather than being able to do them on our own time.”
• Why does Quora use MySQL as the data store rather than NoSQLs such as Cassandra, MongoDB, CouchDB, etc? Possible reasons: MySQL works, NoSQL is a fad, NoSQL is unstable, Quora isn't Facebook so scale-up works.
• Map rendering on EC2 by Andy Allan. EC2 is, more or less, exactly not what you want from a tileserver. Expensive to run, slow disks. So why is it popular? First off is buzzwords – cloud, scalable and so on.
• Scalability. If we wanted to let any of our eight million active users sort their comments history by score, we'd need $20,000 in dedicated servers. I love how Reddit is so honest about the cost of features in their system and how they balance what to do and what not to do based on the ROI.
• In defence of SQL at Seldo.com. So go forth, use your OMADS, keep an RDBMS in your back pocket, and stop being so mean to poor old SQL.
• Google Pregel Graph Processing by Ricky Ho. Pregel can be thought as a generalized parallel graph transformation framework. I think the Pregel model is general enough for a large portion of classical graph algorithm.
• Royans Tharakan summarizes a talk by Theo Schlossnagle: Thoughts on scalable web operations. Covers: optimization, tools, cookies, datastores, automation, revision control, networking, caching, people, systems, and moderation.
• Amazon opens up a high performance compute grid, but it will cost you says Miha Ahronovitz. More perspective on the release by James Hamilton.
• Wordnik passes 9 billion record mark with MongoDB. More on Hacker News.
• Going to a silicon beach this summer? The folks at GigaOM have a summer reading list for you. The "Upside of Irrationality” is on my wish list.

Comments

[Greg Linden]

agastiya tweets, "Focus on stability and features first, scalability and manageability second, per-unit performance last of all."

I don't really agree with the last part of of this one. I think performance is a feature. A fast website is a usable website.

[Jeff Darcy]

Not to blow my own horn too much, but the tweet from agastiya is actually a quote from me.

[HighScalability Team]

Sorry Jeff, I didn't see the link tucked at the end. I added a reference to your post.

[Greg Linden]

Oh, in the full context of Jeff's (not agasitya's) quote, I agree with him. He is talking about premature optimization, focusing on performance for performance sake, without regard to whether it actually impacts the user experience. The only point I wanted to make is that performance of the website from the user's point of view is important, since it is a factor in usability and satisfaction. But that is a lot different than focusing on performance timings before you know if those performance optimizations are noticeable to people who are using the website.

[HighScalability Team]

Definitely agree Greg.

[Evan Bradley]

Forethought is the most important activity in web service development, the decisions up front can save you from so much pain, important scalability questions to ask up front...1) If we build out the infrastructure to handle peak load for a given time period, how do we use that infrastructure in the non-peak days/hours to monetize or capture revenue, (idle infrastructure verses latency on peak loads) 2) Carefully plan the platform services that are critical to scaling : DNS/DHCP/NTP, User MGMT, OS PROP, CODE PROP, Trusted Host, Monitoring, Policy & Configuration Mgmt, Rapid provisioning, automated differentiation, load balancing etc..3) HW fails and load balancing is the norm: write code and use apps that incorporate strategies for risk mitigation when failures occur...4) shard/replicate data that is important so you dont lose sleep, 5) Decouple, use asynchronous communications, "code smart" for load balanced stateless programming wherever you can.

That's off the top of my head, but still, tenets to live by.