Sponsored Post: ClearStone, Schooner, deviantART, ScaleOut, aiCache, WAPT, Karmasphere, Kabam, Newrelic, Cloudkick, Membase, Joyent, CloudSigma, ManageEngine, Site24x7

Who's Hiring?

• deviantART is looking for Infrastructure and Database Operations Engineers! 
• Kabam is looking for a Quantitative Analyst and a Senior Data Engineer to join the Business Intelligence group at our social gaming startup.

Fun and Informative Events

• Interested in CouchDB Training? The CouchDB Training World Tour starts this month with new CouchDB training classes in five major cities.

Cool Products and Services

• APM (Application Performance Management) for NOSQL, Java and More - Try ClearStone 5.0. Download ClearStone 5.0 today!  http://www.evidentsoftware.com/download/
• Schooner delivers true enterprise-grade performance and availability with Membrain: a flash-optimized memcached cache and NoSQL data store for x86 servers. Download a free trial copy today!
• ScaleOut StateServer - Scale Out Your Server Farm Applications!
• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. 
• WAPT is a load, stress and performance testing tool for websites and web-based applications.
• Karmasphere is bringing Apache Hadoop power to developers and analysts. Download your Free Community Edition today!
• Newrelic - What are you doing to ensure the performance of your apps?
• Cloudkick - monitor & manage your servers better with a FREE Cloudkick developer account.
• Learn how two game developers prepared for rapid user growth in this recorded Joyent webinar: http://bit.ly/hzBoib.
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

For more information on each sponsor please see below...

Click to read more ...

Stuff The Internet Says On Scalability For March 18, 2011

Submitted for your reading pleasure on this day of wind and rain...

• The cloud is falling. Or at least shared networked storage as Reddit has a couple of long periods of downtime. Good write up at Why reddit was down for 6 of the last 24 hours. Upshot: Reddit is moving off EBS to local disks.
• Quotable Quotes:
  • @thomleggett: 32 or so joins - the sweet-spot of suck for MySQL - @emileifrem #nosql #neo4j #qconlondon
  • @jkalucki: The [Twitter] Streaming API pushes 100MB in less than a second.
  • @LusciousPear: Finding the true test of a DB is recovery when things go wrong, not "who's most web-scale" on paper #nosql
  • @TimelessP: Functionality, scalability, security... pick two.
  • @beaknit: #ccevent #nosql @adrianco: A year from oracle to simpledb. A week from simpledb to cassandra. Mental shift biggest hurdle.
  For more stuff the Internet says please read on...

  Click to read more ...

Are long VM instance spin-up times in the cloud costing you money?

Are long VM instance spin-up times in the cloud costing you money? That's the question that immediately came to mind when James Urquhart, in an interview at the Stata Conference, made this thought provoking comment: the faster you can get the resources into the hands of the people who use them, the more money you save overall.

One of the many super powers of the cloud is elasticity, the ability to dynamically acquire and release resources in response to demand. But like any good superhero, their strength must also form the basis of a not quite fatal flaw. Years and years of angsty episodes are usually required to explore this contradiction.

In the case of the cloud, the weakness reveals itself in slow VM spin-up times. Spinning up a VM in EC2 can take a little as 1-3 minutes, or can average 5-10 minutes, or it can take much longer if there's heavy usage in your availability zone. EC2 is not alone. A common complaint about Google App Engine is the cold-start problem. When a request comes in, an application must be initialized to handle it, which takes time, which means the end-user experiences increased latency.

For the rest of the article please click through...

Click to read more ...

Sponsored Post: Schooner, deviantART, ScaleOut, aiCache, WAPT, Karmasphere, Kabam, Newrelic, Cloudkick, Membase, Joyent, CloudSigma, ManageEngine, Site24x7

Who's Hiring?

• deviantART is looking for Infrastructure and Database Operations Engineers! 
• Kabam is looking for a Quantitative Analyst and a Senior Data Engineer to join the Business Intelligence group at our social gaming startup.

Fun and Informative Events

• Interested in CouchDB Training? The CouchDB Training World Tour starts this month with new CouchDB training classes in five major cities.

Cool Products and Services

• Schooner delivers true enterprise-grade performance and availability with Membrain: a flash-optimized memcached cache and NoSQL data store for x86 servers. Download a free trial copy today!
• ScaleOut StateServer - Scale Out Your Server Farm Applications!
• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. 
• WAPT is a load, stress and performance testing tool for websites and web-based applications.
• Karmasphere is bringing Apache Hadoop power to developers and analysts. Download your Free Community Edition today!
• Newrelic - What are you doing to ensure the performance of your apps?
• Cloudkick - monitor & manage your servers better with a FREE Cloudkick developer account.
• Learn how two game developers prepared for rapid user growth in this recorded Joyent webinar: http://bit.ly/hzBoib.
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

For more information on each sponsor please see below...

Click to read more ...

Twitter by the Numbers - 460,000 New Accounts and 140 Million Tweets Per Day

Twitter has taken some hits lately, but they are still pumping out the tweets and adding massive numbers of new users. Here are some numbers they just published, hot off the analytics engine:

Click to read more ...

6 Lessons from Dropbox - One Million Files Saved Every 15 minutes

Dropbox saves one million files every 15 minutes,  more tweets than even Twitterers tweet. That mind blowing statistic was revealed by Rian Hunter, a Dropbox Engineer, in his presentation How Dropbox Did It and How Python Helped at PyCon 2011.

The first part of the presentation is some Dropbox lore, origin stories and other foundational myths. We learn that Dropbox is a startup company located in San Francisco that has probably one of the most popular file synchronization and sharing tools in the world, shipping Python on the desktop and supporting millions of users and growing every day. 

About half way through the talk turns technical. Not a lot of info on how Dropbox handles this massive scale was dropped, but there were a number of good lessons to ponder:

Click to read more ...

Google and Netflix Strategy: Use Partial Responses to Reduce Request Sizes

This strategy targets reducing the amount of protocol data in packets by sending only the attributes that are needed. Google calls this Partial Response and Partial Update.

Netflix posted about adopting this strategy in their recent Netflix API redesign. We've seen previously how Netflix improved performance by creating less chatty protocols.

As a consequence packet sizes rise as more data is being stuffed into each packet in order to reduce the number of round trips. But we don't like large packets either (memory usage and packet processing overhead), so we have to think of creative ways to shrink them back down.

The change Netflx is making is to conceptualize their API as a database. What does this mean?

Click to read more ...

Medialets Architecture - Defeating the Daunting Mobile Device Data Deluge

Mobile developers have a huge scaling problem ahead: doing something useful with massive continuous streams of telemetry data from millions and millions of devices. This is a really good problem to have. It means smartphone sales are finally fulfilling their destiny: slaughtering PCs in the sales arena. And it also means mobile devices aren't just containers for simple standalone apps anymore, they are becoming the dominant interface to giant backend systems.

Click to read more ...

Stuff The Internet Says On Scalability For March 4, 2011

Submitted for your reading pleasure on this beautifully blue and sunny Friday...

• @Werner: Each day #AWS adds enough computing muscle to power one whole Amazon.com circa 2000, when it was a $2.8 billion business http://wv.ly/gMr8LQ
• Building servers to rule in hell. Datacenters spend a lot of energy on cooling down processors. Why can't they operate at higher temperatures? This is the proposition addressed by James Hamilton in Exploring the Limits of Datacenter Temprature and Datacenter Knowledge in What’s Next? Hotter Servers with ‘Gas Pedals’.
• Quotable Quotes for 200 Watson:
  • @jreichhold: One thing working at Twitter teaches me daily is that all scale is relative. What seemed impossible last year is now the daily case.
• Scalability Porn:
  • Storage, you ain't seen nothing yet, wait until every smart phone is equipped with a new gigapixel camera. These new one billion plus pixel images will take upwards of 30GB to store.

  Click to read more ...

Stack Overflow Architecture Update - Now at 95 Million Page Views a Month

A lot has happened since my first article on the Stack Overflow Architecture. Contrary to the theme of that last article, which lavished attention on Stack Overflow's dedication to a scale-up strategy, Stack Overflow has both grown up and out in the last few years.

Stack Overflow has grown up by more then doubling in size to over 16 million users and multiplying its number of page views nearly 6 times to 95 million page views a month.  

Stack Overflow has grown out by expanding into the Stack Exchange Network, which includes Stack Overflow, Server Fault, and Super User for a grand total of 43 different sites. That's a lot of fruitful multiplying going on.

Click to read more ...