Sponsored Post: Next Big Sound, ElasticHosts, 1&1, Red 5 Studios, SingleHop, Spokeo, Callfire, Attribution Modeling, Logic Monitor, New Relic, ScaleOut, AppDynamics, CloudSigma, ManageEngine, Site24x7

Who's Hiring?

• Anybody interested in helping manage a 100+ Linux server deployment? Next Big Sound is a an analytics company for the music industry and is looking someone to help them scale.
• Red 5 Studios. Wanted: DBAs and Programmers interested in MySQL scalability and replication. If interested, please see us here. 
• Callfire, one of the largest cloud telephony platforms on the web, is hiring a Sr. Software Engineer. You can learn more here.
• Spokeo is hiring backend & frontend developers, and system administrators to revolutionize the people search industry. Please visit here for more information.

Fun and Informative Events

• Sign up for this free 30-minute webinar exploring how new technology can determine which ads have been seen by users and will discuss the C3 Metrics Labs analysis of over 2 billion impressions. 

Cool Products and Services

• ElasticHosts award winning cloud server hosting launches across North America. Adding data centers in Los Angeles and Toronto. Free trial. Just visit our website.
• SingleHop is IaaS. Flexible + On-Demand Public Hosting and Computing Instances. Design and deploy your Public Cloud Instance in minutes. Sign-up now and the first month is $1.
• 1&1 Internet. Flexible, dynamic high performance cloud servers for professionals. 3 Months Free Trial.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• New Relic - real user monitoring optimize for humans, not bots. Live application stats, SQL/NoSQL performance, web transactions, proactive notifications. Take 2 minutes to sign up for a free trial.
• ScaleOut StateServer® Delivers Map/Reduce Analysis and Scalable Application Performance. Gain competitive advantage with rapid access to business intelligence. Download a free evaluation trial today.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

For a longer description of each sponsor, please read more below...

Click to read more ...

Stuff The Internet Says On Scalability For January 13, 2012

With a name like HighScalability... it has to be good:

• Facebook: 1 Billion Users?; Internet Archive: 500,000 users/day, 6PB of data, 150 billion pages, 1000 queries a second; 6,180: The number of patents granted to IBM in 2011; 676: The number of patents granted to Apple in 2011; Live TV is Dead; Kickstarter: 10,000 successfully funded projects; $82bn: Apple's cash hoard; 100 Billion Planets: Our home sweet galaxy; Creative: 100-core system-on-a-chip; 15 million: Lines of code in the Linux kernel; According to Twitter: Justin Bieber > President Obama.
• Quotable quotes:
  • @florind : I just realized that the Santa story is a classical scalability myth.
  • @juokaz : doesn't always use dating sites, but when he does, he finds out about them on High Scalability http://bit.ly/xYfBmq. True story
  • @niclashulting : The Yahoo! homepage is updated 45,000 times every five minutes." A content strategy is vital.
• Google’s Data Center Engineer Shares Secrets of ‘Warehouse’ Computing. Cade Metz interviews Google's guru of the datacenter, Luiz André Barroso: delivering good Internet service requires designing the software and hardware of entire datacenter to work together as one computer; split application pieces across an array of computers; modesty is key, select modest machines with modest processors and spread applications as thin as possible; wimpy cores won't process work fast enough to be useful, that's too thin.

To read much more the Internet has to say on Scalability, please click below...

Click to read more ...

Peregrine - A Map Reduce Framework for Iterative and Pipelined Jobs

The Peregrine falcon is a bird of prey, famous for its high speed diving attacks, feeding primarily on much slower Hadoops. Wait, sorry, it is Kevin Burton of Spinn3r's new Peregrine project--a new FAST modern map reduce framework optimized for iterative and pipelined map reduce jobs--that feeds on Hadoops.

If you don't know Kevin, he does a lot of excellent technical work that he's kind enough to share it on his blog. Only he hasn't been blogging much lately, he's been heads down working on Peregrine. Now that Peregrine has been released, here's a short email interview with Kevin on why you might want to take up falconry, the ancient sport of MapReduce.

What does Spinn3r do that Peregrine is important to you?

For the rest of the interview please click below...

Click to read more ...

The Etsy Saga: From Silos to Happy to Billions of Pageviews a Month

Seldom do we get to hear stories of the bumps and bruses earned by a popular website during its formative years. Ross Snyder, a Sr. Software Engineer at Etsy, changes that with an engaging talk he gave at Surge 2011: Scaling Etsy: What Went Wrong, What Went Right.

Ross gives a detailed and honest account of how Etsy went from a raw startup in 2005, to a startup struggling with their success in 2007, to the mean, handmade, super scaling, ops driven machine they’ve become in 2011.

There’s lots to learn from this illuminating story of transformation:

Origin Story

Click to read more ...

Stuff The Internet Says On Scalability For January 6, 2012

OMG, it's 2012:

• Harry Bombarda Twilight; 200 Million: Chinese online shoppers; Quantum 150 qubit computer: all the power of today's supercomputers;  Sperm: two aspirins worth could repopulate the world; 1 Billion: the number of iOS and Android apps downloaded in a week; Watson: 250 Servers, 2,880 cores, 10 racks, 16 Terabytes RAM, 80 Teraflops; Reddit: 2 Billion Pageviews
• Quotable Quotes:
  • Robert Martin : The hallmark of a really good architecture is that it allows major decisions to be deferred. 
  • Building Memory-efficient Java Applications: Practices and Challenges : More abstractions = less awareness of costs.
  • Ian Muir : When we do something that Microsoft did not anticipate, it's nothing but pain.
  • @kekline : Want to know a secret - NoSQL's rapid growth is really about NoNormalization
  • Jeremy Zawodny : The fact that I can look back on code I wrote a few years ago and identify ways that I’d do it better is good. It means I’m still learning. But the fact that I can successfully resist the urge to change the code is even better.
  • John Boyd : people first, ideas second, hardware last.
• So cool: Glowing bacteria biopixels: The sensor displays of the future. Bacteria talk to each other using quorum sensing, which means they talk using molecules. This doesn't scale to millions of bacteria. The solution: a microfluidic chips were designed to harness the localized trigger and broadcast it to the plethora of shared colonies existing on the chip. Each of the bacteria cells on the microfluidic chip is called a “biopixel." The future of sensing technology is going to be in living sensors.

The Internet has a lot more to Say on Scalability, please read the whispers below...

Click to read more ...

Shutterfly Saw a Speedup of 500% With Flashcache

In the "should I or shouldn't I" debate around deploying SSD, it always helps to have real-world data. Fiesta! with a live-blog summary of a presentation by Kenny Gorman on Shutterfly on MongoDB Performance Tuning.

What if you still need more performance after doing all of this tuning? One option is to use SSDs. Shutterfly uses Facebook’s flashcache: kernel module to cache data on SSD. Designed for MySQL/InnoDB. SSD in front of a disk, but exposed as a single mount point. This only makes sense when you have lots of physical I/O. Shutterfly saw a speedup of 500% w/ flashcache. A benefit is that you can delay sharding: less complexity.

The whole series of posts has a lot of great information and is worth a longer look, especially if you are considering using MongoDB. 

Click to read more ...

How Facebook Handled the New Year's Eve Onslaught

How does Facebook handle the massive New Year's Eve traffic spike? Thanks to Mike Swift, in Facebook gets ready for New Year's Eve, we get a little insight as to their method for the madness, nothing really detailed, but still interesting.

Problem Setup

Click to read more ...

Sponsored Post: Red 5 Studios, SingleHop, Spokeo, Callfire, Attribution Modeling, Logic Monitor, New Relic, ScaleOut, AppDynamics, CloudSigma, ManageEngine, Site24x7

Who's Hiring?

• Red 5 Studios. Wanted: DBAs and Programmers interested in MySQL scalability and replication. If interested, please see us here. 
• Callfire, one of the largest cloud telephony platforms on the web, is hiring a Sr. Software Engineer. You can learn more here.
• Spokeo is hiring backend & frontend developers, and system administrators to revolutionize the people search industry. Please visit here for more information.

Fun and Informative Events

• Sign up for this free 30-minute webinar exploring how new technology can determine which ads have been seen by users and will discuss the C3 Metrics Labs analysis of over 2 billion impressions. 

Cool Products and Services

• SingleHop is IaaS. Flexible + On-Demand Public Hosting and Computing Instances. Design and deploy your Public Cloud Instance in minutes. Sign-up now and the first month is $1.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• New Relic - real user monitoring optimize for humans, not bots. Live application stats, SQL/NoSQL performance, web transactions, proactive notifications. Take 2 minutes to sign up for a free trial.
• ScaleOut StateServer® Delivers Map/Reduce Analysis and Scalable Application Performance. Gain competitive advantage with rapid access to business intelligence. Download a free evaluation trial today.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

For a longer description of each sponsor, please read more below...

Click to read more ...

Stuff The Internet Says On Scalability For December 30, 2011

Pork. The Other HighScalability:

• PlentyOfFish: 6 Billion Page Views; World: info doubling every 2 years; 2015: 7,910 exabytes of global digital data; Khan Academy: 4 million uniques; G+: 62 million users; Zynga:  leased 9 megawatts of capacity; Heroku: billions of page views a month
• Quoteable quotes:
  • Udi Dhan : Scalability is not boolean.
  • John Boyd: Look at the mission, not the technology. And if you do look at the mission, don't look at the most fashionable mission of the day.
  • @BigDataClouds : I think the fear of change is the biggest challenge that companies are facing
  • @cjzero : If you were wondering, the #Mythbusters scalability test of a Newton's Cradle using wrecking balls? Busted.
  • @Xorlev : Scalability is really really hard. That's why it's fun. It pushes the limits of engineering talent.
• 100 Best Cloud & Data Stats of 2011 by Zenoss. Lots of fun facts about how mind bendingly huge the world of information is exponentially becoming. More from the Economist: Welcome to the yotta world. 

Read more great Stuff the Internet has to say on Scalability by clicking below...

Click to read more ...

Strategy: Guaranteed Availability Requires Reserving Instances in Specific Zones

When EC2 first started the mental model was of a magic Pez dispenser supplying an infinite stream of instances in any desired flavor. If you needed an instance, because of a either a failure or traffic spike, it would be there. As amazing as EC2 is, this model turned out to be optimistic.  

From a thread on the Amazon discussion forum we learn any dispenser has limits:

As Availability Zones grow over time, our ability to continue to expand them can become constrained. In these scenarios, we will prevent customers from launching in the constrained zone if they do not yet have existing resources in that zone. We also might remove the constrained zone entirely from the list of options for new customers. This means that occasionally, different customers will see a different number of Availability Zones in a particular Region. Both approaches aim to help customers avoid accidentally starting to build up their infrastructure in an Availability Zone where they might have less ability to expand.

The solution: if you need guaranteed resources in different zones, purchase Reserved Instances. This will assure capacity when needed. There's no way to know if the instance types you are interested in are available in an availability zone, so reserving instances is the only solution. 

Click to read more ...