SQL + NoSQL = Yes !

This is a guest post by Frédéric Faure (architect at Ysance), you can follow him on twitter.

Data storage has always been one of the most difficult problems to address, especially as the quantity of stored data is constantly increasing. This is not simply due to the growing numbers of people regularly using the Internet, particularly with all the social networks, games and gizmos now available. Companies are also amassing more and more meticulous information relevant to their business, in order to optimize productivity and ROI (Return On Investment). I find the positioning of SQL and NoSQL (Not Only SQL) as opposites rather a shame: it’s true that the marketing wave of NoSQL has enabled the renewed promotion of a system that’s been around for quite a while, but which was only rarely considered in most cases, as after all, everything could be fitted into the « good old SQL model ». The reverse trend of wanting to make everything fit the NoSQL model is not very profitable either.

So, what’s new … and what isn’t?

Click to read more ...

Sponsored Post: Electronic Arts, Joyent, Membase, CloudSigma, ManageEngine, Site24x7

Who's Hiring?

• Electronic Art's is hiring a QA Engineer and a Cloud Architecture Technical Director II for their Pogo.com team.

Fun and Informative Events

• A new round of Membase meetups have been planned for January 2011 for San Diego, Denver, Seattle, Vancouver and Chicago.

Cool Products and Services

• See how Joyent's cloud solutions support game studios from game concept to launch, then through rapid scalability and ongoing success. http://www.joyent.com/solutions/gamestudios/
  
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

Click to read more ...

Netflix: Use Less Chatty Protocols in the Cloud - Plus 26 Fixes

Updated on Friday, February 11, 2011 at 11:26AM by HighScalability Team

In 5 Lessons We’ve Learned Using AWS, Netflix's John Ciancutti says one of the big lessons they've learned is to create less chatty protocols:

In the Netflix data centers, we have a high capacity, super fast, highly reliable network. This has afforded us the luxury of designing around chatty APIs to remote systems. AWS networking has more variable latency. We’ve had to be much more structured about “over the wire” interactions, even as we’ve transitioned to a more highly distributed architecture.

There's not a lot of advice out there on how to create protocols. Combine that with a rush to the cloud and you have a perfect storm for chatty applications crushing application performance. Netflix is far from the first to be surprised by the less than stellar networks inside AWS. 

A chatty protocol is one where a client makes a series of requests to a server and the client must wait on each reply before sending the next request. On a LAN this can work great. LAN's are typically fast, wide, and drop few packets.

Move that same application to a different network, one where round trip times can easily be an order of magnitude or larger because either the network is slow, lossy or poorly designed, and if a protocol takes many requests to complete a transaction, then it will make a dramatic difference in performance.

My WAN acceleration friends says Microsoft's Common Internet File System (CIFS) is infamous for being chatty. Transferring a 30MB file could tally something like 300msecs of latency on a LAN. On a WAN that could stretch to 7 minutes. Very unexpected results. What is key here is how the quality characteristics of the pipe interacts with the protocol design.

OK, chatty protocols are bad. What can you do about it?

Click to read more ...

Stuff the Internet Says on Scalability For December 17th, 2010

• If you missed it here's a link to my webinar and here's the slidedeck for the talk with a buch of additional slides that I didn't have a chance to talk about. The funky picture of Lincoln is classic.
• Can MySQL really handle 1,000,000 req/sec? Sure, when you turn it into a NoSQLish database, skip all the SQL processing, and access the backend store directly. Percona is making this possible with their HandlerSocket plugin based on the work of Yoshinori Matsunobu.
• Quotable Quotes:
  • @labsji: If SQL is an abstraction of Big machines....NoSQL is an abstration of distributed computing.
  • @olov: man this eventual consistency #nosql thingy makes #facebook even more annoying. "you have a new comment, no you dont"
• Nice racks. Time has pictures of a Facebook datacenter. 

Click to read more ...

Still Time to Attend My Webinar Tomorrow: What Should I Do? Choosing SQL, NoSQL or Both for Scalable Web Applications

It's time to do something a little different and for me that doesn't mean cutting off my hair and joining a monastery, nor does it mean buying a cherry red convertible (yet), it means doing a webinar!

• On December 14th, 2:00 PM - 3:00 PM EST, I'll be hosting What Should I Do? Choosing SQL, NoSQL or Both for Scalable Web Applications.
• The webinar is sponsored by VoltDB, but it will be completely vendor independent, as that's the only honor preserving and technically accurate way of doing these things.
• The webinar will run about 60 minutes, with 40 minutes of speechifying and 20 minutes for questions.
• The hashtag for the event on Twitter will be SQLNoSQL. I'll be monitoring that hashtag if you have any suggestions for the webinar or if you would like to ask questions during the webinar. 

Click to read more ...

Sponsored Post: Joyent, Membase, Appirio, CloudSigma, ManageEngine, Site24x7

Who's Hiring?

• Appirio is hiring 5 core product engineers for one of Silicon's Valley's "most promising startups".

Fun and Informative Events

• A new round of Membase meetups have been planned for January 2011 for San Diego, Denver, Seattle, Vancouver and Chicago.
• Announcing My Webinar On December 14th: What Should I Do? Choosing SQL, NoSQL Or Both For Scalable Web Applications

Cool Products and Services

• See how Joyent's cloud solutions support game studios from game concept to launch, then through rapid scalability and ongoing success. http://www.joyent.com/solutions/gamestudios/
  
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

Click to read more ...

What the heck are you actually using NoSQL for?

It's a truism that we should choose the right tool for the job. Everyone says that. And who can disagree? The problem is this is not helpful advice without being able to answer more specific questions like: What jobs are the tools good at? Will they work on jobs like mine? Is it worth the risk to try something new when all my people know something else and we have a deadline to meet? How can I make all the tools work together?

In the NoSQL space this kind of real-world data is still a bit vague. When asked, vendors tend to give very general answers like NoSQL is good for BigData or key-value access. What does that mean for for the developer in the trenches faced with the task of solving a specific problem and there are a dozen confusing choices and no obvious winner? Not a lot. It's often hard to take that next step and imagine how their specific problems could be solved in a way that's worth taking the trouble and risk.

Let's change that. What problems are you using NoSQL to solve? Which product are you using? How is it helping you? Yes, this is part the research for my webinar on December 14th, but I'm a huge believer that people learn best by example, so if we can come up with real specific examples I think that will really help people visualize how they can make the best use of all these new product choices in their own systems.

Here's a list of uses cases I came up with after some trolling of the interwebs. The sources are so varied I can't attribute every one, I'll put a list at the end of the post. Please feel free to add your own. I separated the use cases out for a few specific products simply because I had a lot of uses cases for them they were clearer out on their own. This is not meant as an endorsement of any sort. Here's a master list of all the NoSQL products. If you would like to provide a specific set of use cases for a product I'd be more than happy to add that in.

Click to read more ...

GPU vs CPU Smackdown : The Rise of Throughput-Oriented Architectures

In some ways the original Amazon cloud, the one most of us still live in, was like that really cool house that when you stepped inside and saw the old green shag carpet in the living room, you knew the house hadn't been updated in a while. The network is a little slow, the processors are a bit dated, and virtualization made the house just feel smaller. It has been difficult to run high bandwidth or low latency workloads in the cloud. Bottlenecks everywhere. Not a big deal for most applications, but for many high performance applications (HPC) it was a killer.

In a typical house you might just do a remodel. Upgrade a few rooms. Swap out builder quality appliances with gleaming stainless steel monsters. But Amazon has a big lot, instead of remodeling they simply keep adding on entire new wings, kind of like the Winchester Mystery House of computing.

The first new wing added was a CPU based HPC system featuring blazingly fast Nehalem chips, virtualization replaced by a close to metal Hardware Virtual Machine (HVM) architecture, and the network is a monster 10 gigabits with the ability to specify placement groups to carve out a low-latency, high bandwidth cluster. Bottlenecks removed. Most people still probably don't even know this part of the house exists.

The newest addition is a beauty, it's a graphics processing unit (GPU) cluster as described by Werner Vogels in Expanding the Cloud - Adding the Incredible Power of the Amazon EC2 Cluster GPU Instances . It's completely modern and contemporary. The shag carpet is out. In are Nvidia M2050 GPU based clusters which make short work of applications in the sciences, finance, oil & gas, movie studios and graphics.

Click to read more ...

Sponsored Post: Cloudkick, Strata, Undertone, Joyent, Appirio, CloudSigma, ManageEngine, Site24x7

Who's Hiring?

• Appirio is hiring 5 core product engineers for one of Silicon's Valley's "most promising startups".
• Undertone is looking for a Software Engineer and a QA Engineer.

Fun and Informative Events

• Announcing My Webinar On December 14th: What Should I Do? Choosing SQL, NoSQL Or Both For Scalable Web Applications
• O'Reilly Strata Making Data Work conference. Turn data into decisions.

Cool Products and Services

• Cloudkick - monitor & manage your servers better with a FREE Cloudkick developer account.
• Learn how Kabam enlisted Joyent to be the cloud environment for its growing game empire.  http://www.joyent.com/customers/kabam/
• CloudSigma. Instantly scalable European cloud servers.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

Click to read more ...

Stuff the Internet Says on Scalability For November 29th, 2010

Eating turkey all weekend and wondering what you might have missed?

• James Hamilton on why “all you have learned about disks so far is probably wrong" in Availability in Globally Distributed Storage. It turns out for the same reason our financial systems melt down: black swans. The world is predictably unpredictable. Murat Demirbas also has a good post on the same Google research paper.
• Stack Overflow Hits 10M Uniques
• Vroom...Formula One racecar streams 27 gigabytes of telemetry data during a race weekend! 200 sensors “measuring anything and everything that moves or gets warm. 
• Quotable Quotes:
  • @dmalenko: It is cool to sit by the ocean, oversee the sunset and think about scalability models for a web app
  • @detroitpro: I have to admit; sometimes I think "This would be easier with a SQL DB" #NoSQL #NotOften #ComplextRelationships #FindingRootObjects

Click to read more ...