The Data-Scope Project - 6PB storage, 500GBytes/sec sequential IO, 20M IOPS, 130TFlops

“Data is everywhere, never be at a single location. Not scalable, not maintainable.” –Alex Szalay

While Galileo played life and death doctrinal games over the mysteries revealed by the telescope, another revolution went unnoticed, the microscope gave up mystery after mystery and nobody yet understood how subversive would be what it revealed. For the first time these new tools of perceptual augmentation allowed humans to peek behind the veil of appearance. A new new eye driving human invention and discovery for hundreds of years.

Data is another material that hides, revealing itself only when we look at different scales and investigate its underlying patterns. If the universe is truly made of information, then we are looking into truly primal stuff. A new eye is needed for Data and an ambitious project called Data-scope aims to be the lens.

A detailed paper on the Data-Scope tells more about what it is:

The Data-Scope is a new scientific instrument, capable of ‘observing’ immense volumes of data from various scientific domains such as astronomy, fluid mechanics, and bioinformatics. The system will have over 6PB of storage, about 500GBytes per sec aggregate sequential IO, about 20M IOPS, and about 130TFlops. The Data-Scope is not a traditional multi-user computing cluster, but a new kind of instrument, that enables people to do science with datasets ranging between 100TB and 1000TB.  There  is a vacuum today in data-intensive scientific computations, similar to the one that lead to the development of the BeoWulf cluster: an inexpensive yet efficient template for data intensive computing in academic environments based on commodity components. The proposed Data-Scope aims to fill this gap.

A very accessible interview by Nicole Hemsoth with Dr. Alexander Szalay, Data-Scope team lead, is available at The New Era of Computing: An Interview with "Dr. Data". Roberto Zicari also has a good interview with Dr. Szalay in Objects in Space vs. Friends in Facebook.

The paper is filled with lots of very specific recommendations on their hardware choices and architecture, so please read the paper for the deeper details. Many BigData operations have the same IO/scale/storage/processing issues Data-Scope is solving, so it’s well worth a look. Here are some of the highlights:

Click to read more ...

Sponsored Post: aiCache, Next Big Sound, ElasticHosts, Red 5 Studios, Attribution Modeling, Logic Monitor, New Relic, AppDynamics, CloudSigma, ManageEngine, Site24x7

Who's Hiring?

• Anybody interested in helping manage a 100+ Linux server deployment? Next Big Sound is a an analytics company for the music industry and is looking someone to help them scale.
• Red 5 Studios. Wanted: DBAs and Programmers interested in MySQL scalability and replication. If interested, please see us here. 

Fun and Informative Events

• Sign up for this free 30-minute webinar exploring how new technology can determine which ads have been seen by users and will discuss the C3 Metrics Labs analysis of over 2 billion impressions. 

Cool Products and Services

• aiCache creates a better user experience by increasing the speed scale and stability of your web-site.
• ElasticHosts award winning cloud server hosting launches across North America. Adding data centers in Los Angeles and Toronto. Free trial.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• New Relic - real user monitoring optimize for humans, not bots. Live application stats, SQL/NoSQL performance, web transactions, proactive notifications. Take 2 minutes to sign up for a free trial.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• CloudSigma. Utility style high performance cloud servers in the US and Europe delivered on all 10GigE networking. Run any OS, take advantage of SSD storage and tailored infrastructure options.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

For a longer description of each sponsor, please read more below...

Click to read more ...

37signals Still Happily Scaling on Moore RAM and SSDs

There are so many architectural ideas swirling in the bit wind these days. Two of the biggest battles are cloud vs. bare metal and RAM vs. disk vs. SSD. 37signals has published two solid articles that are counter hype cycle in their message:

Technologists who grew up when RAM cost $1,000 per megabyte can have a hard time dealing with the luxury of RAM being virtually free.

The progress of technology is throwing an ever greater number of optimizations into the “premature evil” bucket never to be seen again.

37signals made quite a stir with their money shot of the 864GB of RAM they bought for a mere $12K as part of their caching layer for Basecamp. That's a lot of memory for not a lot of money. There's nothing like actually seeing it in the flesh to bring the point home. Does that make Memory Based Architectures a little more appealing?

37signals then followed up with another provocative article: Three years later, Mr. Moore is still letting us punt on database sharding. The gist is scaling up is working for them. RAM is getting cheaper and FusionIO is getting faster, so they've been able to avoid architecture complexifications like sharding. Does that make SSD based architectures a little more appealing?

StackExchange is in much the same position, with a different stack, but with sympatico core ideas and comparable results. The learning: In your transaction oriented features, if you aren't Googleish in your requirements, then scale-up using bare metal, RAM, and SSD may be the way to go. The tug you feel towards the cloud and horizontal scaling may just be a strong consensus wind a blowin'.

Some of the key takeways are: 

Click to read more ...

Stuff The Internet Says On Scalability For January 27, 2012

If you’ve got the time, we’ve got the HighScalability:

• 9nm : IBM's carbon nanotube transistor that outperforms silicon; YouTube: 4 Billion Views/Day; 864GB RAM: 37signals Memcache, $12K
• Quotable Quotes:
  • Chad Dickerson: You can only get growth by feeding opportunities.
  • @launchany: It amazes me how many NoSQL database vendors spend more time detailing their scalability and no time detailing the data model and design
  • Google: Let's make TCP faster.
  • WhatsApp: we are now able to easily push our systems to over 2 million tcp connections!
  • Sidney Dekker: In a complex system…doing the same thing twice will not predictably or necessarily lead to the same results.
  • @Rasmusfjord: Just heard about an Umbraco site running on Azure that handles 20.000 requests /*second*
• Herb Sutter with an epic post, Welcome to the Jungle, touching on a lot of themes we've explored on HighScalability, only in a dramatically more competent way. What's after the current era of multi-core CPUs has played out? Mainstream computers from desktops to ‘smartphones’ are being permanently transformed into heterogeneous supercomputer clusters. Henceforth, a single compute-intensive application will need to harness different kinds of cores, in immense numbers, to get its job done. Different parts of even the same application naturally want to run on different kinds of cores. Applications will need to be at least massively parallel, and ideally able to use non-local cores and heterogeneous cores. Programming languages and systems will increasingly be forced to deal with heterogeneous distributed parallelism. Perhaps our most difficult mental adjustment, however, will be to learn to think of the cloud as part of the mainstream machine – to view all these local and non-local cores as being equally part of the target machine that executes our application, where the network is just another bus that connects us to more cores. If you haven’t done so already, now is the time to take a hard look at the design of your applications, determine what existing features – or, better still, what potential and currently-unimaginable demanding new features – are CPU-sensitive now or are likely to become so soon, and identify how those places could benefit from local and distributed parallelism. Now is also the time for you and your team to grok the requirements, pitfalls, styles, and idioms of hetero-parallel (e.g., GPGPU) and cloud programming.

There's so much more the Internet has to say on Scalability. Click below to be in on all the secrets...

Click to read more ...

Google Goes MoreSQL with Tenzing - SQL Over MapReduce

MoreSQL is a cheeky idea Alex Tatiyants invented in NoSQL No More: Let’s double down with MoreSQL advocating that the cure for NoSQL is not less SQL, but even more SQL. Use SQL everywhere, for everything. This is of course creates yet another NewSQL. Hopefully we've exhausted all tasteful variants of the SQL motif. 

While the post is ironic (I think), Google may be into MoreSQL in a big way, as described in a well written and exceptionally detailed paper: Tenzing - A SQL Implementation On The MapReduce Framework. If you are looking for the secrets of how to get back to the good old days of joins and aggregate operators using a SQL syntax, while enjoying the scalability of NoSQL, this paper is a must read.

Abstract:

Tenzing is a query engine built on top of MapReduce for ad hoc analysis of Google data. Tenzing supports a mostly complete SQL implementation (with several extensions) combined with several key characteristics such as heterogeneity, high performance, scalability, reliability, metadata awareness, low latency, support for columnar storage and structured data, and easy extensibility. Tenzing is currently used internally at Google by 1000+ employees and serves 10000+ queries per day over 1.5 petabytes of compressed data. In this paper, we describe the architecture and implementation of Tenzing, and present benchmarks of typical analytical queries.

The State of NoSQL in 2012

This is a guest post by Siddharth Anand, a senior member of LinkedIn's Distributed Data Systems team. 

Preamble Ramble

If you’ve been working in the online (e.g. internet) space over the past 3 years, you are no stranger to terms like “the cloud” and “NoSQL”.

In 2007, Amazon published a paper on Dynamo. The paper detailed how Dynamo, employing a collection of techniques to solve several problems in fault-tolerance, provided a resilient solution to the on-line shopping cart problem. A few years go by while engineers at AWS toil in relative obscurity at standing up their public cloud.

It’s December 2008 and I am a member of Netflix’s Software Infrastructure team. We’ve just been told that there is something called the “CAP theorem” and because of it, we are to abandon our datacenter in hopes of leveraging Cloud Computing.

Huh?

Click to read more ...

Facebook Timeline: Brought to You by the Power of Denormalization

Facebook Timeline is audacious in scope. It wants to compile a complete scrollable version of your life story from photos, locations, videos, status updates, and everything you do. That could be many decades of data (hopefully) that must stored and made quickly available at any point in time. A huge technical challenge, even for Facebook, which we know are experts in handing big data. And they built it all in 6 months.

Facebook's Ryan Mack shares quite a bit of Timeline's own implementation story in his excellent article: Building Timeline: Scaling up to hold your life story. 

Five big takeaways from the article are:

Click to read more ...

Stuff The Internet Says On Scalability For January 20, 2012

If you’ve got the time, we’ve got the HighScalability:

• Google+: 90 million users; Internet 2011: 2.1 billion Internet users, 1 trillion YouTube views, 5.9 billion mobile subscriptions; Fusion-io: One Billion IOPS; 12 atoms: size of IBM's new memory bit; 32 Million: Stack monthly visitors; Gmail: 350 Million Users; TimTebow: 1.5 million Tweets
• Quotable Quotes:
  • Similarity: There is no canonical schema anymore. Instead you should ask: What high-volume queries will I need to serve with my data? Then work backwards from there.
  • @kvirjee: Dis/Agree? -- "there is no problem but scalability, and architecture is its solution"
  • @robpegoraro : Eternal vigilance can be crowdsourced.
• Didn't Bill Gates say once that 48 bits would always be enough for an ID? Well, Oracle ran out of bits: Fundamental Oracle flaw revealed. 64 bits, that's the ticket, ipv6 went 128 bits. 
• The day Kodak died: We developed the world's first consumer digital camera but we could not get approval to launch or sell it because of fear of the effects on the film market... a huge opportunity missed.

To read much more the Internet has to say on Scalability, please click below...

Click to read more ...

Is it time to get rid of the Linux OS model in the cloud?

You program in a dynamic language, that runs on a JVM, that runs on a OS designed 40 years ago for a completely different purpose, that runs on virtualized hardware. Does this make sense? We've talked about this idea before in Machine VM + Cloud API - Rewriting The Cloud From Scratch, where the vision is to treat cloud virtual hardware as a compiler target, and converting high-level language source code directly into kernels that run on it.

As new technologies evolve the friction created by our old tool chains and architecture models becomes ever more obvious. Take, for example, what a team at UCSD is releasing: a phase-change memory prototype  - a solid state storage device that provides performance thousands of times faster than a conventional hard drive and up to seven times faster than current state-of-the-art solid-state drives (SSDs). However, PCM has access latencies several times slower than DRAM.

This technology has obvious mind blowing implications, but an interesting not so obvious implication is what it says about our current standard datacenter stack. Gary Athens has written an excellent article, Revamping storage performance, spelling it all out in more detail:

Click to read more ...

Paper: Feeding Frenzy: Selectively Materializing Users’ Event Feeds

How do you scale an inbox that has multiple highly volatile feeds? That's a problem faced by social networks like Tumblr, Facebook, and Twitter. Follow a few hundred event sources and it's hard to scalably order an inbox so that you see a correct view as event sources continually publish new events.

This can be considered like a view materialization problem in a database. In a database a view is a virtual table defined by a query that can be accessed like a table. Materialization refers to when the data behind the view is created. If a view is a join on several tables and that join is performed when the view is accessed, then performance will be slow. If the view is precomputed access to the view will be fast, but more resources are used, especially considering that the view may never be accessed.

Your wall/inbox/stream is a view on all the people/things you follow. If you never look at your inbox then materializing the view in your inbox is a waste of resources, yet you'll be mad if displaying your inbox takes forever because all your event streams must be read, sorted, and filtered. 

What's a smart way of handling the materialization problem? That's what is addressed in a very good paper on the subject, Feeding Frenzy: Selectively Materializing Users’ Event Feeds, from researchers at Yahoo!, who found:

The best policy is to decide whether to push or pull events on a per producer/consumer basis. This technique minimizes system cost both for workloads with a high query rate and those with a high event rate. It also exposes a knob, the push threshold, that we can tune to reduce latency in return for higher system cost.

I learned about this paper from Tumblr's Blake Matheny, in an interview with him for a forthcoming post. This is broadly how they handle the inbox problem at Tumblr. More details later.

Abstract from the paper:

Click to read more ...