Gigaspaces curbs latency outliers with Java Real Time

Today, most banks have migrated their internal software development from C/C++ to the Java language because of well-known advantages in development productivity (Java Platform), robustness & reliability (Garbage Collector) and platform independence (Java Bytecode). They may even have gotten better throughput performance through the use of standard architectures and application servers (Java Enterprise Edition). Among the few banking applications that have not been able to benefit yet from the Java revolution, you find the latency-critical applications connected to the trading floor. Why? Because of the unpredictable pauses introduced by the garbage collector which result in significant jitter (variance of execution time). In this post Frederic Pariente Engineering Manager at Sun Microsystems posted a summary of a case study on how the use of Sun Real Time JVM and GigaSpaces was used in the context of of a customer proof-of-concept this summer to ensure guaranteed latency per message under 10 msec, with no code modification to the matching engine.

Click to read more ...

[ANN] New Open Source Cache System

The SHOP.COM Cache System is now available at http://code.google.com/p/sccache/ The SHOP.COM Cache System is an object cache system that... * is an in-process cache and external, shared Cache * is horizontally scalable * stores cached objects to disk * supports associative keys * is non-transactional * can have any size key and any size data * does auto-GC based on TTL * is container and platform neutral It was built in-house at SHOP.COM (by me) and has powered our website for years. We are open-sourcing it in the hope that it will be useful to others and to get some help in its maintenance. This is our first open source attempt and we'd appreciate any help and comments.

Click to read more ...

Scaling MySQL on a 256-way T5440 server using Solaris ZFS and Java 1.7

How to scale MySQL on a 32 core system with 256 threads? Diagonal scalability in a box. An impressive benchmark that achieved more than 79,000 SQL queries per second on a single 4 RU server! Is this real? If so what is the role of good old horizontal scalability? The goals of the benchmark:

1. Reach a high throughput of SQL queries on a 256-way Sun SPARC Enterprise T5440
2. Do it 21st century style i.e. with MySQL and ZFS , not 20th century style i.e with OraSybInf... and VxFS
3. Do it with minimal tuning i.e as close as possible as out-of-the-box

Click to read more ...

Java World Interview on Scalability and Other Java Scalability Secrets

OK, this interview is with me on Java scalability issues. I sound like a bigger idiot than I would like, but I suppose it could have been worse. The Java World folks were very nice and did a good job, so there’s no blame on them :-) The interview went an interesting direction, but there’s more I’d like add and I will do so here. Two major rules regarding Java and scalability have popped out at me:

Java – It’s the platform stupid. Java the language isn’t the big win. What is the big win is the ecosystem building up around the JVM, libraries, and toolsets.

Java – It’s the community stupid. A lot of creativity is being expended on leveraging the Java platform to meet scalability challenges. The amazing community that has built up around Java is pushing Java to the next level in almost every direction imaginable. The fecundity of the Java ecosystem can most readily be seen with the efforts to tame our multi-core future. There’s a multi-core crisis going in case you haven’t heard. It’s all you’ll hear mentioned in the halls of the Pentagon. The CPU wizards have maxed out on clock speed and the only way we can scale is by adding more cores. And we don't know how to do that. At 100 cores common ways of doing things break down. Locks don't scale. Cache contention for shared memory slows us down. Bandwidth on the bus is limited. TLB for managing more memory is in short supply. And we need more high speed network cards to handle faster CPUs. And that’s just at the hardware level. It’s worse for programmers. Locking is just a nightmare. I didn't believe that at first. Early in my career I worked on several multi-core systems. I thought everything was cool. Be careful and it will all work out. But work with a group and it all goes to hell. People add functions, locks, take too much time. Problems like deadlock, priority inversion, and high latency all kill a system. What can we do? As we’ll see, Java and more importantly the JVM have become a platform for many interesting technologies and scalability patterns. Let’s take a look at few.

How Java affects both Performance and Scalability

Peter Williams in this very informative blog post discusses how Java affects both performance and scalability. The main points are:

Java does not scale any better, or worse, than any other general purpose language.

How easily can you increase the number of requests/sec the system can handle derives from the architecture of the system (sharding, caching, avoid database etc).

Java’s culture encourages practices that scale only to medium size system because it encourages techniques that do not scale well : * favors multi-threading * shared state * vertical scaling * large monolithic components * multiple tiers, lots of layers

Java is for performance and is the reason to use Java even though the good performance means it scales pretty well out of the box.

Scaling with Java requires going your own way and bucking the culture to implement more scalable practices. Alain Penders makes some good points in the comments:

Java scalable not because it’s better suitable for building scalable systems, but because how to build scalable systems with it is well understood and the components you need to do so are widely available. Not only are those components widely available, they are available from multiple vendors — some free, some commercial — so you’re never locked in, an aspect that’s extremely valuable when producing commercial software.

Ruby & Rails applications can be built to scale well, but the techniques are not as well understood as for Java. Ron takes an apposing view and says Java is less scalable:

During garbage collection all threads are blocked and the garbage collection time can expand to minutes. These huge latencies effectively limit memory which limits scalability.

Increased garbage collection latencies make Java less useful for application that use heart beats, make real-time trades, etc.

Real Time Java API extensions were developed as a way of addressing garbage collection problems. Greg Frank weighs in with some excellent points:

Java language itself has nothing to do with scalability.

Ron’s comments about garbage collection don’t apply to post 1.4 JVMS.

Old-school J2EE is dead.

The monolithic server is being replaced with advanced patterns based on new technologies like jgroups, ehcache and terracotta.

The true force pushing scalability is the java community itself. Java is merely a convenient and commonly spoken language that allows for the formation of a global community.

We should stop debating language internals and start debating sophisticated design patterns that promote scalability.

The Top 10 Ways to Botch Enterprise Java Application Scalability and Reliability

This is a wonderful presentation by Oracle’s Cameron Purdy. Here’s a PDF. Cameron was CEO of Tangosol before Oracle bought them out. Tangosol made Coherence, a distributed cache. Cameron is a long time prolific contributor to the Java community. His presentation is a must see. He’s both entertaining and technically excellent. The main points he makes in the presentation are:

1. Avoid proprietary features/Believe product claims.

2. Assume the network works

3. Use big JVM machine heaps

4. Use a one-size-fits-all architecture

5. Assume disaster-recovery can be added when it becomes necessary

6. Abuse Abstractions/Avoid abstractions

7. Introduce a single point of bottleneck/Introduce a single point of failure

8. Abuse the database/Avoid the database

9. Assume you are smarter than the infrastructure/Follow the rules blindly

10. Optimize performance assuming that it will translate to scalability/Ignore the potential impact of performance on scalability (and vice-versa). Remember, iff these seem backwards remember these are botching strategies. You'll want to see the presentation to see each point fleshed out in more detail.

Azul

Azul is a Java Compute Appliance and is the ultimate scale-up play for Java. It kind of does what Google App Engine does at the framework level but does it to the JVM at the hardware level. Current standard practice is to deploy Java application across a cluster of commodity servers. Azul does the opposite. It goes big. The most recent release can contain up to 864 processor cores and 768 GB of memory. That’s big. Azul transparently runs unmodified Java applications on their specialized hardware platform which allows even the most mild mannered of Java apps to scale. Their hardware-assisted garbage collector dramatically reduces application pauses and gives access to hundred of gigabytes of RAM. Some very impressive performance improvements are possible. In one case study Breakthrough Scalability of an Application Constrained to an x86 Server, an application was given access to 384 cores and 128 GB of memory on an Azul compute appliance. The result was a 45x improvements in scalability. Scalability was increased along a number of dimensions (quoted from their article):

Thread count: Initially the application was artificially limiting the number of threads that were allowed to execute. With few threads, the application started at 2,200 OPS on Azul (below the 14,000 native score) but by simply increasing the thread pool size and expanding the heap throughput jumped to 60,000 OPS.

Memory locks: When many threads access the same piece of memory, traditional systems force developers to serialize the threads to make sure two threads do not simultaneously change the same memory location (similar to how airlines make sure the same seat is not assigned to two different people.) Azul appliances can detect such collisions in real time and assure correct execution. The removal of two such "hot locks" and a further increase in the number of threads and heap size achieved a new peak of 115,000 OPS.

Logging: Slow applications can afford to maintain logs of unneeded events or even multiple copies of the same information. As throughput increases, the shear volume of such logs make them difficult to analyze and it becomes more practical to log only what is needed. Reducing the amount of logging and addressing a related single threaded bottleneck raised the peak to 350,000 OPS.

More locks: With the above steps taken, a new lock was exposed as a barrier to scalability. Once that was resolved the compute appliance was able to deliver 630,000 OPS – a 45x improvement over the original native performance! If Azul is so cool why aren’t all applications being run on Azul? Buying a completely proprietary hardware platform is too big a risk without a gigantic throbbing pain point. I would like to see Azul open their own cloud so we could get in with low cost and risk.

X10

X10 is not just an inexpensive home automation control system. X10 is also:

A type-safe, modern, parallel, distributed object-oriented language intended to be very easily accessible to Java(TM) programmers. It is targeted to future low-end and high-end systems with nodes that are built out of multi-core SMP chips with non-uniform memory hierarchies, and interconnected in scalable cluster configurations. A member of the Partitioned Global Address Space (PGAS) family of languages, X10 highlights the explicit reification of locality in the form of places; lightweight activities embodied in async, future, foreach, and ateach constructs; constructs for termination detection (finish) and phased computation (clocks); the use of lock-free synchronization (atomic blocks); and the manipulation of global arrays and data structures. An Eclipse-based Integrated Development Environment (IDE) has been developed at IBM for X10 to help further increase programmer productivity by providing state-of-the-art functionality for viewing, editing, navigating, executing, and manipulating X10 programs.

X10 is built on top of Java. X10 adds:

value types, nullable

Array language * Multi-dimensional arrays, * aggregate operations

New concurrency features * activities (async, future), atomic * blocks, clocks

Distribution * places * distributed arrays X10 does not have:

Dynamic class loading

Java’s concurrency features

thread library, volatile,synchronized, wait, notify X10 restricts:

Class variables and static initialization The result is hopefully a language that can be scaled across a cluster of mulit-core processors yet still has the familiar Java syntax and is developed using familiar Java development tools like Eclipse.

Clojure, Jruby, Jpython

More of the "it’s the platform stupid." We have many different and interesting languages being built on the JVM platform.

Jruby

Jruby is an 100% pure-Java implementation of the Ruby programming language.

Jpython

Jpython is an 100% pure-Java implementation of the Python programming language.

Clojure

I first came upon Clojure researching software transactional memory (STM) as solution to the problem of how to create easy to write massively parallel programs. STM is a concurrency control mechanism analogous to database transactions for controlling access to shared memory in concurrent computing. It functions as an alternative to lock-based synchronization. It’s supposed to make writing parallel programs easier. The idea is you can do away with all those nasty locks that cause so many problems. Some have found STM’s performance very disappointing for larger scale applications. And it may ultimately fail simply because everything that inside a transaction boundary is not memory. Programs routinely call out to other services and peripherals. How can STM work in real world environments? STM may not turn out to be the savior of the multi-core world, but Clojure explores some very new Java territory:

Clojure is a dynamic programming language that targets the Java Virtual Machine. It is designed to be a general-purpose language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language - it compiles directly to JVM bytecode, yet remains completely dynamic. Every feature supported by Clojure is supported at runtime. Clojure provides easy access to the Java frameworks, with optional type hints and type inference, to ensure that calls to Java can avoid reflection. Clojure is a dialect of Lisp, and shares with Lisp the code-as-data philosophy and a powerful macro system. Clojure is predominantly a functional programming language, and features a rich set of immutable, persistent data structures. When mutable state is needed, Clojure offers a software transactional memory system and reactive Agent system that ensure clean, correct, multithreaded designs.

Clojure doesn’t fit my aging mental model. The message-passing actor model of Erlang is more my style. Interestingly the difference between Erlang and Clojure is quite purposeful. Clojure wants to be efficient while operating in the same process rather than taking a message passing hit for every operation. Clojure requires specifying an agent as the receiver of a message where I prefer a more publish-subscribe approach where message senders and consumers are independent. Clojure's use of Java threads makes latency difficult to control. And I'm not sure a S-expression based language can ever become popular. But these are relatively minor issues compared to the task of making Java safe for parallelism. Java does OOP well enough, but sucks at concurrency. Clojure is a nice middle-ground that may be able to make concurrency-oriented programming by real humans in Java a reality.

GigaSpaces, GridGain, Terracotta

GigaSpaces, GirdGain, and Terracotta take Java objects and fairly transparently turn them into highly distributed in-memory grids with amazing out-of-the-box functionality. If RAM is the new disk these products make it very easy to hop on the next generation architecture. Again, their ability to do so much behind the scenes is based on the power of the JVM. Try doing any of this with C++. Can’t happen. Only in the Java world will you see so much cutting edge innovation.

Open Services Gateway Initiative (OSGi)

One of the major problems with using Java is it’s a pain to deploy a new release across many machines in a data center. Deploying patches and upgrades requires bringing containers down. There’s also isn’t a good way to architect WAR files. Creating one WAR file with multiple services doesn’t work for developers. Creating N WAR files with one service doesn’t scale for containers. And how do you run multiple versions of the same service in the same container? OSGI is a solution that should make dynamic and high availability deployment of Java web services a reality. OSGi is a dynamic module system for Java. Class loading done right. OSGi defines an architecture for developing and deploying modular applications and libraries by creating a microkernel-style architecture. There’s a core set of modules that make up a basic platform and new functionality is dynamically layered in with a plugin. Using OSGi these plugins are isolated, secured and controlled from the rest of code. The unit of deployment is an OSGi bundle, which is simply a JAR file with an OSGi manifest. This approach allows loosely-coupled application modules to be developed by a team of developers. Everything is kept in-sync using version numbers and module dependency ranges. If you’ve ever worked with Linux this should sound familiar. It’s basically how packages are installed on Linux.

Many Companies are Successfully Using Java

Many companies are using Java on their websites, they just don't use the full stack. Java is the ultimate service implementation language. There’s a trend like Amazon to develop in terms of separate services that are composed together to produce pages. Put up a cluster of applications, load balance between them and you are set. This is a big move for internal architecture. Web services now have external APIs. Those same APIs can be used internally to build your site. Java is great for larger web sites who need to start thinking in terms of services.

Fotolog. Fotolog is the poster boy for java scalability. They migrated from PHP to a new, Java-based architecture that, in addition to giving greater flexibility and reuse for future code, allows for a faster response time while halving the number of servers.

Amazon. Amazon uses a serviced based architecture. They are not stuck with one particular approach. Some places they use jboss/java, but they use only servlets, not the rest of the J2EE stack.

eBay. Use what you like and toss what you don't need. Ebay didn't feel compelled to use full blown J2EE stack. They liked Java and Servlets so that's all they used. You don't have to buy into any framework completely. Just use what works for you.

Mailinator. Handles over 1.2 billion emails a year on one rickity old server. The web application, the email server, and all email storage run in one JVM. Java doesn't have to be slow.

Tailrank. They use use Java, MySQL and Linux for our cluster. Java is a great language for writing crawlers. The library support is pretty solid (though it seems like Java 7 is going to be killer when they add closures).

Flickr. They use Java for their node service and as a FTP daemon and for several other services.

Linkedin. LinkedIn’s architecture has evolved to scale up to 22 million users. LinkedIn is 99% Pure Java. They use a service oriented architecture, java, jetty, eh-cache, and spring. Clients post messages via asynchronous Java Communications API using JMS. The WebApp doesn’t do everything itself anymore: they split parts of its business logic into Services. Each Service has its own domain-specific database (i.e., vertical partitioning).

Amazon’s Dynamo. In Dynamo, each storage node has three main software components: request coordination, membership and failuredetection, and a local persistence engine. All these components are implemented in Java.

GoogleTalk. Google’s IM system implemented in Java.

FeedBurner. A news feed management system written in Java. We’ve covered a lot of ground. We’ve seen how the excesses of old J2EE scalability failures can be routed around with ease using a number of different scalability patterns. We’ve seen some really innovative and amazing products available to Java developers. And we’ve seen a lot of successful websites use Java. Ah, after all that hard work it’s time for another cup of java. Peet’s coffee is my favorite.

Related Articles

Beautiful concurrency by Simon Peyton Jones, Microsoft Research, Cambridge.

Concurrency Shysters by Bryan Cantrill.

Transactions are tomorrow's loads and stores by Nir Shavit .

Software transactional memory: why is it only a research toy? by too many people to list.

Real-world Concurrency by Bryan Cantrill and Jeff Bonwick.

Cantrill and Bonwick get all concurrent-y up in there... by Keith Adams

Fortress - Fortress is a new programming language designed for high-performance computing (HPC) with high programmability.

Azule’s Cliff Click Jr.’s Blog. The folks at Azul have some very interesting blogs. Check them out.

We Don't Know How To Program by Cliff Click Jr.

JavaOne: Cameron Purdy & ‘The Top 10 Ways to Botch Enterprise Java Scalability and Reliability by Ben Teese

Why most large-scale Web sites are not written in Java by Nati Shalom of GigaSpaces. There are also a number very good blogs written by the GigaSpaces folks.

Raising Web Service Updates Efficiency with Dynamic Technologies by Valery Abu-Eid

Building LinkedIn's Next Generation Architecture with OSGi by Yan Pujante

Clojure could be to Concurrency-Oriented Programming what Java was to OOP by Bill Clementson.

Click to read more ...

Sun's High-Performance and Reliable Web Proxy Solution

As individuals and businesses depend on the Web more than ever to conduct business, rapid and reliable content retrieval is critical. Reducing wait time improves productivity and increases user satisfaction. Web proxy technology has emerged as an effective solution to improve performance, help ensure content availability and enhance network security by caching and filtering Web content. The combination of Sun SPARC Enterprise servers with CoolThreads technology and the Sun Java System Web Proxy Server software provides a compelling foundation for a robust Web proxy solution. Sun SPARC Enterprise T1000 and T2000 servers include the UltraSPARC T1 processor with CoolThreads technology, offering six or eight cores with four threads per core. The Sun Java System Web Proxy Server software is highly threaded and takes advantage of the large number of threads supported by Sun UltraSPARC T1 processors with CoolThreads technology. Together, these products provide a highly scalable solution that accommodates a large number of requests, addresses peak loads, and provides future headroom for growth. This document explores the use of a Sun SPARC Enterprise T1000 server and the Sun Java System Web Proxy Server software as a replacement for an existing Web proxy implementation that used the SQUID Web proxy server software deployed on x86 servers.

Click to read more ...

Google Architecture

Update 2: Sorting 1 PB with MapReduce. PB is not peanut-butter-and-jelly misspelled. It's 1 petabyte or 1000 terabytes or 1,000,000 gigabytes. It took six hours and two minutes to sort 1PB (10 trillion 100-byte records) on 4,000 computers and the results were replicated thrice on 48,000 disks. Update: Greg Linden points to a new Google article MapReduce: simplified data processing on large clusters. Some interesting stats: 100k MapReduce jobs are executed each day; more than 20 petabytes of data are processed per day; more than 10k MapReduce programs have been implemented; machines are dual processor with gigabit ethernet and 4-8 GB of memory. Google is the King of scalability. Everyone knows Google for their large, sophisticated, and fast searching, but they don't just shine in search. Their platform approach to building scalable applications allows them to roll out internet scale applications at an alarmingly high competition crushing rate. Their goal is always to build a higher performing higher scaling infrastructure to support their products. How do they do that?

Information Sources

Video: Building Large Systems at Google

Google Lab: The Google File System

Google Lab: MapReduce: Simplified Data Processing on Large Clusters

Google Lab: BigTable.

Video: BigTable: A Distributed Structured Storage System.

Google Lab: The Chubby Lock Service for Loosely-Coupled Distributed Systems.

How Google Works by David Carr in Baseline Magazine.

Google Lab: Interpreting the Data: Parallel Analysis with Sawzall.

Dare Obasonjo's Notes on the scalability conference.

Platform

Linux

A large diversity of languages: Python, Java, C++

What's Inside?

The Stats

Estimated 450,000 low-cost commodity servers in 2006

In 2005 Google indexed 8 billion web pages. By now, who knows?

Currently there over 200 GFS clusters at Google. A cluster can have 1000 or even 5000 machines. Pools of tens of thousands of machines retrieve data from GFS clusters that run as large as 5 petabytes of storage. Aggregate read/write throughput can be as high as 40 gigabytes/second across the cluster.

Currently there are 6000 MapReduce applications at Google and hundreds of new applications are being written each month.

BigTable scales to store billions of URLs, hundreds of terabytes of satellite imagery, and preferences for hundreds of millions of users.

The Stack

Google visualizes their infrastructure as a three layer stack:

Products: search, advertising, email, maps, video, chat, blogger

Distributed Systems Infrastructure: GFS, MapReduce, and BigTable.

Computing Platforms: a bunch of machines in a bunch of different data centers

Make sure easy for folks in the company to deploy at a low cost.

Look at price performance data on a per application basis. Spend more money on hardware to not lose log data, but spend less on other types of data. Having said that, they don't lose data.

Reliable Storage Mechanism with GFS (Google File System)

Reliable scalable storage is a core need of any application. GFS is their core storage platform.

Google File System - large distributed log structured file system in which they throw in a lot of data.

Why build it instead of using something off the shelf? Because they control everything and it's the platform that distinguishes them from everyone else. They required: - high reliability across data centers - scalability to thousands of network nodes - huge read/write bandwidth requirements - support for large blocks of data which are gigabytes in size. - efficient distribution of operations across nodes to reduce bottlenecks

System has master and chunk servers. - Master servers keep metadata on the various data files. Data are stored in the file system in 64MB chunks. Clients talk to the master servers to perform metadata operations on files and to locate the chunk server that contains the needed they need on disk. - Chunk servers store the actual data on disk. Each chunk is replicated across three different chunk servers to create redundancy in case of server crashes. Once directed by a master server, a client application retrieves files directly from chunk servers.

A new application coming on line can use an existing GFS cluster or they can make your own. It would be interesting to understand the provisioning process they use across their data centers.

Key is enough infrastructure to make sure people have choices for their application. GFS can be tuned to fit individual application needs.

Do Something With the Data Using MapReduce

Now that you have a good storage system, how do you do anything with so much data? Let's say you have many TBs of data stored across a 1000 machines. Databases don't scale or cost effectively scale to those levels. That's where MapReduce comes in.

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model. Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Why use MapReduce? - Nice way to partition tasks across lots of machines. - Handle machine failure. - Works across different application types, like search and ads. Almost every application has map reduce type operations. You can precompute useful data, find word counts, sort TBs of data, etc. - Computation can automatically move closer to the IO source.

The MapReduce system has three different types of servers. - The Master server assigns user tasks to map and reduce servers. It also tracks the state of the tasks. - The Map servers accept user input and performs map operations on them. The results are written to intermediate files - The Reduce servers accepts intermediate files produced by map servers and performs reduce operation on them.

For example, you want to count the number of words in all web pages. You would feed all the pages stored on GFS into MapReduce. This would all be happening on 1000s of machines simultaneously and all the coordination, job scheduling, failure handling, and data transport would be done automatically. - The steps look like: GFS -> Map -> Shuffle -> Reduction -> Store Results back into GFS. - In MapReduce a map maps one view of data to another, producing a key value pair, which in our example is word and count. - Shuffling aggregates key types. - The reductions sums up all the key value pairs and produces the final answer.

The Google indexing pipeline has about 20 different map reductions. A pipeline looks at data with a whole bunch of records and aggregating keys. A second map-reduce comes a long, takes that result and does something else. And so on.

Programs can be very small. As little as 20 to 50 lines of code.

One problem is stragglers. A straggler is a computation that is going slower than others which holds up everyone. Stragglers may happen because of slow IO (say a bad controller) or from a temporary CPU spike. The solution is to run multiple of the same computations and when one is done kill all the rest.

Data transferred between map and reduce servers is compressed. The idea is that because servers aren't CPU bound it makes sense to spend on data compression and decompression in order to save on bandwidth and I/O.

Storing Structured Data in BigTable

BigTable is a large scale, fault tolerant, self managing system that includes terabytes of memory and petabytes of storage. It can handle millions of reads/writes per second.

BigTable is a distributed hash mechanism built on top of GFS. It is not a relational database. It doesn't support joins or SQL type queries.

It provides lookup mechanism to access structured data by key. GFS stores opaque data and many applications needs has data with structure.

Commercial databases simply don't scale to this level and they don't work across 1000s machines.

By controlling their own low level storage system Google gets more control and leverage to improve their system. For example, if they want features that make cross data center operations easier, they can build it in.

Machines can be added and deleted while the system is running and the whole system just works.

Each data item is stored in a cell which can be accessed using a row key, column key, or timestamp.

Each row is stored in one or more tablets. A tablet is a sequence of 64KB blocks in a data format called SSTable.

BigTable has three different types of servers: - The Master servers assign tablets to tablet servers. They track where tablets are located and redistributes tasks as needed. - The Tablet servers process read/write requests for tablets. They split tablets when they exceed size limits (usually 100MB - 200MB). When a tablet server fails, then a 100 tablet servers each pickup 1 new tablet and the system recovers. - The Lock servers form a distributed lock service. Operations like opening a tablet for writing, Master aribtration, and access control checking require mutual exclusion.

A locality group can be used to physically store related bits of data together for better locality of reference.

Tablets are cached in RAM as much as possible.

Hardware

When you have a lot of machines how do you build them to be cost efficient and use power efficiently?

Use ultra cheap commodity hardware and built software on top to handle their death.

A 1,000-fold computer power increase can be had for a 33 times lower cost if you you use a failure-prone infrastructure rather than an infrastructure built on highly reliable components. You must build reliability on top of unreliability for this strategy to work.

Linux, in-house rack design, PC class mother boards, low end storage.

Price per wattage on performance basis isn't getting better. Have huge power and cooling issues.

Use a mix of collocation and their own data centers.

Misc

Push changes out quickly rather than wait for QA.

Libraries are the predominant way of building programs.

Some are applications are provided as services, like crawling.

An infrastructure handles versioning of applications so they can be release without a fear of breaking things.

Future Directions for Google

Support geo-distributed clusters.

Create a single global namespace for all data. Currently data is segregated by cluster.

More and better automated migration of data and computation.

Solve consistency issues that happen when you couple wide area replication with network partitioning (e.g. keeping services up even if a cluster goes offline for maintenance or due to some sort of outage).

Lessons Learned

Infrastructure can be a competitive advantage. It certainly is for Google. They can roll out new internet services faster, cheaper, and at scale at few others can compete with. Many companies take a completely different approach. Many companies treat infrastructure as an expense. Each group will use completely different technologies and their will be little planning and commonality of how to build systems. Google thinks of themselves as a systems engineering company, which is a very refreshing way to look at building software.

Spanning multiple data centers is still an unsolved problem. Most websites are in one and at most two data centers. How to fully distribute a website across a set of data centers is, shall we say, tricky.

Take a look at Hadoop (product) if you don't have the time to rebuild all this infrastructure from scratch yourself. Hadoop is an open source implementation of many of the same ideas presented here.

An under appreciated advantage of a platform approach is junior developers can quickly and confidently create robust applications on top of the platform. If every project needs to create the same distributed infrastructure wheel you'll run into difficulty because the people who know how to do this are relatively rare.

Synergy isn't always crap. By making all parts of a system work together an improvement in one helps them all. Improve the file system and everyone benefits immediately and transparently. If every project uses a different file system then there's no continual incremental improvement across the entire stack.

Build self-managing systems that work without having to take the system down. This allows you to more easily rebalance resources across servers, add more capacity dynamically, bring machines off line, and gracefully handle upgrades.

Create a Darwinian infrastructure. Perform time consuming operation in parallel and take the winner.

Don't ignore the Academy. Academia has a lot of good ideas that don't get translated into production environments. Most of what Google has done has prior art, just not prior large scale deployment.

Consider compression. Compression is a good option when you have a lot of CPU to throw around and limited IO.

Click to read more ...

Olio Web2.0 Toolkit - Evaluate Web Technologies and Tools

How do you evaluate and decide which web technologies (and there are myriads out there) to use for your new web application, which one potentially gives you the best performance, which one will likely give you the shortest time-to-market? The Apache incubator project Olio might help. Olio is a is an open source web 2.0 toolkit to help evaluate the suitability, functionality and performance of web technologies. Olio defines an example web2.0 application (an events site somewhat like yahoo.com/upcoming) and provides three initial implementations : PHP, Java EE and RubyOnRails (ROR). The toolkit also defines ways to drive load against the application in order to measure performance. Apache Olio could be used to

• Understand how to use various web 2.0 technologies such as AJAX, memcached, mogileFS etc. Use the code in the application to understand the subtle complexities involved and how to get around issues with these technologies.
• Evaluate the differences in the three implementations: php, ruby and java to understand which might best work for your situation.
• Within each implementation, evaluate different infrastructure technologies by changing the servers used (e.g: apache vs lighttpd, mysql vs postgre, ruby vs Jruby etc.)
• Drive load against the application to evaluate the performance and scalability of the chosen platform.
• Experiment with different algorithms (e.g. memcache locking, a different DB access API) by replacing portions of code in the application.

Olio started it's life as the web2.0kit developed by Sun Microsystems in colloboration with U.C. Berkeley RAD Lab and was presented on Velocity2008.

• Sun is using Olio to create deployment, reference architecture and performance tuning guides for their customers such as: An Open Source Web Solution - Lighttpd Web Server and Chip Multithreading Technology
• U.C. Berkeley is using Olio in a variety of research projects, the first of which is Cloudstone.
• A few web2.0 companies are using Olio as a training tool to teach their new developers PHP and Rails.

Click to read more ...

Server load balancing architectures, Part 1: Transport-level load balancing

Server farms achieve high scalability and high availability through server load balancing, a technique that makes the server farm appear to clients as a single server. In this two-part article, Gregor Roth explores server load balancing architectures, with a focus on open source solutions. Part 1 covers server load balancing basics and discusses the pros and cons of transport-level server load balancing.

The barrier to entry for many Internet companies is low. Anyone with a good idea can develop a small application, purchase a domain name, and set up a few PC-based servers to handle incoming traffic. The initial investment is small, so the start-up risk is minimal. But a successful low-cost infrastructure can become a serious problem quickly. A single server that handles all the incoming requests may not have the capacity to handle high traffic volumes once the business becomes popular. In such a situations companies often start to scale up: they upgrade the existing infrastructure by buying a larger box with more processors or add more memory to run the applications.

Read the rest of the article on JavaWorld.

Click to read more ...

Need help with your Hadoop deployment? This company may help!

A group of top Silicon Valley engineers (ex-Yahoo, Facebook, Google) have come together to launch a new startup called Cloudera. Not yet launched, it intends to help other companies adopt a promising software platform called Hadoop.

Hadoop is an open-source software project (written in Java) designed to let developers write and run applications that process huge amounts of data. While it could potentially improve a wide range of other software, the ecosystem supporting its implementation is still developing. Which is where Cloudera hopes to make a place for itself.

More on Hadoop: It uses the Google-introduced MapReduce systems framework that divides applications into small blocks of work, creating multiple replicas of data blocks that it places on various computer nodes.

It is already in use at large companies like Yahoo.

Read more about Cloudera here.

Click to read more ...

EE-Appserver Clustering OR Terracota OR Coherence OR something else?

Hi, I am very glad that this site exists, as I have learned more about clustering on this site than for quite some time reading stuff elsewhere. Oftentimes, one can find lots of material about clustering, but the practical real-life information is missing. Not so wih this site. I am currently planning the development of an application which has a lot of enterprise features and requirements. On the other side (if the tiny chance of success might strike us), this application would not be an in-house application of a financial institution, or something like that, but some kind of communit/web 2.0 web site. Thus it is an enterprise application with (hopefully, but surely unlikely) the user numbers of a social networking site. Each user initiated transaction involves huge resssources business logic wise (including insane amounts of encryption oprations). Of course, I do not intend to induldge into premature scaling, but to invest every minute I have into the implementation of business logic features. Nevertheless, I do not want to make some extremely bad choices which would force a complete reimplementation straight after the first tiny success - i.e. I want to start with the right technology and architecture, but wait with the implementation of the scalability and high availyability features. Because of the enterprise aspects of this software, my first thought was to use Java SE 6 and Java EE 5 technologies only in order to get all the JEE features and to be vendor independent at the same time. For implementation and testing purposes I thought of Glassfish v2UR2, Postgresql 8.3 and Solaris 10. As all of the major JEE-Appserver vendors advertise the clustering capabilities, I thought that this could not be a bad move. Hopefully, Glassfish would provide HA and scalability, if not there would always be Geronimo, JBoss, Weblogic, or Websphere. Now it seems that there are vast differences between different products: - JEE-Application servers are scaling only to some degree(?). It seems that JEE is almost exclusively used for enterprise applications like SAP ERP or applications at financial institutions? Therefore, there is no need for extreme scalability. - Terracotta seems to be very nice, as one do not have to learn the insanely huge JEE-technology stack, but can just write a mostly Java-SE-only threaded application(?). But Terracotta does not seem to scale very well either (bottleneck with write-operations caused by the master-worker architecture?) and we would be dependend on the future of the Terracotta Corporation. JEE on the other side is vendor neutral. - Oracle Coherence. This product seems to be the best distributed caching product and the holy grail of scalability(?). But it is oracle-expensive. Absolutely nothing for a tiny start-up with no financing. JEE is vendor neutral and thus possibly much cheaper. Do you think that it is possible that one could produce a JEE-Architecture which could provide massive scalability (many hundreds of AppServer) using only the Glassfish clustering features? Or am I on a completely wrong track? Do we have to plan for Oracle Coherence usage? Are there other possibilities? Thanks a lot for any opinions or hints! regards, mike

Click to read more ...