Hot Scalability Links for June 16, 2010

• You're Doing it Wrong by Poul-Henning Kamp. Don't look so guilty, he's not talking about you know what, he's talking about writing high-performance server programs: Not just wrong as in not perfect, but wrong as in wasting half, or more, of your performance. What good is an O(log2(n)) algorithm if those operations cause page faults and slow disk operations? For most relevant datasets an O(n) or even an O(n^2) algorithm, which avoids page faults, will run circles around it. 
• A Microsoft Windows Azure primer: the basics by Peter Bright. Nice article explaining the basics of Azure and how it compares to Google and Amazon.
• A call to change the name from NoSQL to Postmodern Databases. Interesting idea, but the problem is the same one I have for Postmodern Art, when is it? I always feel like I'm in the post-post modern period, yet for art it's really in the early 1900s. Let's save future developers from this existential time crisis.
• Constructions from Dots and Lines by Marko A. Rodriguez, Peter Neubauer. Delightful yet in-depth explanation of the complex world of graph data structures. To make use of the graphs beyond simply representing their explicit structure, graph traversal frameworks and algorithms have been developed in order to shape graphs by driving the evolution of the entities that they model—e.g. humans and their relationships to one another and the objects of their world

Click to read more ...

Hot Scalability Links for June 3, 2010

• How Big is a Yottabyte? Not so big that the NSA can't hope to store it says CrunchGear: There are a thousand gigabytes in a terabyte, a thousand terabytes in a petabyte, a thousand petabytes in an exabyte, a thousand exabytes in a zettabyte, and a thousand zettabytes in a yottabyte. In other words, a yottabyte is 1,000,000,000,000,000GB.
• The CMS data aggregation system. The Large Hadron Collider project is using MongoDB as a cache. Here we discuss a new data aggregation system which consumes, indexes and delivers information from different relational and non-relational data sources to answer cross data-service queries and explore meta-data associated with petabytes of experimental data.
• Google I/O 2010 Videos are up available (many of them anyway). You might be particularly interested in Google Storage for Developers, Building high-throughput data pipelines with Google App Engine, Batch data processing with App Engine, BigQuery and Prediction APIs, Measure in milliseconds redux: Meet Speed Tracer
• Scale at Facebook by Director of Engineering, Aditya Agarwal. You can't scale Facebook using traditional horizontal partitioning. People make friends across many networks. Every new user can potentially access any other user. There's no way to cut the data to effectively partition the data such that access is within that particular partition.

Click to read more ...

Hot Scalability Links for May 14, 2010

Lots of good ones this week...

• Scalability, Availability & Stability Patterns. Jonas Boner has 197 slides covering a very wide range of scalability topics. One stop scalability shopping.
• Horizontal Scalability via Transient, Shardable, and Share-Nothing Resources. Heroku's Adam Wiggins shares what they've learned about scaling based on their experiences building a cloud platform and the hundreds of apps running on it. He describes the next generation architecture he thinks all software should follow in the future.
• Scalability of the Hadoop Distributed File System. Konstantin V. Shvachko writes a great post analyzing if the limitations imposed on a distributed file system by the single-node namespace server architecture can support 100,000 clients and petabytes of files.
• Cassandra by Example. Eric Evans created a nice Cassandra tutorial using building a Twitter clone as an example. Many people want to see more data modeling examples. Here you are.

Click to read more ...

Hot Scalability Links for April 30, 2010

• I Want a New Data Store. Jeremy Zawodny of Craigslist wants a new database, one that can do what it should: perform alter table operations faster, has efficient queries when most of the data is on disk and not in RAM, and matches their data that now looks more document oriented than relational. A lot of people willing to help.
• Computer Science Unplugged. An extensive collection of free resources that teach principles of Computer Science such as binary numbers, algorithms and data compression through engaging games and puzzles that use cards, string, crayons and lots of running around. And it's free! Fascinating Interview with Tim Bell on teaching complex computing concepts, creating makers not just users, and how to change schools. From O'Reilly Radar. 
• Akamai’s Network Now Pushes Terabits of Data Every Second. Akamai handles 12 million requests per second, logs more than 500 billion requests for content per day, and sends 3.45 terabits per second of data.

Click to read more ...

Hot Scalability Links for April 8, 2010

1. Scalability porn (SFW). Real time meter for the number of ads being served by doubleclick. Amazing. A constant ~390,000 impressions a second are being served and 25 trillion since 1996. Thanks to Mike Rhoads for title idea.
2. Scalability? Don't worry. Application complexity? Worry by Joe McKendrick. The next challenge on enterprise agendas: application complexity. This is something that lots of hardware — whether from the cloud or internal data center — cannot fix
3. Leo Laporte and Steve Gibson talked about how the iPad was a denial of service attack on UPS delivery schedules. UPS trucks were filled with iPads.
4. Cassandra: Fact vs fiction. Jonathan Ellies puts the beatdown on Cassandra misinformation. Don't you dare say Cassandra can't work across datacenters!
5. JIT'd code calling conventions. Cliff Click Jr shows how Java’s calling convention can match compiled C code in speed, but allows for the flexibility of calling (code,slow) non-JIT'd code. Some assembly code required.
6. Stonebraker on CAP Theorem and Databases. James Hamilton: Don’t throw full consistency out too early. For many applications, it is both affordable and helps reduce application implementation errors.

Click to read more ...

Hot Scalability Links for April 1, 2010

1. Why NoSQL Will Not Die. Stephan Schmidt explains why you may wait a long time for NoSQL to go to that great bit bucket in the sky.
2. DBMS Musings: Distinguishing Two Major Types of Column-Stores by Daniel Abadi. I have noticed that Bigtable, HBase, Hypertable, and Cassandra are being called column-stores with increasing frequency, due to their ability to store and access column families separately. This makes them appear to be in the same category as column-stores such as Sybase IQ, C-Store, Vertica, VectorWise, MonetDB, ParAccel, and Infobright, which also are able to access columns separately.
3. Cloud Economics, By The Square Foot by Rich Miller. But cloud computing offers a middle path, offering cost and usability advantages for customers, as well as an attractive return for providers.
4. PostgreSQL: meet your queue by Theo Schlossnagle. I really think that cueing your database to publish over AMQP is the bees knees and it turns out I wasn't alone!
5. Scaling GIS Data in Non-relational Data Store by Mike Malone. How SimpleGEO uses NoSQL and other technologies. Yes, the still use memcached. Caching ain’t going anywhere.
6. CLTV45: The Evolution of the Graph Data Structure from Research to Production. In this recording from “NoSQL Live Boston” we learn how Graph Data Structures evolved from research into production.
7. Spanner: Google’s next Massive Storage and Computation infrastructure by Royans. MapReduce, Bigtable and Pregel have their origins in Google and they all deal with “large systems”. But all of them may be dwarfed in size and complexity by a new project Google is working on. .

Click to read more ...

Hot Scalability Links for March 19, 2010

1. The Changelog Episode 0.1.8 - NoSQL Smackdown! This podcast was recorded at SXSW and features some energetic trash talking by: Stu Hood from Cassandra, Jan Lehnardt from CouchDB, Wynn Netherland from The Changelog, subbing for MongoDB, Werner Vogels CTO at Amazon. It's fun hearing these guys step out of their sober advocacy roles and let loose a little with why they are great and the other products suck, hard.
2. Algorithmic Graph Theory . It's FREE! A GNU-FDL book on algorithmic graph theory by David Joyner, Minh Van Nguyen, and Nathann Cohen. This is an introductory book on algorithmic graph theory.
3. HBase vs Cassandra: why we moved by Dominic Williams.
4. Benchmarking Cloud Serving Systems with YCSB by lots of people from Yahoo! Research. We present the Yahoo! Cloud Serving Benchmark (YCSB) framework, with the goal of facilitating performance comparisons of the new generation of cloud data serving systems. We define a core set of benchmarks and report re- sults for four widely used systems: Cassandra, HBase, Yahoo!’s PNUTS, and a simple sharded MySQL implementation.
5. All recordings from NoSQL Live Boston now online!. It's almost like you were there.

Click to read more ...

Hot Scalability Links for March 3, 2010

Getting Real about NoSQL and the SQL-Isn't-Scalable Lie by Dennis Forbes. Buoyed by Canada's Olympic success, Dennis is going for the gold in that least real of sports, the NoSQL vs SQL pursuit.

Design Patterns for Distributed Non-Relational Databases by Todd Lipcon. Great coverage of consistent hashing, consitency models, data models, storage layouts, log-structured merge trees, and gossip protocols.

Brewer's CAP Conjecture is False. Jim Starkey makes the case the CAP is crap.

Kaazing Pushes Web Sockets to Make Browsers Real Time. Bi-directional communication comes to the web, but shouldn't sockets be able to accept connections too?

4 Months with Cassandra, a love story. Cloudkick likes its Linear scalability, Massive write performance, Low operational costs. We'll likely keep moving more data into Cassandra as we need to, but for some data the ability to write arbitrary SQL queries is still very useful. 

Click to read more ...

Hot Scalability Links for February 24, 2010

Cassandra @ Twitter: An Interview with Ryan King. Great interview by Alex Popescu on Twitter's thought process for switching to Cassandra. Twitter chose Cassandra because it had more big system features out of the box. Is that Cassandra FTW?

I Had Downtime Today. Here’s What I’m Doing About It by Patrick McKenzie. Awesome deep dive into went wrong with Bingo Card Creator. Sh*t happens. How do you design a process to help prevent it from happening and how do you deal with problems with integrity when they do?

High Availability Principle : Request Queueing by Ashish Soni. Queue request to ride out traffic spikes: 1) Request Queuing allows your system to operate at optimal throughput. 2) Your users only experience linear degradation versus exponential degradation. 3) Your system experiences NO degradation.

pfffft twatter tweeter by Knowbuddy. The reason you should care [about NoSQL] is because now you have more options--you're not stuck trying to wedge your system into a relational model if you don't want to. And isn't /. all about freedom of choice?

Wordpress, Varnish and Edge Side Includes. Using Varnish to go from .63 requests per second to 537.44 requests per second.

Facebook’s Petabyte Scale Data Warehouse using Hive and Hadoop by Ashish Thusoo and Namit Jain. How does Facebook deal with 12 TB of compressed new data everyday? They get a bad case of the Hives.

Click to read more ...

Hot Scalability Links for February 12, 2010

1. My Life With Hbase by Lars George. The hardscabble tale of Hbase's growth from infancy to maturity. A very good introduction and overview of Hbase.
2. NoSQL Alternatives -- Common Principles and Patterns for Building Scalable Applications. Explore the common principles behind the major NOSQL alternatives and how they compared with traditional database approach in terms of consistency, transaction and query semantics. We will also explore how we can make the transition between the two models smoothers through the support of standard interfaces such as JPA.
3. Moore’s Law: The Future of Cloud Computing from the Bottom Up. Will Intel's 48 mega core chip change the world or be just another Spruce Goose?
4. Rent or Own: Amazon EC2 vs. Colocation Comparison for Hadoop Clusters. It's much cheaper to own when you have a large relatively fixed size cluster and can find really cheap labor to maintain it all.
5. A cloud in a plug - brilliant. A tiny, low-power, low-cost home server and NAS device powered by Tonido software that allows you to access your apps, files, music and media from anywhere.
6. Seeking A Database That Doesn't Suck by Pixy Misa. Quick recap of databases that suck - or at least, suck for my purposes - and some that I'm still investigating.

Click to read more ...