What does the next generation data center look like?

That's what people at the NGDC Conference get together and talk about. A lot of interesting subjects: data center virtualization HPC & grid; advanced facilitates management and planning; advanced network and services; applications; data center optimization and security; managing and protecting information. The Grid – Distributed Computing at Scale presentation is an interesting one.

Click to read more ...

TypePad Architecture

TypePad is considered the largest paid blogging service in the world. After experience problems because of their meteoric growth, they eventually transitioned to an architecture patterned after their sister company, LiveJournal. Site: http://www.typepad.com/

The Platform

MySQL

Memcached

Perl

MogileFS

Apache

Linux

The Stats

As of 2005 TypePad sends 250mbps of traffic using multiple network pipes for 3TB of traffic a day. They were growing by 10-20% each month. I was unable to find more recent statistics.

The Architecture

Original Architecture: - Single server running Linux, Apache, Postgres, Perl, mod_perl - Storage was NFS on a filer.

A Devastating Crash Caused a New Direction - A RAID controller failed and spewed data across all RAID disks. - The database was corrupted and the backups were corrupted. - Their redundant filers suffered from "split brain" syndrome.

They move to LiveJournal Architecture type architecture which isn't surprising since TypePad and LiveJounral are both owned by Six Apart. - Replicated MySQL clusters partitioned by ID. - A global DB generated globally unique sequence numbers and mapped users to partitions. - Other data was mapped by role.

Highly Available Database Configuration: - A master-master MySQL replication model is used. - The Linux clustering heartbeat was used to failover using virtual IP addresses.

MogileFS is used to serve images.

Perlbal is used as reverse proxy and to load balance requests.

A reliable, asynchronous job dispatch system called TheSchwartz is used to support moblogging, adding comments, future publishing, cache invalidation, and publishing.

Memcached is used to store counts, sets, stats, and heavyweight data.

Migration from the old architecture to the new architecture was tricky: - All users were migrated over without service interruption. - Postgres was removed. - During the migration images were served from NFS and MogileFS.

Benefits of their new architecture: - Can easily add new machines and adjust workload. - More highly available and is cheaply scalable

Lessons Learned

Small details are important.

Every mistake is a learning experience.

Success requires coordination and cooperation.

Related Articles

LiveJournal Architecture.

Linux High Availability.

Click to read more ...

What is the best hosting option?

The questions was extracted from: http://highscalability.com/plentyoffish-architecture#comment-126 For startup like Markus, what is the best hosting option (and grow more later)? host your own server or use ISP co-location option? He still has to pay huge money on the bandwidth with that payload, right?

Click to read more ...

What tech is used to build your favorite site?

Find out with Builtwith.com. It scans a site and guesses how the site is built. I ran it on this site and it said: Apache, Windows, PHP, Adsense, RSS, CSS, Javascript, and UTF-8 encoding. Correct, yet I think it should have guessed Drupal was the CMS and it should have been able to determine which AJAX library is used. Though it's kind of cool to see which sites use PHP and other technologies.

Click to read more ...

Scaling Secret #2: Denormalizing Your Way to Speed and Profit

Alan Watts once observed how after we accepted Descartes' separation of the mind and body we've been trying to smash them back together again ever since when really they were never separate to begin with. The database normalization-denormalization dualism has the same mobius shaped reverberations as Descartes' error. We separate data into a million jagged little pieces and then spend all our time stooping over, picking them and up, and joining them back together again. Normalization has been standard practice now for decades. But times are changing. Many mega-website architects are concluding Watts was right: the data was never separate to begin with. And even more radical, we may even need to store multiple copies of data.

Information Sources

Normalization Is for Sissies by Pat Helland

Data normalization, is it really that good? by Arnon Rotem-Gal-Oz

When Not to Normalize your SQL Database by Dare Obasanjo

MegaData by Joe Gregorio

Audio of talk by Adam Bosworth at the MySQL Users Conference 2005 We normalize data to prevent anomalies. Anomalies are bad things like forgetting to update someone's address in an all the places its been stored when they move. This anomaly happens because the address has been duplicated. So to prevent the anomaly we don't duplicate data. We split everything up so it is stored once and exactly once. Bad things are far less likely to happen if we follow this strategy. And that's a good thing. The process of getting rid of all potential bad things is called normalization and we have a bunch of rules to follow to normalize our data. The price of normalization is that when we want a person's address we have to go find the person and their address in separate operations and bring the data together again. This is called a join. The problem is joins are relatively slow, especially over very large data sets, and if they are slow your website is slow. It takes a long time to get all those separate bits of information off disk and put them all together again. Flickr decided to denormalize because it took 13 Selects to each Insert, Delete or Update. If you say your database is the bottleneck then the finger is pointed back and you and you are asked what you are doing wrong. Have you created proper indexes? Is your schema design good? Is your database efficient? Are you tuning your queries? Have you cached in the database? Have you used views? Have you cached complicated queries in memcached? Can you get more parallel IO out of your database? And all these are valid and good questions. For your typical transactional database these would be your normal paths of attack. But we aren't talking about your normal database. We are talking about web scale services that have to process loads higher than any database can scale to. At some point you need a different approach. Many mega-scale websites with billions of records, petabytes of data, many thousands of simultaneous users, and millions of queries a day are doing is using a sharding scheme and some are even advocating denormalization as the best strategy for architecting the data tier. We sees this with Ebay who moved all significant functionality out of the database and into applications. Flickr shards and replicates their data to reach high performance levels. For Flickr this moves transaction logic back into their application layer, but the win is higher scalability. Joe Gregorio has identified some common themes across these new mega-data systems:

Distributed - The data has to be distributed across multiple machines.

Joinless - No joins, and no referential integrity, at least at the data store level.

De-Normalized - De-normalization is needed if you are avoiding joins.

Transcationless - No transactions It's the web model pushed to the data tier. Ironically, it may take a web model on the back-end to support a web model on the front-end.

The Great Data Ownership Wars: The Database vs. The Application

A not so subtle clue as to who won the data wars is to look at the words used. Data that are split up are considered "normal." Those who keep their data whole are considered "de-normal." All right, that's not what those words mean, but it was to good to pass up. :-) Traditionally the database owns the data. Referential integrity, triggers, stored procedures, and everything else that keeps the data safe and whole is in the database. Applications are prevented from screwing up the data. And this makes sense until you scale. Centralizing all behavior in the database won't mega-scale as the web does, which is why Ebay went completely the other way. Ebay maintains data integrity through a service layer that encapsulates all data access. The service layer handles referential integrity, managing replicated copies, doing joins, and so on. It's more error prone than having the database do all this work, but you are able to do scale past what even the highest end databases can handle. All this sharding and denormalization and duplicating at one levels feels so wrong because it's so different than we were all taught. And unless you are a really large website you probably don't need to worry about this level of complexity. But it's a really fascinating and unexpected evolution in design. Scaling to handle the world wide web requires techniques and strategies that are often at odds with our years of experience. It will be fun to see where it all leads.

Related Articles

Flickr both denormalizes and duplicates data. Horror!

Ebay is the most radical in moving almost all functionality out of the database and into the application.

Plenty of Fish also advocates denormalization as a key strategy.

Hadoop - a framework for running applications on large clusters of commodity hardware using a computational paradigm named map/reduce.

Click to read more ...

Can you profit from the coming Content Delivery Network wars?

Playing like the big boys may be getting cheaper. The big boys, like YouTube, farm the serving of their most popular videos to a third party CDN. A lot of people were surprised YouTube didn't serve all their content themselves, but it makes sense. It allows them to keep up with demand without a large hit for infrastructure build out, much like leasing computers instead of buying them. The problem has been CDNs are expensive. Om Malik reports in Akamai & the CDN Price Wars that may be changing. CDN service could be becoming affordable enough that you might consider using them as part of your scaling strategy. Akamai, once the clear leader in the CDN field, is facing strong competition from the likes of Limelight Networks, Level 3, Internap, CDNetworks, Panther Express and EdgeCast Networks. This commoditization may be bad for their stock prices, but it's good for website builders looking for new scaling strategies. EdgeCast, for example, passes on the cost savings when when their bandwidth costs drop. Other services lock you into fix cost contracts. So competition is good. New cheaper, faster, and easier possibilities for scaling your website are coming on line. Maybe CDNs can help you.

Related Articles

Akamai & the CDN Price Wars

Are CDNs Becoming Commoditized, Again?

YouTube Architecture

EdgeCast Ready To Take On Akamai, Limelight

Click to read more ...

Product: Cacti

Cacti is a network statistics graphing tool designed as a frontend to RRDtool's data storage and graphing functionality. It is intended to be intuitive and easy to use, as well as robust and scalable. It is generally used to graph time-series data like CPU load and bandwidth use. The frontend is written in PHP; it can handle multiple users, each with their own graph sets, so it is sometimes used by web hosting providers (especially dedicated server, virtual private server, and colocation providers) to display bandwidth statistics for their customers. It can be used to configure the data collection itself, allowing certain setups to be monitored without any manual configuration of RRDtool.

Click to read more ...

Try Squid as a Reverse Proxy

This scalability strategy is brought to you by Erik Osterman: My recommendations for anyone dealing with explosive growth on a limited budget with lots of cachable content (e.g. content capable of returning valid expiration headers) is employ a reverse proxy as mentioned in this article. In the last week, we had a site get AP'd, triggering 100K unique visitors to a single IIS server in under 5 hours. It took out the IIS server. Placing a single squid infront of the server handled the entire onslaught with a max server load of 0.10 on a modest Intel IV 3Ghz. It's trivial to implement for anyone interested...

Click to read more ...

Running Hadoop MapReduce on Amazon EC2 and Amazon S3

Excellent article on using Hadoop in Amazon's services environment to solve real problems for very little money. It's excellent because it shows how the stack works together and it actually seems like something a real human could do.

Click to read more ...

Product: Mashery

All APIs are different. At its core, an API provides direct access deep into a web service (lower case - a service that is provided on the web) and turns it into a Web Service (upper case) that people can use as a building block. What makes it an API is the infrastructure that sits in front of it, attracts developers to use it, secures it from misuse and provides the metrics and management needed to turn an internal web service into a Web Service managed through an effective distribution channel, and providing strategic and/or financial benefit. While each API is different, the infrastructure I have described is consistent across virtually all of them, so it is neither economical nor effective to reinvent the wheel for each API someone wants to release. It is similar to the concept of an adserver - all websites have different content and fucntionality, but the concept of selecting and serving an ad, tracking it, and targeting it is pretty consistent across sites; as a result, there are many sites that use a handful of adserver providers. In addition to allowing companies to focus on their core business without having to build peripheral, non-core services, using a third-party service who is focused on providing that service allows you to benefit from ongoing development and enhancement, and from features that would be prohibitively expensive to build for just a single provider. As for an example? check out sites such as developer.trulia.com or developer.compete.com, our first two customers (we have many more, but I like to give props to our early adopters). In addition to documentation and community, they have developer key issuance, instant self-service developer provisioning, usage and rate throttling, and tracking. What you don't see, but our clients enjoy, is a dashboard where they can assign different access levels, rates or limits to each developer on a key-by-key basis, customize error messages and other API parameters, and see detailed reports of API usage on a developer-by-developer or overall basis. Building all of that takes time and money; we offer it as an instantly-deployable on-demand service with no up-front investment, and our customers seem to find it an excellent value. Oren Michels, CEO Mashery

Click to read more ...