Web Accelerators - snake oil or miracle remedy?

Perhaps this question is borderline off-topic but since high scalability solutions often have a global aspect I will give it a try... Have anybody had any experience with different techniques for speeding up their application to places that have a problem with poor ping response time? Ideally I would love to be running only one data center world-wide but one day I know that our sales department will sign up a customer with an unacceptable response time... Could installing a web-accelerator in front of our application extend the reach of our current data center or will we just add complexity and another source of potential errors?

Click to read more ...

S3 Failed Because of Authentication Overload

Being an authentic human being is difficult and apparently authenticating all those S3 requests can be a bit overwhelming as well. Amazon fingered a lot of processor heavy authentication requests as the reason for their downtime: Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types. Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue. Interesting problem. Same thing happens with sites using a lot of SSL. They need to purchase specialized SSL concentrators to handle the load which makes capacity planning a lot trickier and more expensive. In the comments Allen conjectured What caused the problem however was a sudden unexpected surge in a particular type of usage (PUT's and GET's of private files which require cryptographic credentials, rather than GET's of public files that require no credentials). As I understand what Kathrin said, the surge was caused by several large customers suddenly and unexpectedly increasing their usage. Perhaps they all decided to go live with a new service at around the same time, although this is not clear. We see these kinds of bring up problems all the time. The Skype failure was blamed on software updates which caused all nodes to relogin at the same time. Bring up a new disk storage filer and if you aren't load balancing requests all new storage requests will go to that new filer and you'll be down lickity split. Booting is one of the most stressful times on large networks. Bandwidth and CPU all become restricted which causes a cascade of failures. ARP packets can get dropped or lost and machines never get their IP addresses. Packets drop which causes retransmissions which chews up bandwidth which uses CPU and causes more drops. CPUs spike which causes timeouts and reconnects which again spirals everything out of control. When I worked at a settop company we had the scenario of a neighborhood rebooting after a power outage. Lots of houses needing to boot large boot images over asymmetric low bandwidth cable connections. As a fix we broadcasted boot image blocks to all settops. No settops performed your typical boot image download. Worked like a charm. Amazon's problem was a subtle one in a very obscure corner of their system. It's not surprising they found a weakness. But I'm sure Amazon will be back even bigger and better once they get their improvements on line.

Click to read more ...

What's your scalability plan?

How do you plan to scale your system as you reach predictable milestones? This topic came up in another venue and it reminded me about a great comment an Anonymous wrote a while ago and I wanted to make sure that comment didn't get lost. The Anonymous scaling plan was relatively simple and direct: My two cents on what I'm using to start a website from scratch using a single server for now. Later, I'll scale out horizontally when the need arises. Phase 1

Single Server, Dual Quad-Core 2.66, 8gb RAM, 500gb Disk Raid 10

OS: Fedora 8. You could go with pretty much any Linux though. I like Fedora 8 best for servers.

Proxy Cache: Varnish - it is way faster than Squid per my own benchmarks. Squid chokes bigtime.

Web Server: Lighttpd - faster than Apache 2 and easier to configure for me.

Object Cache: Memcached. Very scalable.

PHP Cache: APC. Easy to configure and seems to work fine.

Language: PHP 5 - no bloated frameworks, waste of time for me. You spend too much time trying to figure out the framework instead of getting work done.

Database - MySQL 5. I didn't consider Postgres because I've never used it. There are just a lot more tools available for MySQL. Phase 2

Max Ram out to 64 GB, cache everything Phase 3

Buy load balancer + 2 more servers for front end Varnish/Memcached/Lighttpd.

Use original server as MySQL database server. Phase 4 Depending on my load & usage patterns, scale out the database horizontally with an additional server. I don't expect the db to be a bottleneck for my website as only metadata info is stored there. I'll mostly be serving images stored on the file system. Possibly separate Varnish / Memcached / Lighttpd tier into separate tiers if necessary. But I'll carefully evaluate the situation at this point and scale out appropriately and use CDN for static content if necessary. Phase 5

Max all servers to 64gb of RAM, cache, cache, cache. Phase 6

If I get this far then I'm a multi-millionaire already so I'll replace all of the above machines with whatever the latest and greatest is at that time and keep scaling out. The important point is that I know how to scale each layer when/if the need arises. I'll scale the individual machines when necessary and scale horizontally too. In previous post we also read where ThemBid has a nice simple scalability plan too :

Use Munin to tell when to think about upgrading. When your growth trend will soon cross your resources trend, it's time to do something.

Move MySQL to a separate server. This frees up resources (CPU, disk, memory). What you want to run on this server depend on its capabilities. Maybe run a memcached server on it.

Move to a distributed memory cache using memcached.

Add a MySQL master/slave configuration.

If more webservers are needed us LVS on the front end as a load balancer. The great insight ThemBid had was to use monitoring to predict when performance is hitting a limit so you can take action before the world ends. Take a look at Monitoring for some options. Some examples of how monitoring is used: Feedburner Architecture, Scaling Early Stage Startups, Thembid Architecture, Flickr Architecture, Ebay Architecture. Most problems are pretty predictable, especially if you read this site. Have a plan in mind for what you want to do when you grow. You don't have to do all now, but make the path easier by starting in the right direction now. You'll also be much less stressed when all hell breaks loose.

Click to read more ...

Search the tags across all post

Let suppose i have table which stored tags .Now user can enter keywords and i have to search through all the records in table and find post which contain tags entered by user .user can enter more than 1 keywords. What strategy ,technique i use to search fast .There maybe more than millions records and many users are firing same query. Thanks

Click to read more ...

We want to cache a lot :) How do we go about it ?

We have a lot of dependencies to our SQL databases and we have heard that caching does help a lot as we move into scaling and providing better performance. So the question is what are some reliable software products out there that we could consider in this space ? We want to put a lot of frequently called database calls that do not change frequently into this caching layer. Also what would be an easy way to move only those database changes into the cache as opposed to reloading or pulling it into cache every few mins or hours. We need something smart that would just push changes to the caching layer as it happens. I guess we could build our own, but are there any good reliable products out there ? Please also mention how they play with regards to pricing 'cos that would be a determining factor as well. Thanks

Click to read more ...

Product: rPath - Creating and Managing Virtual Appliances

Update: GIGAOM on rPath Burns EC2 Appliances in a Web Portal. rBuilder adds a portal that lets users turn software into virtual appliances. rPath demoed their virtual appliance management system at Monday's AWS Meetup. What they do is help you build a generic virtual machine image deployable on Amazon, VMWare, Xen and other targets. The idea is to build your software application independent of the underlying operating system and deploy it in your own or someone else's datacenter without worrying about all the details. To put their service in context think of rPath as how you build, deploy, and upgrade images and someone like Right Scale has how you can run and managed a cluster of deployed images. To build a Virtual Appliance you pull together all your packages through their web interface or through a Python based "recipe" system, select a VM target, and "cook" it all into a VM image you can immediately deploy and run. To make this magic happen they use the Conary package manager system and they have their own RedHat compatible OS. One of their major features is a very fine grained package management systems which allows them to perform minimal inplace upgrades of deployed images. The downside is you must use their packaging system and their OS for this to work. Any code you want to install must be installable using their packaging system. There's a free community version available on their website for Open Sourcers.. They make their money from people buying a Virtual Appliance of their build and packaging system and deploying it internally. So you can integrate their Virtual Appliance system as part of your build and deployment infrastructure. As part of your nightly build create appliances and have them automatically deployed to your test jigs. Once testing is complete you can deploy into your datacenter. Their smart upgrade features are very nice for a datacenter. Usually package management during upgrades is a complete nightmare. For cloud deployment I think this feature is less useful as I would simply create a new image, fire up a new instance using the new image, and bring down my old images without the cost of a software upgrade. Of course you still have to worry about protocol and data compatibilities. rPath's Virtual Appliance is kind of a hard idea to really understand because it still ahead of curve of what most people are doing. But I think as we move into a world of multiple clouds we must seed with our images, a layer above the clouds is necessary to manage the whole process. rPath is saying we've already built that layer so you don't have to.

Click to read more ...

Yahoo Live's Scaling Problems Prove: Release Early and Often - Just Don't Screw Up

Tech Crunch chomped down on some initial scaling problems with Yahoo's new live video streaming service Yahoo Live. After a bit of chewing on Yahoo's old bones, TC spat out: If Yahoo cant scale something like this (no matter how much they claim it’s an experiment, it’s still a live service), it shows how far the once brightest star of the online world has fallen. This kind of thinking kills innovation. When there's no room for a few hiccups or a little failure you have to cover your ass so completely nothing new will ever see the light of day. I thought we were supposed to be agile. We are supposed to release early and often. Not every 'i' has to be dotted and not every last router has to be installed before we take the first step of a grand new journey. Get it out there. Let users help you make it better. Listen to customers, make changes, push the new code out, listen some more, and fix problems as they come up. Following this process we'll make something the customer wants and needs without a year spent in a dark room with a cabal of experts trying to perfectly predict an unknowable future. Isn't this what we are supposed to do? Then give people some space to work things out before you declare their world ended and that they are an embarrassment to their kind.

Click to read more ...

clusteradmin.blogspot.com - blog about building and administering clusters

A blog about cluster administration. Written by a System Administrator working at HPC (High Performance Computing) data-center, mostly dealing with PC clusters (100s of servers), SMP machines and distributed installations. The blog concentrates on software/configuration/installation management systems, load balancers, monitoring and other cluster-related solutions.

Click to read more ...

Looking for good business examples of compaines using Hadoop

I have read the blog about Mailtrust/Rackspace as well the interesting things with Google and Yahoo. Who else is using Hadoop/MapReduce to solve business problems. TIA johnmwillis.com

Click to read more ...

SLA monitoring

Hi, We're running a enterprise SaaS solution that currently holds about 700 customers with up to 50.000 users per customer (growing quickly). Our customers have SLA agreements with us that contains guaranteed uptimes, response times and other performance counters. With an increasing number of customers and traffic we find it difficult to provide our customer with actual SLA data. We could set up external probes that monitors certain parts of the application, but this is time consuming with 700 customers (we do it today for our biggest clients). We can also extract data from web logs but they are now approaching about 30-40 GB a day. What we really need is monitoring software that not only focuses on the internal performance counters but also lets us see the application from the customers viewpoint and allows us to aggregate data in different ways. Would the best approach be to develop a custom solution (for instance a distributed app that aggregates data from different logs every night and store them in a data warehouse) or are there products out there that are suitable for a high scalability environment? Any input is greatly appreciated!

Click to read more ...