Small Imrovements, makers of a hosted, lightweight feedback platform, have written an excellent article on Performance issues on GAE, and how we resolved them. They show how they trimmed most of their requests to between 300ms and 800ms, some still take 2 seconds when memcache is stale, and others clock in at 150ms. Not zippy overall, but acceptable, especially if you really like GAE's PaaS promise.
What's tricky with PaaS is if your performance is poor, there's often not a lot you can do about it. But the folks at Small Improvements have been clever and diligent, giving many specific details and timings. Though their advice is specifically for GAE, it will apply to a lot of different situations as well.
Here are the 15 ways they made small performance improvements:
- Understand App Engine has bad days. App Engine can have bad days where performance can degrade. Your design needs to take this potential for high latency variability into account. Don't always assume the best case.
- Always prefer GETs to QUERYs. Getting directly by ID takes 250ms. Querying by key took 800ms.
- Combine Batch GETs with ThreadLocal caches. Instead of getting keys individually, collect them into a set and do them in one batch GET. Store the results into a HashMap and then use them in rendering. In a modular application you want to avoid passing all these little bits of state around, so stick them in ThreadLocal so any module can access them at any time.
- Memcaching Query results. Using the appstats tool to figure out which queries are used often enough to cache.
- Pre login memcache warming. Fire off an Ajax request before users have fully logged into to cause their data to be cached.
- Post login memcache warming. Once a user logs in, cache the next 5 pages they are likely to visit.
- Interstitial pages when speed is abysmal. Measure initial queries, if they are slow, then send them to alternate faster page instead. While that page is being viewed, cache the data for the previous query.
- If you couldn't memcache it, load it asynchronously. Graceful degradation. If a query is slow make a decision not to render that part of the page. Render it what the query completes.
- Inactivity rewarming. Recache data that would have expired after a period of inactivity so it's there if the user returns.
- Denormalise data. For a query like "who are this managers' direct subordinates" that would take 300ms, store the direct reports with the managers and recalculate when the relationships changes. The results is now accessible with a GET and will be much faster.
- Use JARs! Their biggest win. With thousands of classes loading classes took up to 400ms because of the disk access involved. JARing all the classes together improved load times made the system fly.
- Warmup requests. Use GAE's warmup feature and stick as much work as you can there. Exercise code paths, write data to memcache, render some core UI elements, run key queries, simulate a login, make some date-time calculations. All this warms up the VM and the data.
- Defer writes. Queue writes to the task queue if they will take a lot of time or if a lot of queries are already being made.
- Asynchronous mails. Send email from task queues.
- Asynchronous queries. Run queries in parallel.