15 Ways to Make Your Application Feel More Responsive under Google App Engine

Small Imrovements, makers of a hosted, lightweight feedback platform, have written an excellent article on Performance issues on GAE, and how we resolved them. They show how they trimmed most of their requests to between 300ms and 800ms, some still take 2 seconds when memcache is stale, and others clock in at 150ms. Not zippy overall, but acceptable, especially if you really like GAE's PaaS promise.

What's tricky with PaaS is if your performance is poor, there's often not a lot you can do about it. But the folks at Small Improvements have been clever and diligent, giving many specific details and timings. Though their advice is specifically for GAE, it will apply to a lot of different situations as well.

Here are the 15 ways they made small performance improvements: 

1. Understand App Engine has bad days. App Engine can have bad days where performance can degrade. Your design needs to take this potential for high latency variability into account. Don't always assume the best case.
2. Always prefer GETs to QUERYs. Getting directly by ID takes 250ms. Querying by key took 800ms.
3. Combine Batch GETs with ThreadLocal caches. Instead of getting keys individually, collect them into a set and do them in one batch GET. Store the results into a HashMap and then use them in rendering. In a modular application you want to avoid passing all these little bits of state around, so stick them in ThreadLocal so any module can access them at any time. 
4. Memcaching Query results. Using the appstats tool to figure out which queries are used often enough to cache.
5. Pre login memcache warming. Fire off an Ajax request before users have fully logged into to cause their data to be cached.
6. Post login memcache warming. Once a user logs in, cache the next 5 pages they are likely to visit.
7. Interstitial pages when speed is abysmal. Measure initial queries, if they are slow, then send them to alternate faster page instead. While that page is being viewed, cache the data for the previous query.
8. If you couldn't memcache it, load it asynchronously. Graceful degradation. If a query is slow make a decision not to render that part of the page. Render it what the query completes.
9. Inactivity rewarming. Recache data that would have expired after a period of inactivity so it's there if the user returns.
10. Denormalise data. For a query like "who are this managers' direct subordinates" that would take 300ms, store the direct reports with the managers and recalculate when the relationships changes. The results is now accessible with a GET and will be much faster.
11. Use JARs! Their biggest win. With thousands of classes loading classes took up to 400ms because of the disk access involved. JARing all the classes together improved load times made the system fly.
12. Warmup requests. Use GAE's warmup feature and stick as much work as you can there. Exercise code paths, write data to memcache, render some core UI elements, run key queries, simulate a login, make some date-time calculations. All this warms up the VM and the data.
13. Defer writes. Queue writes to the task queue if they will take a lot of time or if a lot of queries are already being made.
14. Asynchronous mails. Send email from task queues. 
15. Asynchronous queries. Run queries in parallel.

Comments

[Per]

Hi, and thanks for the mention!

Since we posted that article a few weeks ago, we've seen further improvements to the underlying GAE infrastructure. We now average at 400ms for our rather heavy front page, and the GAE dashboard reports averages of 100ms for all dynamic requests combined (this is what the acutal users experience, so it also including simple ajax actions, but also the heavier reporting pages). Overall, it's still not lightning fast, but 2s requests are really rare these days, and we're very happy with GAE again.

Cheers,
Per

[Toby DiPasquale]

I'd be curious to hear the reasons to use GAE if its performance is that slow and that highly variable. Is the time to market/ease of development that much better to make it worth it to put up with all that?

[Per]

As outlined in the post, there were issues in early August, which have been resolved since then. Also, some our code was just inefficient to begin with. Variance is down, and we're very happy with GAE now. Also, the application we're writing is pretty complex, and there's certainly a LOT more we could be tuning to get response times down even further. But at 300ms, other feature work is really more relevant to our customers.

I'd never ever want to maintain my own server again, the optimisations we did here were miniscule compared to what we'd have to do if we were to maintain our own servers, app servers and databases.

Other reasons to use GAE: It's just super easy to code for it, there are lots and lots of useful APIs built in. All deployment issues are taken care of (we deploy 1-2 times a day or so, but we could do 20 times if we wanted to just as easily). And scalability of course.