The email sent will contain a link to this article, the article title, and an article excerpt (if available). For security reasons, your IP address will also be included in the sent email.

Debugging complex problems is 90% persistence and 50% cool tools. Brendan Gregg in 10 Performance Wins tells a fascinating story of how a team at Joyent solved some weird and challenging performance issues deep in the OS. It took lots of effort, DTrace, Flame Graphs, USE Method, and writing custom tools when necessary. Here's a quick summary of the solved cases:
- Monitoring. 1000x improvement. An application blocked while paging anonymous memory back in. It was also blocked during file system fsync() calls. The application was misconfigured and sometimes briefly exceeded available memory, getting page out.
- Riak. 2x improvement. The Erlang VM used half the CPU count it was supposed to, so CPUs remained unused. Fix was a configuration change.
- ...