Update 2015-12-15: Since 2014, a lot of good material has been written about flamegraphs and java profilers. You might want to watch Java Profiling from the Ground Up! from Nitsan Wakart. You might also want to know that since JDK 8 update 60, an unpatched JDK can be used to generate mixed mode flame graphs. See Brendan Gregg's Java Mixed-Mode Flame Graphs at Netflix, JavaOne 2015. And finally, please be aware that hprof will be removed from JDK 9. It's time to update your skills and tools!
For a few months now, I have been working with Mediametrie to help them to finish a project aiming at publishing the audiences and uses of all French Internet sites.
I joined the team well after the project kick-off. Most of the code was already laid down, but some of the modules exhibited poor performance. Unable to troubleshoot the issues, the team was struggling to achieve performance requirements. Performance analysis is a scarce skill, and Hadoop pipelines usually make things much harder.
I introduced flamegraphs, profiling tools like hprof, Flight Recorder or perf to the team, then explained how to use them in a Hadoop context and described some performance analysis methods. After a few weeks, most of the issues were ...