Profiling the JVM with async-profiler

Most of the older Java profilers (VisualVM, JProfiler’s legacy sampling modes, the JVM’s built-in sampling CPU profiler) only capture stacks when threads hit a safepoint. The trouble is that threads aren’t evenly spread across the code at the moment they reach one. Some methods are full of safepoints, others have none at all. The profile you get is skewed, and you can end up optimising code that isn’t the real culprit. That’s safepoint bias.

async-profiler doesn’t have that problem. It uses the kernel’s perf_events (or an internal timer) to interrupt threads anywhere, then grabs the stack through AsyncGetCallTrace. Overhead stays in the low single-digit percent, which makes it fine to run in production.

Start with one command

Since 3.x the binary is called asprof. To profile the CPU of a running process for 30 seconds and get a flamegraph out of it:

asprof -d 30 -f profile.html <pid>

The default event is cpu. The output format comes from the file extension: a .html gives you an interactive flamegraph. For a first diagnosis, that’s all you need.

Choose what you measure

What makes the tool worth it is that one binary covers several dimensions through -e:

asprof -d 30 -e cpu   -f cpu.html   <pid>   # CPU time
asprof -d 30 -e alloc -f alloc.html <pid>   # heap allocations
asprof -d 30 -e lock  -f lock.html  <pid>   # lock contention
asprof -d 30 -e wall  -f wall.html  <pid>   # wall-clock (real time)

The distinction that really matters day to day is CPU versus wall-clock. The cpu mode only counts time threads actually spend running on a core. A thread blocked on I/O or waiting on a lock is invisible to it. If your latency comes from a network call or a slow SQL query, the CPU profile will be close to empty even though the problem is very real. That’s when you reach for wall.

The alloc mode is invaluable for chasing GC pressure. It shows you where the heap gets allocated, which points straight at the loops churning out too many temporary objects.

Accurate stacks: DebugNonSafepoints

So that Java frames land in the right place instead of being snapped to the nearest safepoint, start the JVM with:

-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints

Without these, JIT-inlined code can be misattributed. The two flags cost nothing measurable, so turn them on by default on any environment you plan to profile.

Permissions and containers

perf_events needs kernel access. Usually that means:

sysctl kernel.perf_event_paranoid=1   # allow user-space profiling
sysctl kernel.kptr_restrict=0         # kernel symbols in the stacks

In a container these settings aren’t always reachable, and perf may be missing entirely. async-profiler then offers an engine that doesn’t rely on perf:

asprof -d 30 -e ctimer -f profile.html <pid>

ctimer (or itimer on older versions) is built on a POSIX timer and works without any special capability. You lose kernel frames, but for profiling application code that’s more than enough.

Profiling from startup

To capture what happens at boot, or to bake profiling into an automated run, attach the agent directly:

java -agentpath:/opt/async-profiler/lib/libasyncProfiler.so=start,event=cpu,file=profile.jfr \
     -jar app.jar

The jfr (JDK Flight Recorder) output opens in JDK Mission Control and lets you line the profile up against the JVM’s other events.

What next?

You’ve got a profile.html in front of you. A flamegraph reads fast once you know the rules, and that’s exactly what the next article covers: How to read a flamegraph.