Kotlin coroutines in production: dispatchers, cancellation and leaks
Coroutines are the normal way to write concurrent code in Kotlin. Ktor is built on them. Spring accepts them in its controllers. They have been stable since 2018, and they’re well documented.
And yet, they break in production. Not because of a bug in the library. Because of four misunderstandings, always the same ones. A blocking call in the wrong place. A catch that is a little too wide. One runBlocking too many. And a thread dump that shows nothing, right when you need it most.
This article goes through those four points. Everything was verified on kotlinx-coroutines 1.11.0 and a JDK 25, on a 10-processor machine. The output is real.
A coroutine is not a thread
Everyone knows the sentence. Few people follow it to its conclusions.
A coroutine is an object on the heap. The compiler turns every suspend function into a state machine. When the coroutine waits, on a delay, a network call, a query to a non-blocking database, it blocks no thread. It stores its state in an object, the continuation, and gives the thread back. When the result arrives, a thread picks it up where it left off. Not necessarily the same thread.
You can measure what that costs. The following program starts 100,000 waiting coroutines, then compares memory before and after:
fun memory(): Long {
System.gc(); Thread.sleep(200)
val r = Runtime.getRuntime()
return r.totalMemory() - r.freeMemory()
}
fun main() = runBlocking {
val before = memory()
val jobs = List(100_000) { launch { delay(Long.MAX_VALUE) } }
delay(500)
val delta = memory() - before
println("100,000 suspended coroutines: ${delta / 1024 / 1024} MB, ${delta / 100_000} bytes each")
println("JVM threads: " + Thread.getAllStackTraces().size)
jobs.forEach { it.cancel() }
}100,000 suspended coroutines: 24 MB, 254 bytes each
JVM threads: 6A few hundred bytes per coroutine. And six threads in total, the JVM’s own. A platform thread, by contrast, reserves on the order of a megabyte for its stack, as the virtual threads article points out.
That is the whole point. And it is also the whole trap. A coroutine costs nothing as long as it suspends. The moment it blocks, it costs a thread. And there aren’t many threads.
How many threads, exactly
Coroutines run on dispatchers. A dispatcher is a thread pool plus a rule for placing work on it. Two of them matter on the server side.
Dispatchers.Default is for computation. Dispatchers.IO is for blocking calls. The following program starts 200 blocking tasks on each, and counts the distinct threads that served them:
suspend fun count(d: CoroutineDispatcher): Int = coroutineScope {
val names = ConcurrentHashMap.newKeySet<String>()
repeat(200) { launch(d) { names += Thread.currentThread().name; Thread.sleep(500) } }
names.size
}cpus=10
Default: 10 threads
IO: 64 threads
IO.limitedParallelism(100): 100 threadsDefault has as many threads as processors, with a minimum of two. IO has 64, or the number of processors if that is higher. The ceiling is set with the kotlinx.coroutines.io.parallelism system property. And a limitedParallelism view on IO can go above 64, as the last line shows. That has been the case since 1.6, and it is specific to IO: the same view on Default stays bounded by the processor count.
Two details matter in production.
First: the two dispatchers share the same threads. Look at the names:
[DefaultDispatcher-worker-45, DefaultDispatcher-worker-75, DefaultDispatcher-worker-84, ...]There is no IO-worker thread. One pool serves both, and each dispatcher applies its own limit on top of it. In a thread dump, the thread name will not tell you which one a worker is serving. The stack will: a task dispatched through IO carries a kotlinx.coroutines.internal.LimitedDispatcher$Worker.run frame two lines below your last frame, right after DispatchedTask.run. A Default task has none. A limitedParallelism view, on either dispatcher, carries one too: that frame means “went through a limit”, and IO is one.
Second: in a container, that processor count is what the JVM reads from the cgroup. A pod limited to one CPU has a Dispatchers.Default of two threads. Everything in Tuning the JVM in a container applies directly to your coroutines.
Blocking on Default, the first trap
Here is the most common trap, and the easiest to measure. A hundred tasks that each wait one second, three ways to do it:
val t1 = measureTimeMillis { coroutineScope { repeat(100) { launch(Dispatchers.Default) { Thread.sleep(1000) } } } }
val t2 = measureTimeMillis { coroutineScope { repeat(100) { launch(Dispatchers.Default) { delay(1000) } } } }
val t3 = measureTimeMillis { coroutineScope { repeat(100) { launch(Dispatchers.IO) { Thread.sleep(1000) } } } }100 x Thread.sleep(1000) on Default: 10036 ms
100 x delay(1000) on Default: 1014 ms
100 x Thread.sleep(1000) on IO: 2015 msThe same one-second wait, and a factor of ten between the first line and the second.
Thread.sleep blocks its thread. On Default, there are only ten. So the hundred tasks go through ten at a time. delay suspends the coroutine, the thread is freed immediately, and all hundred tasks wait at the same time. On IO, Thread.sleep still blocks, but with 64 threads the damage is smaller.
Thread.sleep is an example. In real life it will be JDBC. A JDBC call always blocks its thread: there is no such thing as non-blocking JDBC. Or an old HTTP client, a file read, a Thread.sleep hidden in a retry library. On Default, all of that eats your ten threads. The application does not crash. It gets slow, the CPU sleeps, and the profiler shows nothing because nothing is computing.
The rule is simple. Whatever blocks goes on IO, explicitly:
suspend fun loadCustomer(id: Long): Customer = withContext(Dispatchers.IO) {
jdbcTemplate.queryForObject("select ... where id = ?", mapper, id)
}And the rule matters twice as much inside a framework. On Spring WebFlux, a suspend handler runs on the Netty threads, and there are about as many of those as there are cores, four at minimum. A JDBC call placed there blocks a thread that serves every request on the server. Ktor, with the Netty engine, also runs its handlers in coroutines, on a call pool of one thread per processor. Same rule: blocking work goes through withContext(Dispatchers.IO).
runBlocking inside a coroutine
runBlocking does what its name says. It blocks the current thread until its coroutine completes. That is fine in main, in a test, or to call suspend code from an API that is not.
Inside a coroutine, it is a bomb. The following program runs on a single-thread dispatcher. The outer coroutine does a runBlocking, which in turn wants to run work on that same dispatcher:
val one = Dispatchers.Default.limitedParallelism(1)
runBlocking(one) {
println("in the coroutine, thread=" + Thread.currentThread().name)
val r = runBlocking { withContext(one) { "ok" } }
println("result=$r")
}in the coroutine, thread=DefaultDispatcher-worker-1
[killed after 8 s]The second line never comes. The inner runBlocking holds the dispatcher’s only thread. The withContext(one) waits for a thread of that dispatcher to become free. Nobody moves. It’s a deadlock.
One thread is the extreme case. But ten threads is the Dispatchers.Default of an ordinary machine. It only takes ten requests making the same detour at the same time, and the server stops. It never happens in tests. It happens on a Monday morning, under load.
So runBlocking stays at the edges of the program: main, tests, and the adapter to a blocking API you do not control. Never in a suspend function, never in a handler.
Cancellation is cooperative
A cancelled coroutine does not stop on its own. Cancellation is cooperative: the coroutine has to pass through a suspension point, a delay, a yield, a network call, to notice it has been cancelled. Without one, it keeps going.
The demonstration fits in three lines:
val job = launch(Dispatchers.Default) { while (true) { n++ } }
delay(100); job.cancel()
withTimeoutOrNull(1000) { job.join() }while(true): join after cancel = still alive after 1 s
while(isActive): finishedThe while (true) loop survives its cancel(). It has no suspension point, so no chance to stop. The worker is lost until the process ends. The while (isActive) version stops cleanly. ensureActive() or yield() inside the loop do the same job.
That case is well known. The next one is much less so, and nastier.
The catch that swallows cancellation
Cancellation travels as an exception, CancellationException. Every suspension point of the library, delay, yield, withContext, channel operations, throws it once the job is cancelled. The problem is that CancellationException extends Exception. So a catch (e: Exception) catches it, along with everything else. So does a catch (e: RuntimeException), since it extends that too.
Here is a polling loop of the kind you see everywhere. It waits, it works, and it protects itself against errors:
val job = launch(Dispatchers.Default) {
while (true) {
try {
delay(50)
// work
} catch (e: Exception) {
// log, and carry on
}
rounds.incrementAndGet()
}
}
delay(120); job.cancel()catch(Exception): rounds before cancel=2, 100 ms after cancel=178461, isCompleted=falseTwo rounds before cancellation. 178,461 rounds in the 100 ms that follow. And the job still has not completed.
What happens: once the job is cancelled, delay throws CancellationException immediately, on every call. The catch catches it, the loop goes round again, delay throws again. A 50 ms wait has become a tight loop. The worker runs at 100% CPU, for nothing, until the process ends. And the runBlocking around all this, since it waits for its children, never returns.
This is the most common coroutine leak I know of, and it passes every code review. The fix is one line at the top of the catch:
} catch (e: Exception) {
if (e is CancellationException) throw e
// log, and carry on
}Or coroutineContext.ensureActive() in the same place, which rethrows the cancellation if the job is cancelled. Or a catch (e: CancellationException) { throw e } placed before the general catch.
Cleanup in finally
One last point on cancellation. A cancelled coroutine does run its finally blocks. But inside them, the job is already cancelled. Any suspension point there throws CancellationException again:
delay in finally: kotlinx.coroutines.JobCancellationException: StandaloneCoroutine was cancelledIf your cleanup has to suspend, closing a connection with a non-blocking client, sending one last message, you have to say so explicitly:
} finally {
withContext(NonCancellable) {
connection.close()
}
}Where exceptions go
Coroutines are structured: a launch has a parent, and that parent waits for its children. That structure also decides what becomes of an exception. There are four cases, and you need to know all four.
Inside a coroutineScope, an exception in a child cancels the siblings, then goes up to the parent, which rethrows it. That also holds for an async nobody awaited:
coroutineScope {
async { delay(50); error("boom async") }
launch { delay(1000); println("never printed") }
}coroutineScope rethrew: boom asyncThe launch is cancelled, the coroutineScope rethrows. This is the safest behavior. Nothing gets lost.
Inside a supervisorScope, or under a SupervisorJob, a failing child does not take the others down. The supervisorScope itself completes normally. The exception from a launch goes to the context’s CoroutineExceptionHandler, or, if there is none, to the uncaught-exception handler of the thread the coroutine was running on, which prints the stack and moves on:
handler saw: java.lang.IllegalStateException: boom supervisor
the sibling survivesAn async under a supervisor, on the other hand, keeps its exception to itself. It is delivered when await() is called. If nobody calls await(), nobody will ever see it:
unawaited async under supervisor: isCancelled=true, nothing printedNo log, no handler. The Deferred is cancelled in silence. An async whose result you do not wait for is a launch that forgot to say so, and that loses its errors as well.
withTimeout, finally, is the case almost nobody knows about. It throws TimeoutCancellationException. And that exception is a CancellationException. Look at what happens inside a launch:
val job = scope.launch {
withTimeout(100) { delay(1000) }
println("never printed")
}withTimeout in launch: isCancelled=true, parent active=trueThe launch is cancelled. The parent is not, and that is not the SupervisorJob at work: under a plain Job, the parent and the siblings survive too. And the CoroutineExceptionHandler saw nothing: to the library, a cancellation is a normal event, not an error. Your timeout expired, the work was not done, and nothing reported it. withTimeoutOrNull returns a null you are forced to handle. It is often the better choice. Otherwise, catch TimeoutCancellationException at the edge of the launch and log it yourself.
The scope that never dies
A coroutine lives in a scope. A scope lives until someone cancels it. A scope nobody ever cancels is a leak.
The classic case:
class Synchronisation {
private val scope = CoroutineScope(Dispatchers.Default)
fun start() {
scope.launch { while (isActive) { synchronise(); delay(10_000) } }
}
}Nothing closes that scope. If Synchronisation is created again, on every test, every reload, every request, the old loop keeps running. Two instances, two loops. A hundred instances, a hundred loops. Memory climbs slowly, so does CPU, and the heap dump will show StandaloneCoroutine objects hanging on to things nobody uses anymore.
GlobalScope is the same problem, only worse, since you cannot even cancel it. It is marked @DelicateCoroutinesApi for that reason.
The rule: a scope has an owner, and the owner closes it. With a SupervisorJob, so that one failure does not take the other tasks down:
class Synchronisation : AutoCloseable {
private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
fun start() { scope.launch { /* ... */ } }
override fun close() = scope.cancel()
}In Spring, close() is called from a @PreDestroy. In Ktor, from an application shutdown hook. What matters is that it gets called.
In a thread dump, they do not exist
This is the point that makes everything else hard to diagnose.
Take 10,000 named coroutines, suspended on a delay, and a thread dump taken while they wait:
val jobs = List(10_000) { i ->
launch(Dispatchers.Default + CoroutineName("order-$i")) { delay(Long.MAX_VALUE) }
}threads in the dump: 30, of which workers: 10
occurrences of 'order' in the dump: 0Thirty threads. Ten workers. And zero trace of the 10,000 coroutines. The workers themselves are all in the same state:
"DefaultDispatcher-worker-1" #25 daemon prio=5 ... waiting on condition
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@25.0.1/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.1/LockSupport.java:408)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:833)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:781)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:751)That makes sense. A suspended coroutine is on no thread. It is on the heap. A thread dump photographs threads, so it only sees the coroutines currently running. The ones that are waiting, meaning nearly all of them, meaning the ones you are looking for, are invisible. The whole approach in Reading a thread dump with jstack stops at the coroutines’ door. It is exactly the same limit as with virtual threads, and for the same reason.
Two tools fill the gap.
Debug mode
With -Dkotlinx.coroutines.debug, the library appends the coroutine’s name to the thread name, while it runs:
thread name in coroutine: DefaultDispatcher-worker-2 @order-0#2That still only shows the mounted coroutines. But at least, when a worker is busy, you know what with. You do have to name your coroutines, with CoroutineName. Otherwise the dump says @coroutine#2, which helps nobody.
The coroutine dump
The real tool is kotlinx-coroutines-debug. It hooks into coroutines as they are created and can list the suspended ones, with their stacks. The safest way is to install it as an agent at startup:
java -javaagent:kotlinx-coroutines-debug-1.11.0.jar -Dkotlinx.coroutines.debug ...DebugProbes.install() does the same thing from code. It attaches to its own JVM through Byte Buddy and JNA, which Gradle pulls in with the dependency. If you just drop the jars on a classpath, without JNA, you get No compatible attachment provider is available. And since JDK 21, that dynamic attach prints a warning, ahead of being disallowed by default. The agent avoids all of that. The coroutine name in the dump, for its part, comes from -Dkotlinx.coroutines.debug: without the flag, the agent lists anonymous coroutines.
A call to DebugProbes.dumpCoroutines() then prints this, for a coroutine suspended three functions down:
Coroutine "order-42#2":StandaloneCoroutine{Active}@215be6bb, state: SUSPENDED
at E9Kt.callApi(E9.kt:11)
at E9Kt.loadCustomer(E9.kt:10)
at E9Kt.processOrder(E9.kt:9)
at E9Kt$main$1$1.invokeSuspend(E9.kt:4)The name, the state, and the stack of suspend functions. Where the thread dump showed nothing, you see the coroutine stuck in callApi, called by loadCustomer. That is what you need to find out who is waiting for what.
One surprising detail: a suspend function whose last act is to call another suspend function does not appear in that stack. The compiler optimizes it as a tail call, with no continuation of its own. If a function is missing from the dump, that is probably why.
These probes have a cost. They track every coroutine created. Keep them for a workstation, or a test environment under load, for as long as it takes to understand.
Limiting concurrency
The thread pool used to limit concurrency without saying so. With one launch per request, that limit is gone. And as in the virtual threads article, it is the connection pool behind it that hits the wall.
To throttle calls to a resource, the library has its own Semaphore. Not the one in java.util.concurrent, which would block the thread. The one in kotlinx.coroutines.sync, which suspends:
private val limit = Semaphore(20)
suspend fun call(r: Request): Response = limit.withPermit {
client.send(r)
}Twenty calls in flight at most. The extra coroutines wait, without holding a thread.
limitedParallelism(n) does a similar job, but on the dispatcher: at most n coroutines running at once. It bounds CPU, not the number of calls waiting for a reply. For a remote resource, the semaphore is what you want.
And do not add anything in front of a connection pool. HikariCP already does that. Set its size and its acquisition timeout, as the virtual threads article describes.
Coroutines and virtual threads
Since Java 21, the question comes up on every migration: do virtual threads make coroutines pointless?
One point first, and one that is often misunderstood. On a JDK 25, Dispatchers.IO still uses platform threads. Sixty-four of them, as measured above. The library has no built-in dispatcher for virtual threads, and it never switches to them on its own.
But you can make one, in a single line:
val vt = Executors.newVirtualThreadPerTaskExecutor().asCoroutineDispatcher()And the difference shows straight away, on a thousand one-second blocking calls:
1000 x Thread.sleep(1000) on virtual threads: 1018 ms, 1000 distinct threads
1000 x Thread.sleep(1000) on IO: 16060 msOne second versus sixteen. On IO, a thousand blocking calls go through 64 at a time. On virtual threads, they all go at once, each on its own thread, and blocking no longer costs anything. For blocking code you cannot make suspend, JDBC above all, that is a real gain, immediate, with no change to the calling code.
The two do not replace each other. A virtual thread removes the cost of a blocked thread. A coroutine adds structure on top: the scope, cancellation that propagates, exceptions that go up to the parent, Flow. You can have both. The dispatcher above gives you structured coroutines, running on virtual threads. Keep Default for computation, and replace IO with that dispatcher wherever you block.
One caveat: the virtual thread traps apply from then on, starting with pinning. The dedicated article covers them.
In short
Default has one thread per processor, two at minimum, IO has 64, and they share the same pool. In a container, that number comes from the cgroup.
A blocking call on Default costs one thread in ten. JDBC, old HTTP clients, files: all of that goes on IO, through withContext.
runBlocking stays at the edges of the program. Inside a coroutine, it is a deadlock waiting for its Monday morning.
Cancellation is cooperative. A loop with no suspension point does not stop. And catch (e: Exception) swallows CancellationException: rethrow it, or the coroutine spins in a tight loop until the process ends.
withTimeout inside a launch fails silently. So does an unawaited async under a supervisor.
A scope has an owner, and the owner cancels it. GlobalScope has none.
jstack does not see suspended coroutines. Name them, turn on -Dkotlinx.coroutines.debug, and keep kotlinx-coroutines-debug at hand for a real coroutine dump.
Finally, Dispatchers.IO gets nothing from virtual threads. One asCoroutineDispatcher() on a virtual executor, and your blocking calls stop queueing.