Debugging a Deadlock in a Python asyncio Application

By James Nguyen Updated September 24, 2026

A worker process handling background jobs would occasionally just stop, no crash, no error in the logs, no CPU usage, just silence, with a handful of jobs sitting permanently in a "processing" state until we manually restarted the process. That's a deadlock, and debugging one in asyncio code is a different exercise than debugging a deadlock in a traditional multi-threaded application, since the usual thread-dump-based tools don't map cleanly onto a single-threaded event loop juggling many coroutines.

Why asyncio Deadlocks Look Different From Thread Deadlocks

A classic multi-threaded deadlock involves two threads each holding a lock the other needs, visible in a thread dump as two threads both blocked on lock acquisition. asyncio runs on a single thread with a single event loop cooperatively scheduling coroutines, so there's no OS-level thread blocking to inspect, the process shows as idle because the event loop itself is technically running fine, it's just that every pending coroutine is waiting on something that will never resolve, an awaited Future that nothing will ever set the result on.

The Actual Culprit: An asyncio.Lock Acquired but Never Released

The specific bug was a lock acquired at the top of a job handler, with the release meant to happen at the end, but an unhandled exception partway through the function meant the code path that released the lock never executed, leaving the lock permanently held. Every subsequent job needing that same lock queued up waiting forever, which is exactly the silent, CPU-idle deadlock we were seeing, not a crash, just an ever-growing queue of coroutines awaiting a lock nothing would ever release.

# The bug: exception between acquire and release skips the release entirely
async def process_job(job):
    await job_lock.acquire()
    result = await risky_operation(job)  # raises here sometimes
    job_lock.release()  # never reached on exception
    return result

The Fix Is a Context Manager, Not Manual Acquire/Release

asyncio.Lock supports the async with statement specifically to guarantee release happens even when an exception propagates through the block, the same reasoning that makes a context manager preferable to manual try/finally in synchronous code. Once I stopped calling .acquire() and .release() by hand and switched to async with lock:, an exception inside the block still correctly released the lock on the way out, which is the actual fix, not a workaround.

async def process_job(job):
    async with job_lock:
        result = await risky_operation(job)
        return result
    # lock is released here regardless of whether an exception occurred

Finding Which Coroutine Was Actually Stuck

Knowing the theoretical cause didn't immediately tell me it was actually happening in this specific process. asyncio.all_tasks() returns every currently scheduled task, and calling it from a debug endpoint I added temporarily, then printing each task's current stack via task.get_stack(), showed several tasks all suspended at the exact same await job_lock.acquire() line, which was the concrete confirmation pointing at the lock rather than a network call or database query as the actual hang point.

@app.get('/debug/tasks')
async def debug_tasks():
    tasks = asyncio.all_tasks()
    return [
        {'name': t.get_name(), 'stack': [str(f) for f in t.get_stack()]}
        for t in tasks
    ]

asyncio's Debug Mode Surfaces Slow Callbacks Proactively

Running with PYTHONASYNCIODEBUG=1, or passing debug=True to asyncio.run(), makes the event loop log a warning whenever a callback or coroutine step takes longer than a threshold to yield control back to the loop, which wouldn't have caught this specific deadlock directly but did surface a separate, related issue: a synchronous, blocking call to a database driver's non-async method sitting inside an async function, silently blocking the entire event loop for its duration and starving every other coroutine of scheduling time while it ran.

asyncio.wait_for as a Blunt but Effective Safety Net

Beyond fixing the specific lock bug, wrapping lock acquisition, and other awaits with genuine potential to hang indefinitely, in asyncio.wait_for with a reasonable timeout turns a silent permanent deadlock into a loud, logged TimeoutError that at least surfaces the problem and lets the job fail explicitly rather than hanging the worker forever. This isn't a substitute for fixing the actual bug, but as a defensive measure against the next unknown deadlock, it converts "silent and undetected" into "visible and handled" by default.

try:
    async with asyncio.timeout(30):  # Python 3.11+
        async with job_lock:
            result = await risky_operation(job)
except TimeoutError:
    logger.error(f"job {job.id} timed out waiting on lock, likely deadlock")
    raise

Final Verdict

An asyncio deadlock doesn't announce itself the way a crashed thread does, it just goes quiet, and the debugging approach has to lean on asyncio-specific tools, all_tasks() and get_stack() for finding the stuck coroutines, debug mode for catching a blocking call hiding inside async code, rather than the thread-dump instincts a traditional multi-threaded deadlock would call for. The actual fix was almost embarrassingly small, async with instead of manual acquire and release, but finding it required understanding that asyncio deadlocks hide differently than the ones I was used to debugging.