Python asyncio timeout — TimeoutError, Blocked Loop, and Slow Tasks

Diagnose asyncio timeouts by timing coroutines, finding blocking calls, checking downstream latency and concurrency limits, then applying explicit timeout budgets.

intermediatedebuggingpythonasyncio

Symptoms

asyncio.TimeoutError or TimeoutError is raised from wait_for or asyncio.timeout
Requests that are fast in isolation time out under concurrency
The event loop appears frozen and unrelated coroutines stop progressing
Task exception was never retrieved warnings appear after cancellations

Possible causes

A synchronous call such as requests, time.sleep, or a blocking DB driver stalls the event loop
The downstream service is slower than the configured timeout
Too many concurrent tasks exhaust a connection pool, so work queues behind the timeout
Cancellation is swallowed by a broad except block and the timeout never propagates

Diagnosis steps

Time the coroutine in isolation

python -X dev -c "import asyncio, time; from app.client import fetch; t=time.perf_counter(); print(asyncio.run(fetch())); print('elapsed', time.perf_counter()-t)"

Interpretation

Dev mode surfaces slow callbacks and unawaited coroutines. If a single call already exceeds the timeout, the timeout value or downstream latency is the issue.

Find blocking calls

PYTHONASYNCIODEBUG=1 python -m app.main 2>&1 | grep -i 'took .* seconds' | head -20

Interpretation

Asyncio debug reports callbacks that block the loop for too long. A recurring entry around the timeout points to synchronous I/O in a coroutine.

Profile the running process for blocked coroutines

py-spy top --pid $(pgrep -f "python -m app.main")

Interpretation

Time spent in socket reads, time.sleep, or synchronous database calls instead of the event loop indicates blocking I/O. py-spy attaches without code changes.

Compare latency with and without concurrency

python -m pytest tests/test_client.py -k timeout -q && python -m pytest tests -q --maxfail=1

Interpretation

A timeout test that passes alone but fails in the full suite indicates pool exhaustion or shared state rather than a wrong timeout constant.

Root cause

asyncio timeouts cancel a coroutine after a deadline, but they cannot preempt synchronous code holding the event loop. If a coroutine blocks, every other task stops and deadlines expire together. Even with non-blocking I/O, a small connection pool or unbounded concurrency can queue work longer than the timeout, so the deadline is reached before the request is ever sent.

Fix

Give each network call an explicit timeout, use asyncio.timeout around the whole operation, and replace blocking libraries with async equivalents or run them in asyncio.to_thread. Bound concurrency with a semaphore sized to the connection pool, add retries with backoff only for idempotent operations, and never swallow CancelledError with a bare except.

Risk note

Very short timeouts cause retry storms and duplicate writes on non-idempotent endpoints. Wrapping blocking code in to_thread increases thread usage and does not make it non-blocking at the database level. Suppressing CancelledError can leave tasks running after shutdown.

Verify the fix

Run the load test again and confirm p95 latency stays below the timeout, with no asyncio.TimeoutError under normal traffic. Trigger a deliberate slow dependency and verify the timeout fires, the task is cancelled cleanly, and no unretrieved task warnings appear.

A timeout is not always the slow part

TimeoutError means the deadline passed. It does not tell you whether the downstream was slow, the request never left because the pool was full, or the event loop was blocked by synchronous code.

asyncio.exceptions.TimeoutError
  File "/app/client.py", line 42, in fetch
    async with asyncio.timeout(2):

Measure the loop, not just the call

PYTHONASYNCIODEBUG=1 python -m app.main
python -X dev -c "import asyncio, time; from app.client import fetch; ..."

Asyncio debug mode logs callbacks that block longer than expected. A time.sleep, requests.get, or synchronous database driver inside a coroutine stops every other task, so all deadlines expire at once. Replace it with an async client or await asyncio.to_thread(...).

Apply a bounded, explicit budget

import asyncio
import httpx

sem = asyncio.Semaphore(20)

async def fetch(client: httpx.AsyncClient, url: str) -> dict:
    async with sem:
        async with asyncio.timeout(2.5):
            response = await client.get(url)
            response.raise_for_status()
            return response.json()

Set connect, read, and total timeouts on the client as well; a total deadline cannot fix a connection that never opens if the pool is exhausted. Size the semaphore to the pool and downstream capacity, and retry only idempotent calls.

Verify

Run the concurrency test and confirm p95 stays under the deadline with no timeouts. Then introduce a deliberate delay: the timeout should fire, cancellation should propagate, and shutdown should complete without “Task exception was never retrieved”. Check that no request was written twice after a timeout on a non-idempotent endpoint.