Go — context deadline exceeded

Find which deadline expired in a Go call chain, distinguish a caller timeout from a dependency timeout, and fix propagation instead of increasing timeouts blindly.

intermediatedebugginggocontexttimeouts

Symptoms

The error string contains context deadline exceeded
An HTTP handler returns 504 or the client sees a timeout while the service is still running
Only the slowest fraction of requests fail, often under load
Retries make the failure rate worse instead of better

Possible causes

A parent context has a shorter deadline than the child operation needs
The HTTP client or database driver has its own timeout that fires first
A goroutine performs blocking work without checking context cancellation
A new context.Background is created downstream, so the caller's deadline is lost
A retry loop starts after the deadline has already expired

Diagnosis steps

Check which error the code actually received

go test ./... 2>&1 | grep -B3 -A5 'deadline exceeded'

Interpretation

Use errors.Is(err, context.DeadlineExceeded) and log the operation name, not only the error text. The operation identifies the layer that timed out.

Log the context error and remaining time at boundaries

logger.Info("call failed", "op", op, "err", ctx.Err(), "remaining", time.Until(deadline))

Interpretation

A remaining value near zero means the budget was exhausted before the call. A healthy remaining value points to the dependency's own timeout.

Trace the request with timing

curl -w 'total=%{time_total}\n' -H 'x-request-id: test' http://localhost:8080/slow

Interpretation

Compare client time, server middleware time, and outbound call time. The gap shows where the deadline is being spent.

Inspect client and server timeout configuration

grep -R 'Timeout\|WithTimeout\|WithDeadline' --include='*.go' .

Interpretation

Multiple independent timeouts can fire in an unexpected order. Every layer should derive from the inbound context.

Root cause

A context deadline expired before the operation completed. The deadline may be too short for the workload, but more often the context is not propagated, a retry starts too late, or a dependency has its own shorter timeout.

Fix

Derive child contexts from the inbound context, pass them to every I/O call, check ctx.Err() before retries, and align HTTP client and database timeouts with the remaining budget. Reduce work or add caching when the operation genuinely exceeds the budget.

Risk note

Increasing every timeout hides the problem and can exhaust connection pools. A longer timeout without cancellation still holds resources after the caller gives up.

Verify the fix

Reproduce with a slow dependency and assert that the handler returns a clear timeout response, the downstream operation is cancelled, and no goroutine remains blocked. Check p95 latency and error rate under the normal load.

There is more than one deadline

context deadline exceeded means some deadline in the chain expired. It does not say which one. The inbound HTTP server may have a read timeout, the client may have set a 2-second timeout, the database driver may have its own, or the parent context may have a shorter deadline than the operation. Log the operation name and ctx.Err() at each boundary so the failing layer is identifiable.

ctx, cancel := context.WithTimeout(r.Context(), 1500*time.Millisecond)
defer cancel()

row := db.QueryRowContext(ctx, query, id)
if err := row.Scan(&value); err != nil {
    if errors.Is(err, context.DeadlineExceeded) {
        return fmt.Errorf("query user %s: budget exhausted: %w", id, err)
    }
    return fmt.Errorf("query user %s: %w", id, err)
}

The child context derives from r.Context(), so a caller cancellation also cancels the query. context.Background() at this point would discard the caller’s deadline and keep the query running after the client is gone.

Do not retry past the deadline

A retry loop should check the remaining budget before starting another attempt. If the deadline is in 30 milliseconds and the operation needs 500, a retry only adds load. Compute the attempt timeout from the remaining time, and stop when it is below a useful minimum.

if deadline, ok := ctx.Deadline(); ok && time.Until(deadline) < 50*time.Millisecond {
    return ctx.Err()
}

Distinguish slow from stuck

A slow dependency may need optimization, caching, or a larger budget. A stuck dependency needs cancellation and a fallback. Compare the operation’s normal latency distribution with the deadline. If p99 is close to the budget, ordinary tail latency will produce timeouts under load. If the operation usually finishes in 50 milliseconds and occasionally hangs for minutes, the problem is a missing cancellation or a blocked connection, not a too-small timeout.

Verify cancellation end to end

Send a request, cancel the client, and confirm the handler and downstream query stop. Use a query or fetch against a test server that blocks until its context is cancelled. Check goroutine counts before and after; a growing count after cancellations indicates work that ignores the context.

Fix and verify

Fix the chain by deriving every child context from the inbound request, passing it to each I/O call, and checking the remaining budget before a retry. Verify with a test that cancels the client mid-request and asserts that the handler returns promptly, the downstream call is cancelled, and the goroutine count returns to baseline.