Go — gRPC DeadlineExceeded cascading through services

Diagnose cascading gRPC DeadlineExceeded by propagating context deadlines and shortening downstream timeouts.

advanceddebugginggogrpc

Symptoms

rpc error: code = DeadlineExceeded desc = context deadline exceeded
One slow dependency causes many services to time out
The error appears under load but not in isolation
Client timeouts propagate down the call chain

Possible causes

A context deadline is set too low for the end-to-end path
Deadlines are not propagated, so downstream calls run without one
A slow dependency consumes the whole budget before returning
Retries without budget multiply the delay

Diagnosis steps

Trace the deadline through the chain

grpcurl -d '{}' -connect-timeout 5 host:port service/Method

Interpretation

A manual call isolates one hop and shows whether that service itself is slow or only when part of a chain.

Check context propagation in handlers

grep -n "context.Background\|WithTimeout\|WithDeadline" server/*.go

Interpretation

Using context.Background instead of the request context drops the incoming deadline and any cancellation.

Budget across hops

server side: ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)

Interpretation

Each hop should shorten the remaining budget so total latency fits within the original deadline.

Root cause

The end-to-end request exceeded its deadline because time was spent across multiple hops without a shared budget, often because deadlines were not propagated and a slow dependency consumed the allowance.

Fix

Propagate the request context to every downstream call, derive shorter per-hop deadlines from the parent, and avoid retries that exceed the remaining budget.

Verify the fix

Exercise the full path under load and confirm no DeadlineExceeded, and confirm the slow dependency is now bounded by its hop deadline.

Symptom walkthrough

Calls fail with DeadlineExceeded, and under load a single slow dependency makes many services time out at once. The deadline is a shared budget: if each hop runs with its own generous timeout, a slow tail consumes the total and the original caller gives up. When handlers use context.Background instead of the incoming context, the deadline is never propagated and each hop runs unbounded.

Fix in practice

Pass the request context to every downstream call so cancellation and deadline flow with it, and derive a shorter timeout per hop from the remaining budget so total latency stays within the caller limit. Avoid retrying beyond the remaining deadline, since retries that cannot complete only add delay.

Verification

Exercise the full path under representative load and confirm no DeadlineExceeded occurs, then confirm the previously slow dependency is now bounded by its hop deadline. Meeting the budget end to end confirms the fix.

Diagnostic commands

Check the behavior with commands like these:

grpcurl -d '{}' -connect-timeout 5 host:port service/Method
grep -n "context.Background\|WithTimeout\|WithDeadline" server/*.go
server side: ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)