Go — context canceled Cascade When Clients Disconnect

Tell apart context deadline exceeded from context canceled, trace client disconnects cascading through call chains, and separate user-initiated cancellation from system errors.

intermediatedebugginggocontexthttp

Symptoms

Logs fill with context canceled errors across many services at the same moment a client disconnects
Downstream jobs or caches report failures that were really just the caller walking away
The distinction matters in metrics: cancellation storms inflate error rates and page on-call

Possible causes

The HTTP request context is canceled when the client disconnects, and it propagates to every RPC, query, and goroutine derived from it
Handlers derive work contexts from the request context, so background side effects get canceled too
Errors from ctx.Err() are classified with the same handler as failures, polluting dashboards
A proxy timeout cancels its upstream request context, cascading canceled into healthy dependencies

Diagnosis steps

Classify the two errors precisely

grep -Rn "context.Canceled\|context.DeadlineExceeded\|ctx.Err()" src/ | head -20

Interpretation

Canceled means the parent context was canceled (client left); DeadlineExceeded means a timer fired. Different causes, different responses, different metrics.

Correlate the cancellation timestamp

grep "context canceled" svc-b.log | head -3 && grep -n "client disconnected" edge.log | head -3

Interpretation

Cancellation lines synchronized with disconnects at the edge confirm propagation rather than independent failures.

Find wrongly derived contexts

grep -Rn "ctx, cancel := context.WithCancel(r.Context())" src/ | head

Interpretation

Background work derived from the request context dies with the request; it needs its own root context.

Root cause

The request context carries cancellation from the client connection. Every operation derived from it inherits the signal, so a disconnect cancels the entire tree of in-flight calls. Treating that propagated signal as a component failure misclassifies a normal user behavior as an outage.

Fix

Classify context.Canceled from the request context as expected: log at info, count it separately from errors, and respond 499 if you report per-client codes. Background work that must outlive a request takes a detached root context with its own cancellation and timeout.

Risk note

Swallowing DeadlineExceeded along with Canceled hides real timeout problems. Split the two classes before deciding what is benign.

Verify the fix

Disconnect clients mid-request in a test harness; assert side effects still complete under their own contexts and cancellation metrics count the events without triggering alerts.

Symptom walkthrough

Two error values that look identical in logs and mean opposite things:

rpc error: code = Canceled desc = context canceled
rpc error: code = DeadlineExceeded desc = context deadline exceeded

Canceled: someone called cancel — most often the HTTP layer because the client closed the connection. DeadlineExceeded: a timer fired. The cascade signature is timing: pick one canceled line, and the same second shows the edge log entry for a disconnect. When five services log canceled simultaneously, one user left; when five services log DeadlineExceeded, something is actually slow.

Fix in practice

First, classify at the boundary where the error surfaces:

if errors.Is(err, context.Canceled) && req.Context().Err() != nil {
    // client walked away: expected, not a failure
    log.Info("request abandoned by client", "trace", traceID)
    metrics.Cancellations.Add(1)      // separate counter
    return
}
if errors.Is(err, context.DeadlineExceeded) {
    // real timeout: alert-worthy
    metrics.Timeouts.Add(1)
}

Second, stop deriving long-lived work from the request context:

func handler(w http.ResponseWriter, r *http.Request) {
    // background job must survive the request
    jobCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
    defer cancel()
    queue.Submit(jobCtx, payload)

    // response work correctly follows the request lifetime
    render(w, r.Context())
}

The rule: anything the user will never see the result of may keep the request context; anything with a side effect that outlives the connection gets its own root.

Verification

　　Build a harness that disconnects clients at random points mid-request. Assert three outcomes: canceled metrics count the disconnects without the error-rate dashboards moving, background side effects (audit writes, cache fills) complete under their detached contexts, and DeadlineExceeded paths still alert — proving the classification split real timeouts from abandonment rather than hiding both.