Go HTTP client timeout — context deadline exceeded Explained

Diagnose Go HTTP timeouts by measuring connect, TLS, and first-byte latency, then setting client, transport, and context deadlines that match real upstream behavior.

intermediatedebugginggohttp

Symptoms

Get https://api.example.com/health: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Requests succeed with curl but fail from the Go service
Timeouts appear only during traffic spikes or after a DNS change
Goroutines and connections accumulate while latency rises

Possible causes

http.Client.Timeout is too short for the upstream's real latency
The default transport has no response header timeout, so a stalled server holds the request
A new client or transport is created per request, preventing connection reuse
The request context is cancelled by the caller or an earlier deadline

Diagnosis steps

Measure each phase of the request

curl -sS -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} first_byte=%{time_starttransfer} total=%{time_total}\n' https://api.example.com/health

Interpretation

If first_byte is close to the Go timeout, the upstream is slow rather than unreachable. High connect time points to network or DNS; high TLS time points to handshake overhead.

Run the failing client path with verbose transport logs

GODEBUG=http2debug=1 go test ./internal/client -run TestFetch -v -count=1

Interpretation

The log shows connection reuse, stream errors, and GOAWAY frames. Repeated new connections suggest the transport is not being reused or is being closed by the server.

Inspect the configured deadlines

go test ./internal/client -run TestFetch -v -count=1 2>&1 | grep -iE 'timeout|deadline|context' | head -20

Interpretation

The error text names which deadline fired. Client.Timeout covers the whole request including body reads; a context deadline may come from the caller and expire earlier.

Check for leaked response bodies and idle connections

go tool pprof -top http://localhost:6060/debug/pprof/goroutine

Interpretation

Many goroutines blocked in net/http reading a response indicate bodies that are never closed, which also prevents connection reuse and increases latency.

Root cause

net/http applies no default timeout to a Client, and the zero value waits forever. When a timeout is set, it covers the entire exchange: connect, TLS, request write, response headers, and body read. A value shorter than the upstream's p99 surfaces as context deadline exceeded, while a Transport without ResponseHeaderTimeout lets a silent server hold a connection until the client deadline fires.

Fix

Reuse one http.Client with an explicit Timeout, and configure the Transport with DialContext timeout, TLSHandshakeTimeout, ResponseHeaderTimeout, IdleConnTimeout, and MaxIdleConnsPerHost. Pass a context with its own deadline for per-request budgets, read and close the response body on every path, and retry only idempotent requests with backoff.

Risk note

A global client timeout that is too generous lets a single slow dependency exhaust goroutines. Retrying a timed-out POST can duplicate writes. Setting MaxIdleConnsPerHost too low causes connection churn under concurrency, which looks like network latency.

Verify the fix

Run the load test again and confirm p99 latency stays below the client timeout with no deadline errors. Check that idle connections are reused across requests and that goroutine count returns to baseline after the test.

The zero timeout waits forever

A Go http.Client with no Timeout can block indefinitely. When you add one, it covers the whole exchange, not just the connection:

Get "https://api.example.com/health": context deadline exceeded
(Client.Timeout exceeded while awaiting headers)

That message means the deadline expired before response headers arrived. It does not distinguish DNS, connect, TLS, or a slow application.

Measure the phases outside Go

curl -sS -o /dev/null -w \
  'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} first_byte=%{time_starttransfer} total=%{time_total}\n' \
  https://api.example.com/health

If first_byte is near the Go timeout, the upstream is genuinely slow. If connect is high, look at network and DNS. Repeating the command shows whether the problem is steady or intermittent.

Set a coherent budget

client := &http.Client{
    Timeout: 10 * time.Second,
    Transport: &http.Transport{
        DialContext:           (&net.Dialer{Timeout: 2 * time.Second}).DialContext,
        TLSHandshakeTimeout:   2 * time.Second,
        ResponseHeaderTimeout: 5 * time.Second,
        IdleConnTimeout:       90 * time.Second,
        MaxIdleConnsPerHost:   32,
    },
}

Reuse the client, close the body (defer resp.Body.Close()), and pass a context with a deadline when the caller has a tighter budget. Creating a client per request defeats keep-alive and shows up as connection churn.

Verify

Run the load test and watch p99: it should stay under the client timeout. Use httptrace or the pprof goroutine profile to confirm connections are reused and goroutines return to baseline. Then inject a slow upstream and check the timeout fires within the configured budget, with the response body closed and no leaked connections.