nginx 504 — Upstream Timed Out While Reading Response Headers

Diagnose nginx 504 upstream timed out errors by measuring the backend directly, checking proxy timeouts and buffering, and fixing slow handlers before raising limits.

intermediatedebuggingnginxreverse-proxytimeouts

Symptoms

504 Gateway Timeout returned to the client
upstream timed out (110: Connection timed out) while reading response header from upstream
Only slow endpoints fail, while health checks stay green
The backend eventually logs the request, but the client already received 504

Possible causes

The backend handler takes longer than proxy_read_timeout to produce the first byte
The upstream is unreachable or accepting connections without responding, such as a wedged worker or a full accept queue
A garbage collection pause, lock wait, or synchronous external call blocks the request
proxy_next_upstream is disabled and one bad upstream fails the whole request
Request buffering and large uploads push the exchange past proxy_send_timeout

Diagnosis steps

Read the error log with the upstream address and request

tail -n 200 /var/log/nginx/error.log | grep -E 'upstream timed out|no live upstreams|connect\(\) failed'

Interpretation

The message names whether the timeout occurred while connecting, sending, or reading headers. Reading headers points at a slow application response, not a network failure.

Print the effective proxy timeouts

nginx -T 2>/dev/null | grep -nE 'proxy_(read|connect|send)_timeout|proxy_next_upstream'

Interpretation

Defaults are 60 seconds for read and send. Location-level overrides may differ from what you expect; check the block that actually matches the request.

Time the upstream directly, bypassing nginx

curl -o /dev/null -s -w 'connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}' http://127.0.0.1:3000/slow-endpoint

Interpretation

A TTFB above proxy_read_timeout confirms the application is slow. Fast direct responses point to nginx configuration, DNS, or the network between nginx and the upstream.

Check upstream health and saturation at failure time

ss -ltnp | grep ':3000'; ss -tan state established '( sport = :3000 )' | wc -l

Interpretation

A full accept queue or hundreds of established connections indicates the backend is saturated rather than merely slow.

Root cause

nginx did not receive the upstream response headers within proxy_read_timeout, so it returned 504 and closed the client connection. The upstream request may still be running or may have completed after the client gave up. The timeout is a budget mismatch or a stalled backend, not an nginx defect.

Fix

Fix the slow handler or external dependency first. Then set proxy_connect_timeout low, keep proxy_read_timeout appropriate per location, enable upstream keepalive with HTTP/1.1, and configure proxy_next_upstream for idempotent requests only. Offload genuinely long work to a background job and return a job id instead of holding a request open.

Risk note

Raising proxy_read_timeout ties up worker connections and can cascade into exhaustion under load. proxy_next_upstream on non-idempotent methods can duplicate writes. A 504 does not cancel the upstream request, so long handlers may keep consuming resources after the client is gone.

Verify the fix

Replay the slow request and confirm it returns 200 within the expected budget, with no new upstream timed out entries in the error log. Under load, check that TTFB P95 stays below proxy_read_timeout and that established connections to the upstream remain bounded.

504 is a budget, not a crash

nginx returns 504 when the upstream did not deliver response headers within proxy_read_timeout. The application may still be working, or may have finished after nginx stopped waiting. That distinction matters: raising the timeout changes who waits, not how long the work takes.

2026/10/21 09:14:02 [error] 812#812: *1934 upstream timed out (110: Connection timed out)
while reading response header from upstream, client: 10.0.4.19, server: api.example.com,
request: "GET /reports/monthly HTTP/1.1", upstream: "http://127.0.0.1:3000/reports/monthly"

The phrase while reading response header points at the application. while connecting points at reachability or a full listen backlog.

Measure the upstream without nginx

curl -o /dev/null -s -w 'connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
  http://127.0.0.1:3000/reports/monthly
nginx -T 2>/dev/null | grep -nE 'proxy_(read|connect|send)_timeout'

If TTFB direct is above the configured read timeout, nginx is reporting the truth. Compare the handler’s own timing and downstream calls: a report endpoint that waits on an unindexed query or a synchronous export will always exceed a 60-second budget under load.

Fix the slow path, then align budgets

upstream app_backend {
    server 127.0.0.1:3000;
    keepalive 32;
}

location /reports/ {
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_connect_timeout 2s;
    proxy_read_timeout 120s;   # only for endpoints that legitimately need it
    proxy_next_upstream error timeout;
    proxy_pass http://app_backend;
}

Keep the global timeout conservative and widen it only on the specific location that needs it. Long exports belong in a queue with a status endpoint, so the HTTP request stays short. Enable upstream keepalive to avoid connection churn.

Verify

Replay the failing request and confirm a 200 within budget, then run a load test and watch TTFB P95, established connections to the upstream, and new error log entries. A clean log at the previous failure rate is the signal that the backend, not just the timeout value, changed.