HTTP 504 — Locating Which Layer Timed Out in a Proxy Chain

Trace 504 and Cloudflare 524 errors through CDN, load balancer, and app layers by reading response headers and per-layer logs, then align timeout budgets.

intermediatedebugginghttptimeoutsinfrastructure

Symptoms

Clients receive 504 Gateway Timeout or 524 (a timeout occurred) after a fixed duration like 30s, 60s, or 100s
The application logs show the request still processing (or completed after the client gave up)
Different error codes and durations appear depending on which layer gives up first

Possible causes

Each proxy in the chain (CDN, WAF, LB, ingress) has its own timeout, and the smallest wins
The application genuinely needs longer than any intermediate allows (slow report, cold cache)
The upstream is healthy but slow, and intermediate retries multiply load behind the timeout
Cloudflare 524 specifically: the origin did not answer within 100 seconds (free/pro plans are fixed)

Diagnosis steps

Read the server header of the failure

curl -sI https://app.example.com/slow -H "Authorization: ..." | grep -iE "server|cf-ray|via|alt-svc"

Interpretation

The server header names the layer that produced the response: cloudflare 524, nginx 504, or the ingress — the giver-upper, not the culprit.

Align the durations with layer budgets

grep -iE "timeout" /etc/nginx/nginx.conf; kubectl get ingress -o yaml | grep -i timeout

Interpretation

Matching the observed failure duration to a configured timeout identifies which budget fired; unmatched durations suggest app-level cancellation.

Correlate timestamps across layers

grep "trace-id" cdn.log lb.log app.log | head

Interpretation

App logs showing the request completing at 95s while the client saw 524 at 100s prove the app was late, not down; app completion at 40s with client 504 at 30s indicts a middle layer.

Root cause

A 504 is a deadline declared by a specific proxy, not proof the origin failed. The layer whose timeout expires first generates the response; the origin may be slow, healthy-but-queued, or even already finished after the client-side give-up. Diagnosing requires identifying which budget fired, not which layer is "down".

Fix

Fix the duration when the work legitimately exceeds budgets (async job with 202 plus polling, or a faster path), otherwise align budgets monotonically outward (app > ingress > LB > CDN) so the layer closest to the work controls the deadline, and keep per-layer retries from stacking behind the smallest timeout.

Risk note

Raising every timeout to the slowest path turns one slow endpoint into a capacity sink for the whole chain. Scope long budgets to the routes that need them.

Verify the fix

The slow route completes within its scoped budget, faster routes keep tight guards, and the error layer in logs matches the intended owner of the deadline.

Symptom walkthrough

The fixed duration is the fingerprint — 30, 60, or exactly 100 seconds means a configured budget, not random failure. Identify the responder first:

$ curl -sI https://app.example.com/slow | grep -iE "server|cf-ray"
HTTP/2 524
server: cloudflare
cf-ray: 8a1b2c3d4e5f

524 with a cf-ray means Cloudflare timed out at its fixed 100s waiting for the origin. Next, cross the timestamps: the app completing the request at 95s while the client already saw 524 proves a slow-but-alive origin; the app completing at 40s while the client saw 504 at 30s indicts an intermediate (nginx proxy_read_timeout or an ingress annotation) that gave up early.

Fix in practice

Align budgets outward — the layer closest to the work should own the deadline:

app internal deadline:   90s   (the real work bound)
ingress timeout:        100s
LB timeout:             110s
CDN:                    524-capable or bypassed for this route

When work legitimately exceeds user-facing budgets, change the shape:

POST /reports HTTP/1.1
...
HTTP/1.1 202 Accepted
Location: /reports/jobs/8812

And route-scoped overrides keep one slow path from loosening the whole chain:

location /reports/ {
    proxy_read_timeout 300s;   # scoped: only the slow route pays
}