阅读中文版本

Isolate HTTP 502 Upstream Reset

Correlate gateway access logs with upstream connection resets to prove which backend instance is closing connections early.

awk '$9==502 {print $NF}' gw.log | sort | uniq -c | sort -rn | head

grep -f <(grep 502 gw.log | awk '{print $NF}') upstream.log | head

What this recipe does

Map gateway 502s to the specific upstream instance and connection phase that is resetting.

A load balancer returns 502 sporadically while backend dashboards stay green, hiding connection-level problems.

Prerequisites

• Gateway access logs with upstream metadata
• Upstream request logs
• Correlation ids propagated end to end

Procedure

1. Extract 502 events with upstream host and phase from gateway logs.

awk '$9==502 {print $NF}' gw.log | sort | uniq -c | sort -rn | head

Expected result: The upstream address responsible for most 502s stands out by frequency.

2. Join the gateway event with the upstream request log by correlation id.

grep -f <(grep 502 gw.log | awk '{print $NF}') upstream.log | head

Expected result: The upstream may show no matching request, proving the connection failed before the handler ran.

How to confirm it worked

Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows one upstream instance is identified as the reset source, and removing it from the pool eliminates the 502 rate.

Troubleshooting

No upstream log at all — A reset before request dispatch points at the connection pool or a crashing worker.

Only one instance affected — Drain and inspect that instance for keep-alive or resource exhaustion issues.

Limitations

• Requires end-to-end correlation ids.
• Does not distinguish TLS resets from TCP resets by itself.
• Gateway logs may truncate upstream detail.