Debug HTTP/2 Stream Resets and Mysterious Failed Requests

Reproduce intermittent HTTP/2 RST_STREAM errors with nghttp, read the error code, and fix the framing or intermediary that resets streams silently.

~30 minapihttp2debugging

Task

Capture an HTTP/2 RST_STREAM error, decode its error code, and identify whether the origin or an intermediary is resetting the stream.

When to use this

After enabling HTTP/2, one API in ten requests fails with a reset error no curl retry reproduces. The server logs show the request never arrived.

Prerequisites

nghttp or h2load from the nghttp2 toolkit installed
Direct network access to the origin to bypass intermediaries for comparison
Access to load balancer or CDN configuration if present

Steps

1. Reproduce the failure with verbose HTTP/2 framing.

nghttp -v https://api.example.com/v2/checkout 2>&1 | grep -iE "rst_stream|goaway|error"

Expected output: The trace prints the stream id and the error code, for example RST_STREAM stream=5 error_code=2 (INTERNAL_ERROR).

2. Hit the origin directly to bypass the intermediary.

nghttp -v --header "host: api.example.com" https://origin.internal:443/v2/checkout 2>&1 | tail -5

Expected output: If direct requests never reset, the intermediary is the resetting party and the origin is exonerated.

3. Check for known content-length mismatches in the trace.

nghttp -v https://api.example.com/v2/checkout 2>&1 | grep -iE "content-length|end_stream" | head -10

Expected output: A declared content-length larger than the body sent, with no end_stream, matches error_code 8 (CANCEL) from intermediaries that detect framing drift.

4. Compare HTTP/1.1 behavior on the same path.

curl -s --http1.1 -o /dev/null -w "%{http_code}\n" https://api.example.com/v2/checkout

Expected output: Consistent success over HTTP/1.1 confirms the defect is HTTP/2-specific framing rather than application logic.

5. Fix the framing bug and cap concurrent streams while verifying.

for i in $(seq 1 50); do nghttp -u https://api.example.com/v2/checkout >/dev/null 2>&1 || echo "fail $i"; done

Expected output: After the fix, fifty sequential requests report zero failures, and the loop becomes the regression check.

Troubleshooting

Problem

Error code 1 (PROTOCOL_ERROR) on large uploads

Solution

An intermediary mishandles CONTINUATION frames or window sizes; update it, or cap request body size under the failing threshold until patched.

Problem

Failures only under concurrency

Solution

The peer enforces a low SETTINGS_MAX_CONCURRENT_STREAMS and resets excess streams; serialize requests or respect the advertised limit in the client pool.

Problem

GOAWAY appears right before failures

Solution

The server is draining for a graceful restart; the client should reconnect on GOAWAY and replay only unfinished requests, which most SDKs do automatically.

Limitations

Error codes are coarse; distinguishing framing bugs from middlebox bugs still requires the direct-versus-proxied comparison.
Some clients hide resets as generic network errors, so reproduce with nghttp before trusting SDK messages.

A reset is an error code, not a mystery

HTTP/2 multiplexes many streams over one connection, and any party can reset a single stream without killing the rest. Clients surface this as a vague network error, but the wire carries a specific error code. nghttp prints every framing event, so the failure stops being intermittent noise and becomes a numbered, documented failure you can attribute to the origin or to an intermediary.

nghttp -v https://api.example.com/v2/checkout 2>&1 \
  | grep -iE "rst_stream|goaway|error"

# prints e.g.
# [RST_STREAM stream=5 error_code=2 (INTERNAL_ERROR)]

The direct-versus-proxied experiment

Two measurements isolate the resetting party: one straight to the origin with a host header override, one through the public path. If resets only appear on the public path, the origin code is innocent and the conversation moves to the CDN or load balancer firmware. This experiment settles in minutes what log archaeology cannot settle in days.

Respect the advertised limits

Error code 7 REFUSED_STREAM under concurrency usually means the peer refuses streams beyond SETTINGS_MAX_CONCURRENT_STREAMS. The correct client response is a bounded connection pool that honors the setting, not retries that immediately re-exceed it and get reset again.