Catch a Retry Storm by Capturing Packets with tcpdump

Capture retransmissions and connection churn at the NIC level, correlate them with application retry counters, and size backoff correctly before disabling retries.

~35 minapinetworkingreliability

Task

Use tcpdump to count TCP retransmissions and new connections per second toward one dependency, proving a retry amplification loop.

When to use this

A dependency slows down for ten seconds, every client retries three times immediately, and the queue depth shows 12x normal traffic with no code change deployed.

Prerequisites

tcpdump installed with root or capture capability
A reproducing window where the dependency degrades
Application logs with a retry counter or attempt number

Steps

1. Capture SYN packets toward the dependency to count fresh connections.

sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-syn) != 0 and tcp[tcpflags] & (tcp-ack) == 0 and dst host 10.0.4.12 and dst port 5432' -c 200 2>/dev/null | wc -l

Expected output: In a healthy window the count is near zero per minute; during the incident it climbs sharply, revealing connection churn from retries.

2. Capture retransmissions specifically on the data stream.

sudo timeout 30 tcpdump -i eth0 -nn 'host 10.0.4.12 and (tcp[tcpflags] & tcp-retransmission != 0)' 2>/dev/null | awk '{print $3}' | sort | uniq -c | sort -rn | head

Expected output: Retransmission counts per source port appear; a few ports dominating means the same connections keep retransmitting, not new ones failing.

3. Correlate the packet timeline with application retry logs.

grep -c "attempt=2" /var/log/app/app.log | tail -1; grep "attempt=3" /var/log/app/app.log | tail -5 | jq -r '"\(.ts) \(.dependency)\(if .backoffMs then " backoff=\(.backoffMs)" else " backoff=0" end)'"

Expected output: Retry log lines print with timestamps and backoff values; backoff=0 next to a packet spike confirms retries fire with no wait between them.

4. Fix the client with real backoff and jitter, then re-capture to verify.

async function callWithBackoff(fn, { attempts = 5, baseMs = 200, capMs = 8000 } = {}) {
  for (let n = 0; ; n++) {
    try { return await fn(); }
    catch (err) {
      if (n + 1 >= attempts || !isRetryable(err)) throw err;
      const wait = Math.min(capMs, baseMs * 2 ** n);
      await sleep(wait + Math.random() * wait * 0.3); // full jitter
    }
  }
}

Expected output: Re-running the capture during the same degradation window shows SYNs and retransmissions spread over time instead of one spike.

Troubleshooting

Problem

tcpdump shows no packets at all

Solution

Traffic rides a different interface or a service mesh sidecar; list interfaces with ip addr, or capture on the sidecar port and filter by process instead of host.

Problem

Retransmit filter matches nothing but the app still struggles

Solution

The dependency may accept connections fine but respond slowly; switch the filter to capture payloads and measure response gaps instead of flags.

Problem

Cannot run tcpdump in the container

Solution

Capture at the node level with the pod IP as the host filter, since node-level captures see pod traffic regardless of container privileges.

Limitations

Packet captures show behavior, not intent; pair them with retry counters to attribute volume to specific client code.
Long-running captures at high traffic produce huge files; always bound them with timeout or a packet count.

Retries without backoff are an amplifier

Three immediate retries per request is a 4x multiplier aimed straight at a struggling dependency. tcpdump makes the amplification visible at the packet level: bursts of SYNs and retransmissions during the exact seconds the dependency degraded, coming from your own clients.

# Count fresh connections to the dependency over 30s
sudo timeout 30 tcpdump -i eth0 -nn \
  'tcp[tcpflags] & tcp-syn != 0 and dst host 10.0.4.12' 2>/dev/null \
  | grep -c "S"   # healthy: ~2/min, storm window: 900+/30s

# Same window after adding jittered backoff: ~5/min

Jitter is not optional

Exponential backoff alone still synchronizes clients: everyone retries at 200, 400, 800 milliseconds. Adding 30 percent random jitter decorrelates them, which is the difference between a sawtooth of spikes and a smooth recovery curve you can watch in the re-capture.

Fix the policy, not the symptom

The tempting response to a retry storm is disabling retries. The durable fix is retry budgets and backoff with jitter, plus only retrying idempotent operations. Verify each element with a new capture before declaring the incident closed.