Debug an SSE Reconnect Storm on the Server Side

Recognize the signature of every EventSource client reconnecting at once, add heartbeat frames, and fix client backoff so idle proxies stop amplifying drops.

~30 minapisseresilience

Task

Diagnose a periodic thundering herd of SSE reconnects, stop idle-timeout disconnects with heartbeats, and stagger client reconnects with backoff.

When to use this

Every sixty seconds the gateway logs a burst of thousands of new SSE connections. Each client reconnects at the same instant, tripling load and eventually cascading.

Prerequisites

Server-side connection logs with per-connection timestamps
Control over the SSE endpoint response stream
The ability to deploy a client-side change or coordinate one

Steps

1. Confirm the periodic pattern in connection logs.

awk '{print substr($4,2,14)}' /var/log/gateway/connections.log | uniq -c | awk '$1 > 100 {print}' | head

Expected output: Minute-bucketed counts show a spike every minute, the signature of a synchronized timeout rather than organic churn.

2. Add a comment heartbeat inside the SSE stream.

res.writeHead(200, {
  "Content-Type": "text/event-stream",
  "Cache-Control": "no-cache",
});
setInterval(() => res.write(": ping\n\n"), 15000);

Expected output: Comment lines keep the connection visibly active for intermediaries, and conforming clients ignore them.

3. Verify the heartbeat defeats the idle timeout.

timeout 180 curl -sN https://api.example.com/stream | grep --line-buffered "^: ping" | head -3

Expected output: Ping comments arrive every fifteen seconds, and the connection survives past the previously fatal sixty-second mark.

4. Stagger client reconnects with exponential backoff and jitter.

function connect(attempt = 0) {
  const es = new EventSource("/stream");
  es.onerror = () => {
    es.close();
    const delay = Math.min(30000, 500 * 2 ** attempt) + Math.random() * 1000;
    setTimeout(() => connect(attempt + 1), delay);
  };
}

Expected output: After a server restart, reconnects spread over tens of seconds instead of arriving in one synchronized spike.

5. Re-check the connection log after both fixes deploy.

awk '{print substr($4,2,14)}' /var/log/gateway/connections.log | uniq -c | awk '$1 > 100 {print}' | wc -l

Expected output: The count of minutes with more than one hundred new connections drops to zero over the same window.

Troubleshooting

Problem

Heartbeat present but drops continue

Solution

An intermediary buffers the stream instead of streaming it; confirm with curl -N that bytes actually arrive incrementally, and disable response buffering on the proxy.

Problem

Clients ignore Last-Event-Id

Solution

The server must echo the id field on every event and resume from the given offset; without ids, clients replay the whole stream and amplify load.

Problem

Backoff causes permanent disconnection

Solution

Cap the maximum delay and keep retrying at the cap; also listen for connectivity events so an awake client reconnects immediately.

Limitations

Heartbeats trade a small constant bandwidth cost for connection stability; pick the interval against your strictest intermediary timeout.
Fixing only the server leaves misbehaving third-party clients capable of storms; rate-limit new SSE connections as a backstop.

One idle timeout becomes a stampede

Server-Sent Events connections look idle to intermediaries because the server only speaks when there is news. At sixty seconds a proxy kills them, every browser reconnects in the same tick, and the gateway experiences a synchronized spike it mistook for a traffic surge. Two fixes end the storm: heartbeats that keep the stream visibly active, and client backoff with jitter that desynchronizes whatever reconnects still happen.

// Server: keep the stream alive for intermediaries
setInterval(() => res.write(": ping\n\n"), 15000);

// Client: stagger reconnects after a drop
const delay = Math.min(30000, 500 * 2 ** attempt) + Math.random() * 1000;

Read the herd in the log

The diagnosis is one awk away: bucket new connections by minute and look for periodic spikes. Organic churn is flat; a timeout-driven storm is a comb. Once you see the comb, the fix targets idle timeouts, not capacity, and adding servers would only delay the next stampede.

Always pair heartbeat with backoff

Heartbeats stop the routine drops, but restarts and deploys still disconnect everyone at once. Backoff with jitter turns one synchronized wave into a spread-out trickle the gateway can absorb. Deploy the client fix before the next scheduled restart, or the deploy itself triggers the storm you just read about.