Go — HTTP Response Body Not Closed: Connections and Goroutines Leak

Fix connection growth and read-loop goroutine leaks caused by unclosed http.Response bodies, verify with fd counts and pprof, and close on every path.

intermediatedebugginggohttpresource-leak

Symptoms

Client-side fd count and ESTABLISHED sockets to the upstream grow linearly with request volume
pprof shows goroutines accumulating in net/http persistConn.readLoop for the http client
Latency climbs over hours until the process hits its fd limit and fails with too many open files

Possible causes

resp is used without defer resp.Body.Close() on some path, including error paths
The body is closed only on success; early returns skip it
A HEAD or 204 response is assumed to need no close, but the contract requires it anyway
The body is drained partially then closed, preventing connection reuse and leaking the read loop

Diagnosis steps

Count connections to the upstream

ss -tan state established "( dport = :443 )" | wc -l && ls /proc/$(pgrep app)/fd | wc -l

Interpretation

Growth proportional to requests rather than to real concurrency indicates connections are never returned to the pool.

Inspect the goroutine profile

curl -s http://localhost:6060/debug/pprof/goroutine?debug=1 | grep -c "persistConn.readLoop"

Interpretation

One readLoop goroutine per leaked connection; hundreds of them is the fingerprint of unclosed bodies.

Audit every resp usage path

grep -Rn -A3 "http.Get\|client.Do" src/ | grep -v "Body.Close" | head -20

Interpretation

Do/Get call sites without a nearby Body.Close on all branches are the leak owners.

Root cause

Until the response body is closed, the http client keeps the connection and its persistConn readLoop goroutine alive. An unclosed body therefore leaks one connection plus one goroutine per request, and connections are never reused, compounding the cost with a fresh TCP and TLS handshake each time.

Fix

Close the body on every path immediately after checking the error: defer resp.Body.Close() right after a successful Do, drain the body with io.Copy(io.Discard, resp.Body) when reuse matters, and restructure helpers so the close lives in one place with defer.

Risk note

Closing without draining prevents connection reuse and silently multiplies handshakes. Drain before close on hot paths.

Verify the fix

Drive a fixed request count, then assert ESTABLISHED count returns to pool baseline and readLoop goroutines return to zero within seconds.

Symptom walkthrough

The leak is visible from three independent angles. Sockets to the upstream grow linearly with request count instead of staying at pool size:

ss -tan state established "( dport = :443 )" | wc -l
# 1204   <- pool is 100; 1104 connections will never come back

The goroutine profile names the exact machinery:

$ curl -s "http://localhost:6060/debug/pprof/goroutine?debug=1" | grep -c "persistConn.readLoop"
1104

One readLoop per leaked connection — the client keeps the read loop alive until the body is closed, and it never is. Hours in, the process hits its fd ceiling and every new request fails with too many open files.

Fix in practice

The canonical shape, with close on every path and drain for reuse:

resp, err := client.Do(req)
if err != nil {
    return err // no body to close on transport error
}
defer resp.Body.Close()           // every path from here on

body, err := io.ReadAll(io.LimitReader(resp.Body, maxBody))
if err != nil {
    return err
}
// body fully read: connection returns to the pool for reuse

When you abandon a body early but want the connection back:

io.Copy(io.Discard, io.LimitReader(resp.Body, drainLimit)) // drain, then deferred close

Fix helpers that return resp themselves — they scatter the close obligation across every caller. Either read and close inside the helper and return bytes, or return a wrapper with an explicit Close. And the contract detail people forget: 204 and HEAD responses also require Body.Close().

Verification

　　Drive exactly 10,000 requests through the fixed path, then take the three measurements again: ESTABLISHED sockets to the upstream must return to the configured pool size, readLoop goroutines must drop to zero within seconds, and the process fd count must return to baseline. Re-run with early-abandon traffic (large bodies where you only need headers) to prove the drain path preserves pooling too. Add a periodic pprof snapshot in staging so readLoop count becomes a standing leak alarm rather than a post-mortem tool.