Go — Goroutine Count Climbs Steadily: a Leak Found with pprof

Diagnose steadily growing goroutine counts, capture pprof goroutine profiles, read the blocked-on stacks, and close the channel or context that never fires.

advanceddebugginggogoroutinespprof

Symptoms

runtime metrics show goroutine count rising linearly with request volume and never falling after load ends
pprof goroutine profile shows thousands of goroutines parked on chan receive or select with identical stacks
Memory RSS grows alongside the count because each leaked goroutine pins its stack and captured variables

Possible causes

A goroutine waits on a channel that no one ever sends to or closes
A select includes a ctx.Done() case but the context was never canceled by the owner
A worker reads from a queue that the producer stops feeding without closing
An HTTP response body is not closed, keeping the read loop goroutine alive

Diagnosis steps

Graph the goroutine count over time

curl -s http://localhost:6060/debug/pprof/goroutine?debug=1 | head -3; watch -n10 "curl -s http://localhost:6060/debug/pprof/ " | grep -o 'goroutine_count": [0-9]*'

Interpretation

A count that tracks requests and stays elevated after load ends is a leak; a count that plateaus and returns to baseline is just steady-state concurrency.

Group goroutines by stack

curl -s "http://localhost:6060/debug/pprof/goroutine?debug=1" | grep -A3 "^goroutine" | grep -E "chan receive|select|sync" | sort | uniq -c | sort -rn | head

Interpretation

Thousands of goroutines sharing one blocked stack is the leak signature; the stack names the wait and the code that owns it.

Reproduce with a bounded driver

go test -run TestLeak -count=1 -v ./worker/ && goleak.VerifyTestMain(m)

Interpretation

goleak fails the test if goroutines outlive it, turning the leak into a deterministic regression check.

Root cause

Goroutines blocked forever on a receive or select are never garbage collected, because a blocked goroutine is reachable from the runtime scheduler. Each one pins its stack plus every object it captured, so a per-request goroutine that misses its shutdown signal accumulates memory indefinitely.

Fix

Guarantee the exit signal fires: contexts must be canceled by their owner with defer cancel(), producers must close channels when done, select loops must include ctx.Done(), and every loop needs a reachable termination path. Add goleak to package tests to keep the invariant.

Risk note

Forcing goroutine exits mid-operation can drop in-flight work. Cancel cooperatively and let the goroutine finish its current unit before returning.

Verify the fix

After the fix, drive a fixed request count, then assert the goroutine count returns to pre-load baseline within seconds of load ending.

Symptom walkthrough

The leak announces itself as a monotonic counter in the runtime profile:

$ curl -s http://localhost:6060/debug/pprof/
goroutine_count: 8421

Then group by wait state — this is the actual diagnosis:

$ curl -s "http://localhost:6060/debug/pprof/goroutine?debug=1" | head -40
# ... 6800 @ 0x45b7c1 0x44d5c2
# chan receive, 9 minutes
#   main.(*Worker).listen(...)
#       /app/src/worker.go:87

Six thousand eight hundred goroutines parked on the same chan receive for nine minutes, all inside one function: that function is leaking one goroutine per request because nothing ever closes or sends on the channel it waits for. Each parked goroutine holds its stack and everything it captured — which is why RSS climbs in lockstep.

Fix in practice

Give every goroutine a reachable exit. The context pattern covers most cases:

func (w *Worker) listen(ctx context.Context) {
    defer close(w.done)
    for {
        select {
        case <-ctx.Done():
            return // reachable exit: owner cancels
        case job := <-w.queue:
            w.process(job)
        }
    }
}

ctx, cancel := context.WithCancel(context.Background())
defer cancel() // the owner responsibility — missing defer is the leak

Producers own channel closure:

func produce(out chan<- Job) {
    defer close(out) // consumers exit their range loop
    for _, j := range jobs {
        out <- j
    }
}

And keep the regression net in tests:

import "go.uber.org/goleak"

func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m)
}