Go Deadlock — all goroutines are asleep

Diagnose Go deadlocks from channel sends, WaitGroup misuse, and lock ordering by reading goroutine dumps from SIGQUIT, test timeouts, and pprof profiles.

advanceddebugginggoconcurrencydeadlock

Symptoms

fatal error: all goroutines are asleep - deadlock!
The process exits with status 2 and prints goroutine stacks
go test hangs until panic: test timed out
An HTTP handler never returns while other requests still work

Possible causes

A send on an unbuffered channel has no receiver, or a receive has no sender
sync.WaitGroup.Wait runs before Add, or a Done call is missing on an error path
A sync.Mutex is locked twice in the same goroutine, which is not reentrant
Two goroutines acquire the same two mutexes in opposite order
A worker exits without closing a results channel that the consumer ranges over

Diagnosis steps

Reproduce with a test timeout to get a full goroutine dump

go test -timeout 15s -run TestCheckout ./internal/checkout

Interpretation

The panic prints every goroutine stack. Look for chan send, chan receive, or sync.runtime_SemacquireMutex and read the top frames to find the blocked pair.

Include all goroutines in the crash output

GOTRACEBACK=all go test -timeout 15s ./...

Interpretation

The default crash hides some runtime goroutines. GOTRACEBACK=all shows the whole picture, which matters for WaitGroup and worker-pool deadlocks.

Dump goroutines of a running server through pprof

curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=2' | head -80

Interpretation

debug=2 prints full stacks. Goroutines stuck at the same chan send or semacquire frame across two dumps are deadlocked or permanently blocked, not merely slow.

Check for data races and vet findings that precede the hang

go vet ./...; go test -race -timeout 60s ./...

Interpretation

A race on the state guarded by the mutex often explains inconsistent lock ordering. Vet also reports copylocks and lost cancel functions.

Root cause

Every goroutine that could make progress is blocked waiting for another goroutine that will never run. The runtime only reports a deadlock when all goroutines are asleep; a partial deadlock, where other goroutines keep running, appears as a hang instead and must be found from stack dumps or pprof.

Fix

Ensure every channel send has a receiver that is already running, or give the channel a buffer sized to the producer count. Add WaitGroup counters before starting goroutines, close result channels in the producer, use defer wg.Done and defer mu.Unlock, and establish a single lock order for multi-mutex code. Use select with a context or timer when waiting must be bounded.

Risk note

Adding a buffer can postpone a deadlock instead of fixing it. sync.Mutex is not reentrant, so a nested Lock in the same goroutine blocks forever. Sending SIGQUIT terminates the process after dumping stacks, so use it on a replica, not the only instance.

Verify the fix

Run go test -race -timeout 60s ./... repeatedly and confirm it completes. Under load, take two pprof goroutine dumps a minute apart and check that no goroutine remains in chan send, chan receive, or semacquire at the same frame, and that the goroutine count is stable.

The runtime only detects total deadlock

fatal error: all goroutines are asleep - deadlock! is printed when the scheduler finds no runnable goroutine and no timer or network poller that could wake one. It is a strong signal, but the absence of this message does not mean there is no deadlock: if a metrics endpoint or background ticker keeps running, the same blocked handler becomes a silent hang.

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan send]:
main.main()
    /app/main.go:9 +0x37

The stack names the exact operation. chan send with no counterpart, chan receive on a channel nobody closes, and semacquire on a mutex are the three common shapes.

Produce a complete dump

Tests are the easiest place to reproduce it:

go test -timeout 15s -run TestCheckout ./internal/checkout
GOTRACEBACK=all go test -timeout 15s ./...

For a running server, expose net/http/pprof on an internal port and fetch goroutine stacks with debug=2. Take two dumps a minute apart: a goroutine parked at the same frame in both is not slow, it is stuck.

The usual causes

Unbuffered channel send before the receiver goroutine starts.
wg.Wait() called before wg.Add(1), or a Done skipped on an error return.
Double Lock() on a sync.Mutex; Go mutexes are not reentrant.
Lock order inversion between two mutexes.
A producer returns early without closing the results channel that the consumer ranges over.

func process(ctx context.Context, jobs <-chan Job) error {
    var wg sync.WaitGroup
    for j := range jobs {
        wg.Add(1)
        go func(j Job) {
            defer wg.Done()
            handle(ctx, j)
        }(j)
    }
    done := make(chan struct{})
    go func() { wg.Wait(); close(done) }()
    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

The context select prevents an unbounded wait, but it does not cancel handle unless handle respects the context.

Verify

Run the race detector and repeat the test enough times to cover scheduling variation. In production, compare goroutine profiles over time: stable counts and no repeated blocked frames are the evidence that the deadlock is gone, not just that the process has not exited yet.