Python — Interpreter Hangs at Exit Because of Non-Daemon Threads

Find why Python waits for non-daemon threads at shutdown, identify the thread that never finishes, and join threads with timeouts at program exit.

intermediatedebuggingpythonthreadingshutdown

Symptoms

The script prints its final result but the process never exits, requiring kill or CI timeout to end
threading.enumerate at exit shows one or more threads stuck in sleep, recv, or an infinite loop
KeyboardInterrupt reaches the main thread but the process still does not terminate

Possible causes

A worker thread was started without daemon=True and blocks on a socket read or queue.get without timeout
A polling loop with time.sleep has no exit condition wired to shutdown state
The main thread finished but CPython shutdown joins every non-daemon thread before interpreter teardown
KeyboardInterrupt is swallowed inside the worker thread loop

Diagnosis steps

List threads at the hang point

python -c "import threading, time; t=threading.Thread(target=lambda: time.sleep(10**9)); t.start(); print(threading.enumerate())" &

Interpretation

The enumerate output names the stuck thread and its target function; the main thread disappears while the blocker remains.

Dump thread stacks on demand

py-spy dump --pid <pid>

Interpretation

py-spy shows exactly which line each thread rests on — recv without timeout, queue.get, or a sleep loop — without modifying the program.

Check daemon flags at creation

grep -Rn "Thread(target" src/ | grep -v "daemon"

Interpretation

Thread constructors lacking daemon=True (or the daemon property set before start) are the ones the interpreter will wait for.

Root cause

CPython joins all non-daemon threads before finishing interpreter shutdown. A worker blocked indefinitely on I/O or in a loop without an exit condition therefore keeps the process alive after the main thread completed, and the process appears hung despite all real work being done.

Fix

Make shutdown cooperative: join worker threads with a timeout at exit, design loops to poll a threading.Event for cancellation, use timeouts on queue.get and socket operations, and mark genuinely auxiliary threads as daemon=True only when killing them abruptly is acceptable.

Risk note

Daemon threads are killed mid-operation at interpreter exit, which can truncate writes or leave locks held. Use them for liveness signals, not for work that matters.

Verify the fix

Trigger every exit path (normal completion, KeyboardInterrupt, SIGTERM handler) and assert the process terminates within a fixed bound with the correct exit code.

Symptom walkthrough

The program does everything right except stop. Ask the runtime what it is waiting for:

import threading
for t in threading.enumerate():
    print(t.name, t.daemon, t.native_id)
#MainThread          True  4211
#worker-poller       False 4264   <- the interpreter is joining this

A non-daemon worker still listed after main finished is the exact thread the shutdown sequence waits on. Then point py-spy at the process for the resting line:

py-spy dump --pid 4211
# Thread worker-poller: queue.get (blocking, no timeout)

recv without a timeout, queue.get without a timeout, and sleep loops without exit conditions are the three classics.

Fix in practice

Wire cancellation through an Event and make blocking calls bounded:

stop = threading.Event()

def worker():
    while not stop.is_set():
        try:
            item = q.get(timeout=1.0)
        except queue.Empty:
            continue          # re-checks stop every second
        process(item)

# at exit, in the main thread:
stop.set()
t.join(timeout=10)
if t.is_alive():
    log.warning("worker did not stop in 10s")

For socket workers, set a socket timeout or use selectors so reads wake up to re-check the stop flag. Mark threads daemon=True only for things whose mid-operation death is harmless — heartbeats, metrics flushers. Work with side effects gets the cooperative path, because daemon threads are killed without releasing locks or finishing writes.