Rust — Poisoned Mutex: Another Task Failed While Holding the Lock

Handle poisoning errors from std Mutex in Rust, understand why panics poison locks, recover with into_inner or explicit state validation, and prevent poisoning.

intermediatedebuggingrustconcurrencymutex

Symptoms

Err(PoisonError) from lock() after an unrelated panic occurred inside a critical section
Every subsequent user of the mutex fails with poisoning error even though their own code is correct
One bug in one code path escalates into a service-wide outage of the guarded resource

Possible causes

A panic or early ?-return unwound through a critical section while the guard was alive
The poisoned state propagates: the first failure marks the lock, and all later lockers receive Err
Code unwraps the lock result, so the poisoning becomes a second, cascading panic
A bug fix removed the original panic but the long-running process still carries the poisoned lock

Diagnosis steps

Find the original panic

RUST_BACKTRACE=full cargo run 2>&1 | grep -B3 -A12 panicked

Interpretation

The first panic in the timeline — not the poisoning error — is the root cause; its backtrace shows the critical section that unwound.

Audit lock usage for unwrap

grep -Rn "lock().unwrap()\|lock().expect" src/ | head -20

Interpretation

Every unwrap converts poisoning into a new panic, amplifying one failure into many; these lines are the cascade surface.

Decide the recovery policy

cargo check

Interpretation

into_inner when the invariant is provably restored, explicit validation when it is not — the policy must be written, not defaulted.

Root cause

std::sync::Mutex poisons itself when a panic unwinds while a guard is alive, because the protected data may be mid-mutation and its invariant broken. Every later lock() then returns Err(PoisonError) to prevent operating on possibly corrupted state — a safety feature that turns one panic into a persistent resource failure.

Fix

Fix the original panic first. For recovery, use into_inner() when the invariant is verifiably intact, or validate and repair state explicitly before continuing; park() and retry is never right. Long term, keep critical sections panic-free and small, and consider replacing std Mutex with parking_lot (no poisoning) or tokio::sync::Mutex for awaits.

Risk note

into_inner blindly assumes the invariant survived the unwinding. Validate the state or document why the data cannot be mid-mutation.

Verify the fix

A test that panics inside the critical section, then exercises recovery, asserting the guarded data is consistent and later lockers succeed.

Symptom walkthrough

The cascade has a clear timeline. First, an ordinary panic inside a critical section:

thread 'worker' panicked at 'index out of bounds', src/inventory.rs:88

Then, forever after, every user of that mutex fails:

Err(PoisonError<...>) at src/inventory.rs:31

The poisoning is the design working as intended: a panic unwound while the guard was alive, so the protected struct may be half-mutated — later lockers receive Err instead of operating on possibly broken state. Read the timeline, not the last error: the root cause is the first panic, the rest is containment.

Fix in practice

Fix the original panic — nothing else is durable until then. Then choose a recovery policy for the containment layer:

// invariant provably atomic (all-or-nothing update): recover directly
let mut state = match lock.lock() {
    Ok(g) => g,
    Err(poisoned) => poisoned.into_inner(), // data intact by construction
};

// invariant repairable: validate then fix
let mut state = match lock.lock() {
    Ok(g) => g,
    Err(poisoned) => {
        let mut g = poisoned.into_inner();
        if !g.valid() { g.rebuild(); }   // explicit repair, documented
        g
    }
};

Prevention shapes the long term: keep critical sections small and panic-free, run validation logic outside the lock, and consider parking_lot::Mutex (no poisoning) or tokio::sync::Mutex (await-safe, no poisoning) where the semantics fit the workload.

Verification

　　Encode the timeline in a test: panic inside the critical section deliberately, catch it with catch_unwind, then run the recovery path and assert the guarded data is consistent and subsequent lockers receive Ok. Re-run with the original panic fixed and assert no poisoning occurs at all. Both assertions together prove the containment works and the cause is gone.