Java deadlock — Read jstack and Fix Lock Ordering

Diagnose Java deadlocks by capturing a jstack thread dump, reading the deadlock report and lock owners, then enforcing a global lock order or using tryLock timeouts.

advanceddebuggingjavaconcurrency

Symptoms

Requests hang indefinitely while CPU usage stays low
jstack reports Found one Java-level deadlock
Two threads are BLOCKED waiting on monitors held by each other
The application recovers only after a restart

Possible causes

Two code paths acquire the same two locks in opposite order
Nested synchronized blocks span calls into other components
A callback acquires a second lock while already holding one
wait/notify or Condition signaling is used without a consistent state check

Diagnosis steps

Find the JVM process

jps -l

Interpretation

The output lists JVM PIDs and main classes. Pick the application process, not the build tool or IDE helper.

Capture a thread dump with lock information

jstack -l 12345 > /tmp/threaddump.txt && grep -A40 'Found one Java-level deadlock' /tmp/threaddump.txt

Interpretation

The report names the deadlocked threads and the monitors each waits for. The stack frames show the exact lines acquiring the locks in opposite order.

Cross-check with jcmd

jcmd 12345 Thread.print -l > /tmp/threaddump2.txt

Interpretation

A second dump confirms the wait is persistent rather than a transient stall. jcmd works when jstack is unavailable or blocked by a security manager.

Map the hot threads to CPU

top -H -p 12345

Interpretation

Deadlocked threads consume no CPU. High CPU with blocked requests suggests a livelock or GC issue instead, so this separates the two.

Root cause

A deadlock requires two threads to hold locks the other needs. Java monitors are not ordered globally, so if thread A locks account 1 then account 2 while thread B locks account 2 then account 1, each waits forever. The JVM detects the cycle for intrinsic monitors and prints it in the thread dump, but it cannot break the deadlock automatically.

Fix

Define a global lock order, for example by sorting account IDs before locking, and apply it in every path. Keep lock scope small and avoid calling external code while holding a lock. Replace nested synchronized blocks with ReentrantLock.tryLock(timeout) when ordering cannot be guaranteed, and unlock in finally. Use concurrent collections or single-writer designs to remove the shared lock entirely.

Risk note

tryLock with a timeout converts a deadlock into a failure that must be handled and retried; ignoring the false return reintroduces the bug. Killing threads is unsafe and deprecated. Restarting the JVM clears the symptom without fixing the ordering, so the deadlock returns under the same load.

Verify the fix

Reproduce the concurrent workload and capture jstack again; it should contain no 'Found one Java-level deadlock' section. Confirm throughput recovers and blocked-thread counts return to zero, then add the scenario to a stress test so the ordering is exercised in CI.

Low CPU plus hung requests

A deadlock does not burn CPU. Threads are blocked on monitors, so the process looks idle while every affected request waits forever. The JVM can detect cycles between intrinsic locks and reports them in a thread dump.

Found one Java-level deadlock:
=============================
"worker-1":
  waiting to lock monitor 0x00007f... (object 0x000000..., a Account),
  which is held by "worker-2"
"worker-2":
  waiting to lock monitor 0x00007f... (object 0x000000..., a Account),
  which is held by "worker-1"

Capture the evidence

jps -l
jstack -l 12345 > /tmp/threaddump.txt
grep -A40 "Found one Java-level deadlock" /tmp/threaddump.txt

Take two dumps a few seconds apart. If both show the same BLOCKED threads with the same lock owners, it is a persistent deadlock, not a slow operation.

Remove the cycle

The reliable fix is a global order:

public void transfer(Account from, Account to, long amount) {
    Account first = from.id() < to.id() ? from : to;
    Account second = first == from ? to : from;

    synchronized (first) {
        synchronized (second) {
            from.debit(amount);
            to.credit(amount);
        }
    }
}

When ordering is impossible, use ReentrantLock.tryLock(200, MILLISECONDS) for both locks and release on failure:

if (!lockA.tryLock(200, MILLISECONDS)) return false;
try {
    if (!lockB.tryLock(200, MILLISECONDS)) return false;
    try { /* critical section */ } finally { lockB.unlock(); }
} finally { lockA.unlock(); }

Verify

Re-run the concurrent test and capture jstack again; the deadlock section must be absent. Watch blocked thread counts and throughput. Add the transfer or callback scenario to a stress test so the lock order is exercised continuously, and review other paths that acquire the same pair of locks to ensure they follow the same order.