Python — multiprocessing Raises Can't pickle on Windows and macOS

Fix PicklingError and AttributeError in multiprocessing under the spawn start method, pass picklable arguments only, and guard module-level side effects.

intermediatedebuggingpythonmultiprocessingpickling

Symptoms

TypeError: cannot pickle '_thread.RLock' object or PicklingError: Can't pickle <function task at 0x...>
The pool works on Linux but raises on Windows or macOS laptops and CI runners
The error appears in the child process with a traceback through multiprocessing/spawn.py

Possible causes

Spawn re-imports the main module in each child, so lambdas and local functions cannot be pickled to the worker
Arguments include unpicklable objects: locks, sockets, open file handles, database connections
The target function is defined inside __main__ without an importable module path
Module-level code (server startup, heavy loads) runs again in every child because of missing __name__ guards

Diagnosis steps

Confirm the start method in play

python -c "import multiprocessing as mp; print(mp.get_start_method())"

Interpretation

fork on Linux hides the problem; spawn on Windows and macOS forces every argument and function through pickle — that difference is the entire failure surface.

Reduce the failing call to the argument

python -c "import pickle; pickle.dumps(task_args)"

Interpretation

If the minimal dumps raises, the argument itself is unpicklable and no pool configuration can save it.

Audit for module-level side effects

grep -Rn "^[a-zA-Z_]* = .*(" app/main.py | head; grep -c "if __name__" app/main.py

Interpretation

Spawn re-executes module top-level code; a missing main guard turns every child into a second server boot.

Root cause

The spawn start method has no shared memory image with the parent, so it pickles the callable and arguments to reconstruct them in the child. Lambdas, closures, locks, and open handles resist pickling by design, and unguarded module-level code re-runs in every child during reconstruction.

Fix

Pass plain data (dicts, tuples, primitives), reference top-level functions by importable name, replace shared locks with Manager or Queue primitives, and guard all module-level side effects with if __name__ == "__main__". Test with spawn set explicitly on Linux so behavior is platform-independent.

Risk note

Forcing fork globally restores speed but breaks under threads and some libraries (BLAS, Cocoa), and hides the latent pickle constraint from CI.

Verify the fix

Run the pool with multiprocessing.set_start_method("spawn") on Linux CI and assert successful completion; verify child count and that module-level side effects executed exactly once.

Symptom walkthrough

The platform split is the signature — green on Linux CI, red on every Windows laptop:

PicklingError: Can't pickle <function <lambda> at 0x7f8c...>:
attribute lookup <lambda> on __main__ failed

Under spawn, the child is a fresh interpreter: the parent must ship the callable and arguments through pickle. A lambda has no importable name, so shipping fails before any work starts. Test the payload directly — python -c “import pickle; pickle.dumps(args)” — and the failure reproduces without any pool at all, which is the fastest way to isolate it.

Fix in practice

Make everything picklable by construction:

def process_batch(items):        # top-level: importable by name
    return [transform(x) for x in items]

if __name__ == "__main__":      # guard: children re-import this module
    with ProcessPoolExecutor(max_workers=4) as pool:
        results = pool.map(process_batch, chunks)  # chunks: plain data

Replace unpicklable coordination objects with managed primitives:

from multiprocessing import Manager

with Manager() as mgr:
    shared = mgr.dict()   # proxy: picklable, process-safe

On Linux, force the strict mode in tests so the constraint is enforced where it is cheap:

import multiprocessing as mp
mp.set_start_method("spawn", force=True)