Parallelize Shell Jobs with xargs -P and Preserve Ordering

1. Confirm the workload is embarrassingly parallel.

for f in docs/*.md; do grep -l "TODO" "$f" 2>/dev/null; done | wc -l; ls docs/*.md | wc -l

Expected output: Counts print; if each item touches only its own input and output files with no shared index or accumulation, the parallelization is safe.

2. Fan out with a bounded concurrency and per-item outputs.

ls docs/*.md | xargs -P "$(nproc)" -I{} sh -c 'in="{}"; out="build/$(basename "{}" .md).html"; pandoc "$in" -o "$out"' && ls build/*.html | wc -l

Expected output: All cores run pandoc simultaneously; the output count matches the input count, with each result written to its own file.

3. Keep outputs attributable when writing to a stream.

#!/usr/bin/env bash
# ordered results despite parallel execution
set -euo pipefail
mkdir -p results

printf "%s\n" docs/*.md | xargs -P 4 -I{} bash -c '
  f="{}"
  n=$(basename "$f" .md)
  pandoc "$f" -t html > "results/$n.html" 2> "results/$n.err" || true
'

# recombine IN INPUT ORDER (not completion order)
for f in docs/*.md; do
  n=$(basename "$f" .md)
  [ -s "results/$n.html" ] && cat "results/$n.html"
done > build/combined.html

# report failures without losing successes
grep -l . results/*.err 2>/dev/null && echo "check .err files above" || echo "all clean"

Expected output: combined.html contains every document in original order, while failed items leave a named .err file instead of aborting the batch.

4. Size the concurrency to the resource, not the core count.

time (ls docs/*.md | xargs -P 4 -n 1 pandoc -t html --output /dev/null) 2>&1 | tail -2; time (ls docs/*.md | xargs -P 16 -n 1 pandoc -t html --output /dev/null) 2>&1 | tail -2

Expected output: Both timings print; for CPU-bound work 16 beats 4, for disk-bound work they tie or invert, telling you which bound you are in.

5. Handle inputs safely when filenames can contain spaces.

find docs -name "*.md" -print0 | xargs -0 -P "$(nproc)" -n 1 sh -c 'pandoc "$1" -o "build/$(basename "${1%.md}").html"' _

Expected output: The -print0/-0 pair round-trips names with spaces and quotes intact, and the _ placeholder gives the inner script a clean $1.

Problem

Outputs interleave into garbage on stdout

Solution

Never let parallel children share stdout; write per-item files and concatenate afterward, or use parallel --results dir which does this structure for you.

Problem

A failing item aborts everything

Solution

xargs returns 123 when any invocation fails but still runs the rest; capture per-item exit into .err files and decide policy at the end instead of set -e inside the fan-out.

Problem

Machine crawls at full concurrency

Solution

The jobs are memory- or IO-bound, not CPU-bound; drop -P to cores/2, or serialize the expensive resource with a lock while keeping compute parallel.