中文
devexamples

Xargs Parallel Job Control Batching

Process file batches in parallel with xargs -P and -N, controlling concurrency, preserving order, and capping memory.

BashCli Tooling

Code

snippet
#!/usr/bin/env bash
set -euo pipefail

readonly MAX_JOBS=4
readonly BATCH_SIZE=10

process_batch() {
  local batch_number=$1
  shift
  echo "batch ${batch_number}: 处理 $# 个文件"
  for file in "$@"; do
    local size
    size=$(stat -c%s "$file" 2>/dev/null || stat -f%z "$file")
    printf '%s\t%d bytes\n' "$(basename "$file")" "${size}"
  done
}
export -f process_batch

# -P 控制并发数，-N 每次传一批参数，--keep-order 稳定输出
find ./data -name '*.log' -type f -print0 \
  | xargs -0 -P "${MAX_JOBS}" -n "${BATCH_SIZE}" \
      bash -c 'process_batch "$@"' _ \
  | sort

# 不分批的高并发形态：一文件一任务
find ./data -name '*.small' -type f -print0 \
  | xargs -0 -P "${MAX_JOBS}" -I{} \
      sh -c 'gzip -k "{}" && echo "compressed {}"'

# 失败可见性：--no-run-if-empty 防空输入报错
find ./data -name '*.missing' -type f -print0 \
  | xargs -0 -r --no-run-if-empty echo "found"

Related Examples