← Debugging Guides

阅读中文版本

Docker — no space left on device

Diagnose Docker disk exhaustion from images, containers, volumes, build cache, logs, or inodes, and reclaim space without deleting active data.

intermediatedebuggingdockerstorage

Symptoms

  • write /var/lib/docker/...: no space left on device
  • Containers fail to start or images fail to pull
  • df shows free space at the host level, but Docker still reports no space
  • The failure recurs days after a cleanup

Possible causes

  • Dangling images, stopped containers, and build cache accumulate
  • Container logs grow without rotation
  • Volumes hold old databases or build artifacts
  • The filesystem is out of inodes even with free bytes
  • Docker's data root is on a small partition separate from the host root

Diagnosis steps

Check bytes and inodes on the Docker filesystem

df -h /var/lib/docker && df -i /var/lib/docker

Interpretation

Free bytes with exhausted inodes produce the same no-space error. The data root may differ from /.

Break down Docker usage

docker system df -v

Interpretation

Images, containers, local volumes, and build cache are reported separately. Identify the largest category before deleting.

Inspect log and volume sizes

du -sh /var/lib/docker/containers/*/*.log /var/lib/docker/volumes/* 2>/dev/null | sort -h | tail

Interpretation

A single container log or volume often dominates. Confirm it is safe to remove or truncate.

Root cause

The filesystem backing Docker's data root has no free blocks or inodes. Accumulated images, logs, volumes, or build cache consumed the space, or the partition is undersized for the workload.

Fix

Remove unused images, stopped containers, and build cache with targeted commands, rotate container logs, archive or delete obsolete volumes after backup, and move the data root to a larger filesystem if needed. Add monitoring and retention.

Risk note

docker system prune -a deletes unused images and cache that may be needed for a fast rollback. Never delete volumes without confirming their contents and backups.

Verify the fix

Confirm df and df -i show headroom, start a container and pull an image, and verify application writes. Set alerts on disk and inode usage and a log rotation policy.

Check bytes and inodes

no space left on device can mean free bytes are zero or inodes are exhausted. Run df -h and df -i on the filesystem that backs /var/lib/docker, which may be a separate mount from the host root. A Docker Desktop VM, rootless setup, or custom data root changes the path.

df -h /var/lib/docker
df -i /var/lib/docker
docker system df -v

Reclaim the right category

docker system df -v separates images, containers, volumes, and build cache. Remove dangling images and stopped containers first. Clear build cache only when you accept slower rebuilds. Truncate or rotate runaway container logs rather than deleting a running container. Volumes are the most dangerous category: inspect and back up before removing.

docker image prune
docker container prune
docker builder prune

Prevent recurrence

Set a log rotation policy in the Docker daemon, add retention for images and build cache, and alert on disk and inode usage before the threshold. If the workload legitimately needs more space, move the data root to a larger volume instead of repeatedly cleaning. A cleanup that runs only after an outage is not capacity management.

Fix and verify

Fix the exhaustion by removing the identified category with targeted prune commands, rotating or truncating runaway logs, and resizing or relocating the Docker data root when demand is legitimate. Verify with df -h, df -i, and docker system df after cleanup, then pull an image and start a container to prove the daemon can write. Confirm active volumes and images needed for rollback were preserved, and that alerts fire before the next threshold.