Linux disk full — No Space Left on Device With Free Space

Diagnose Linux disk-full errors by checking blocks and inodes, finding deleted-but-open files, and locating log or container growth before cleaning safely.

beginnerdebugginglinuxdisk

Symptoms

No space left on device appears even though df shows free space
Applications cannot write logs, temp files, or database segments
Package installs and container pulls fail immediately
df -h shows 100 percent use on / or /var

Possible causes

A log file or application data directory grew without rotation
A deleted file is still held open by a process, so the blocks are not released
The filesystem ran out of inodes rather than bytes
Docker images, layers, or build cache filled /var/lib/docker
Ext4 reserved blocks prevent non-root writes while df still shows free space

Diagnosis steps

Check bytes and inodes

df -h && df -i

Interpretation

Use percent near 100 means blocks are exhausted. IUse near 100 with free blocks means too many small files, often session files or mail queues.

Find the largest directories on the root filesystem

du -xhd1 / 2>/dev/null | sort -h | tail -15

Interpretation

-x stays on one filesystem so mounted volumes do not distort the result. The top entries identify /var/log, /var/lib/docker, or application data.

Find deleted files still held open

sudo lsof +L1 | head -30

Interpretation

A large deleted file with a live process keeps its blocks allocated. df still reports them used until the process closes the file or is restarted.

Check journal and container usage

journalctl --disk-usage && docker system df

Interpretation

Systemd journals and Docker images are common hidden consumers. Both can be large while application directories look normal.

Root cause

A filesystem reports ENOSPC when it cannot allocate a block or an inode for a new file. Deleting a file only frees space once no process holds it open, and a full inode table fails writes even with free blocks. Log growth, container layers, and databases that preallocate or never truncate are the usual sources, while reserved blocks explain failures for non-root users on a filesystem df calls full but not exhausted.

Fix

Truncate growing logs instead of deleting open ones (`: > /var/log/app.log`), configure logrotate with size and retention, remove old images and build cache, and clean temporary files with a defined retention. If deleted open files are responsible, restart the owning service or truncate through /proc/<pid>/fd. Grow the volume or inode table when the workload legitimately needs more, and keep 10 to 15 percent headroom on system filesystems.

Risk note

Deleting files under /var/lib/docker or a database directory can corrupt data or break running containers. Truncating an application log without rotation configuration only resets the clock. Removing files by age may delete the one archive needed for an audit; check ownership and retention first.

Verify the fix

Recheck df -h and df -i and confirm both have headroom, then write a test file of the expected size and confirm it succeeds. Tail the application log to ensure logging resumed, and monitor usage for a day to verify the cleanup was not just a one-time deletion.

ENOSPC has two dimensions

write error: No space left on device

df -h reports block usage, df -i reports inode usage. A filesystem can fail writes with plenty of free bytes if it has no inodes left, which is common with millions of small session or cache files.

df -h
df -i
du -xhd1 / 2>/dev/null | sort -h | tail -15

Check for space that is used but invisible

A deleted file held open by a process keeps its blocks allocated:

sudo lsof +L1 | head -30

The SIZE/OFF column shows the held bytes. Truncating through /proc/<pid>/fd/<fd> or restarting the process releases them. Also check journalctl --disk-usage and docker system df; container layers and journals are frequently the real consumer.

Clean with intent

sudo journalctl --vacuum-size=500M
docker image prune -a --filter "until=168h"
sudo truncate -s 0 /var/log/app/error.log

Configure logrotate for application logs, move large archives to object storage, and keep at least 10 percent free on / and /var. On ext4, reserved blocks can cause non-root write failures before df reports 100 percent; leave headroom or adjust tune2fs -m deliberately.

Verify

Confirm df -h and df -i both show headroom, then create a file of the expected workload size and write to it. Tail the application log to see logging resume. Monitor for a day: if usage climbs back quickly, the cleanup treated the symptom and the growth source still needs a retention or sizing fix.