Configure Docker Container Log Rotation Before It Fills the Disk

Set json-file log driver rotation options per container or daemon-wide, verify max-size and max-file take effect, and alert on remaining disk.

~20 mindeploydockeroperations

Task

Bound container log disk usage with rotation settings and prove the bounds hold under sustained log output.

When to use this

A debug-level logging change filled the root disk overnight. Every container on the host died simultaneously and the node became unreachable over SSH.

Prerequisites

Docker hosts using the default json-file log driver
Root or docker group access to change daemon or container settings
A way to generate sustained log volume for testing

Steps

1. Check current log sizes on the host.

du -sh /var/lib/docker/containers/*/ | sort -rh | head -5

Expected output: One or more containers show multi-gigabyte log files, confirming the unbounded growth.

2. Set rotation in the daemon configuration.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Expected output: Every container without explicit options now rotates at ten megabytes and keeps at most three files, capping disk use per container at thirty megabytes.

3. Override per container where volume differs.

docker run --log-driver json-file \
  --log-opt max-size=50m \
  --log-opt max-file=5 \
  app:2.4.1

Expected output: A chatty container gets a larger but still bounded budget of two hundred fifty megabytes.

4. Restart the daemon and verify the settings apply.

systemctl restart docker && docker inspect app --format '{{.HostConfig.LogConfig}}'

Expected output: The inspect output shows the effective driver and options, proving the change reached running containers.

5. Stress the bounds and confirm rotation.

docker logs -f app & for i in $(seq 1 4000); do docker exec app sh -c "head -c 4096 /dev/urandom | base64" >> /dev/null; done; du -sh /var/lib/docker/containers/*/
-log*

Expected output: After generating tens of megabytes of log output, the on-disk usage stays within the configured cap instead of growing.

Troubleshooting

Problem

Options do not apply to existing containers

Solution

Log options are fixed at container creation; recreate containers, for example via a rolling redeploy, to inherit the new defaults.

Problem

Committed log volume already fills the disk

Solution

Truncate the json log files in place with truncate -s 0 rather than deleting them, because deleting breaks the file handle docker holds.

Problem

Logs are needed longer than rotation keeps them

Solution

Rotation is a disk guard, not retention; ship logs to a central system with its own retention before capping local files.

Limitations

max-file bounds local debugging history only; anything compliance-related must live in centralized storage.
Alternative drivers like local or journald have different options; apply the equivalent bounds for whichever is in use.

Logs are a disk-fill attack you run yourself

The default json-file driver appends forever, so one accidental debug-level change becomes an overnight root-disk exhaustion that kills every container on the host. Rotation bounds the damage mechanically: ten megabytes per file, three files per container, no dependency on anyone remembering to clean up. The settings are trivial; the discipline is applying them before the incident instead of after.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Fixed at creation, so redeploy to apply

Log configuration is captured when the container is created, which means daemon-wide defaults only reach containers started afterwards. Plan the rollout as a normal rolling redeploy, and use docker inspect on the LogConfig of a fresh container to verify the options actually landed rather than assuming.

Guard the guard with an alert

Rotation caps per-container usage, but the sum of all containers plus images plus volumes can still creep toward a full disk. Alert on filesystem usage above eighty percent on every docker host. The alert converts the remaining failure mode from an outage into a maintenance ticket.