Docker Hub pull rate limit — toomanyrequests in CI and Kubernetes

Diagnose Docker Hub rate limit errors by checking the pull token and rate limit headers, then authenticate, mirror, and cache images to keep CI and clusters pulling.

beginnerdebuggingdockerregistry

Symptoms

toomanyrequests: You have reached your pull rate limit
ImagePullBackOff on many nodes at the same time
CI jobs fail while local docker pull still works
The error appears after a scale-out or nightly rebuild

Possible causes

Anonymous pulls share a per-IP quota, so a shared NAT or CI runner exhausts it
The registry is not authenticated in the cluster or pipeline
Every job pulls the same large image again instead of reusing a cache
A tag such as latest is repulled on each run, multiplying requests

Diagnosis steps

Reproduce and read the full error

docker pull nginx:1.27

Interpretation

The message names the registry and links to rate limits. A 429 status is quota exhaustion, not a missing image or authentication failure.

Check which account Docker is using

docker info | grep -iE 'username|registry|credential'

Interpretation

An empty username means anonymous pulls. A username from a personal account in shared CI is also a problem because the quota is tied to that account.

Read the quota headers from the registry API

TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/nginx:pull" | jq -r .token); curl -sI -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/library/nginx/manifests/1.27 | grep -i ratelimit

Interpretation

ratelimit-remaining and ratelimit-limit show the current budget. A remaining value of 0 confirms the limit, independent of the local Docker daemon.

Inspect cluster pull events

kubectl describe pod -n app web-6f7c8d9b5-tx2mq | grep -A6 -E 'Events:|Failed'

Interpretation

Failed to pull image with 429 on many nodes points to a shared egress address or an unauthenticated imagePullSecret.

Root cause

Docker Hub enforces pull quotas per account and per anonymous IP address over a rolling window. Container runtimes on different nodes may all appear to come from one NAT address, and unauthenticated CI runners add to the same anonymous bucket. A scale-out therefore multiplies manifest and layer requests until the registry returns 429, which Kubernetes surfaces as ImagePullBackOff.

Fix

Authenticate pulls with a dedicated service account and imagePullSecret, use a pull-through cache or registry mirror close to the cluster, and pin images by digest so layers are reused. Avoid docker.io/library images in production manifests without a mirror, and remove periodic jobs that repull unchanged tags.

Risk note

Storing a personal Docker Hub password in CI is a credential risk; use an access token with read-only scope. Mirrors can serve stale images, so verify digests. Aggressive retry loops against a rate-limited registry can extend the block.

Verify the fix

Pull the image from a node and from CI and confirm no 429 appears. Check the registry headers show remaining quota above zero, and roll a deployment to confirm new pods reach Running without ImagePullBackOff.

Rate limits look like image failures

Kubernetes reports ErrImagePull or ImagePullBackOff, but the underlying registry response is:

toomanyrequests: You have reached your pull rate limit.
You may increase the limit by authenticating and upgrading.

The image exists. The registry refused to serve the manifest because the caller exhausted its quota.

Separate account quota from IP quota

docker info | grep -iE 'username|registry'
TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/nginx:pull" | jq -r .token)
curl -sI -H "Authorization: Bearer $TOKEN" \
  https://registry-1.docker.io/v2/library/nginx/manifests/1.27 | grep -i ratelimit

If the second call returns 429 with ratelimit-remaining: 0, you have measured the quota directly. Anonymous pulls from a shared NAT are the usual culprit: dozens of nodes look like one client.

Reduce and authenticate pulls

Create a read-only access token for a machine account, store it as a Kubernetes imagePullSecret and as a CI secret.
Run a pull-through cache or registry mirror inside the network.
Pin images by digest and reuse cached layers across jobs.
Stop scheduled jobs that pull latest when nothing changed.

docker login docker.io -u ci-puller --password-stdin < token.txt
docker pull nginx@sha256:0f0f0f...

Verify

Pull once from a node, then roll a Deployment and confirm all new pods start without ImagePullBackOff. Recheck the registry headers: remaining quota should be well above zero. If only one node fails, inspect its egress IP separately, because a per-IP anonymous quota can affect a single NAT gateway.