Throttle an API with a Token Bucket Limiter
Implement a token bucket that refills at a fixed rate, consume one token per request, and return 429 with Retry-After when the bucket is empty.
Task
Protect an API from burst traffic with a token bucket that allows short bursts but caps the sustained rate.
When to use this
A partner integration polls every second but occasionally bursts 50 calls in a row, exhausting the database pool. The team wants to allow the burst up to a cap and reject the rest with a clear retry signal.
Prerequisites
- An API gateway or a middleware point in the app
- A store for the bucket state that is shared across instances
- A decision on the bucket capacity and refill rate
Steps
1. Implement the bucket with a capacity of 20 and a refill of 5 tokens per second.
class TokenBucket {
constructor(capacity, refill) {
this.cap = capacity; this.refill = refill;
this.tokens = capacity; this.last = Date.now();
}
take() {
const now = Date.now();
this.tokens = Math.min(this.cap, this.tokens + (now - this.last) / 1000 * this.refill);
this.last = now;
if (this.tokens >= 1) { this.tokens -= 1; return true; }
return false;
}
}
Expected output: A bucket that allows a burst of 20 and then sustains 5 requests per second, rejecting the rest.
2. Wire the bucket into middleware and return 429 with Retry-After when take returns false.
if (!bucket.take()) {
return res.set("Retry-After", "1").status(429).json({ error: "rate_limited" });
}
Expected output: A burst beyond capacity gets a 429 with a Retry-After header, and the client backs off correctly.
3. Share the bucket state across instances with a Redis-backed store so the limit is global.
node tools/bucket-store.mjs --redis redis://localhost:6379Expected output: Two app instances behind a load balancer enforce a single shared limit, not a per-instance limit.
Troubleshooting
Problem
The limit is per instance instead of global
Solution
Move the bucket state into Redis with an atomic Lua script, so every instance reads and writes the same tokens.
Problem
Clients retry too aggressively and amplify the burst
Solution
Set Retry-After to the time until the next token, and document the backoff; a jittered retry prevents thundering herd.
Limitations
- A token bucket allows bursts up to capacity, so it does not cap the instantaneous rate to the refill value.
- Clock skew between instances affects the refill calculation; keep the skew under 50 milliseconds with chrony.
Burst, then sustain
A token bucket of 20 with a refill of 5 per second lets a client burst 20 requests and then sustain 5 per second. The bucket refills continuously, so a quiet period earns tokens back up to the capacity.
class TokenBucket {
take() {
const now = Date.now();
this.tokens = Math.min(this.cap, this.tokens + (now - this.last) / 1000 * this.refill);
this.last = now;
if (this.tokens >= 1) { this.tokens -= 1; return true; }
return false;
}
}Share the state
A per-instance limit lets a client multiply its rate by the instance count. Move the bucket into Redis with an atomic Lua script so every instance enforces the same global limit.
node tools/bucket-store.mjs --redis redis://localhost:6379Expected outcome
A burst beyond capacity gets a 429 with Retry-After, the sustained rate is capped at the refill, and two instances behind a load balancer enforce a single shared limit. Clients that honor Retry-After back off without amplifying the burst.