API Rate Limit Tuning Workflow
Tune API rate limits from real traffic distribution, setting per-key limits that protect the backend without rejecting the 99th percentile of legitimate callers.
Objective
Set rate limits that protect the backend from abuse while rejecting fewer than 1% of legitimate requests, using the observed traffic distribution.
Environment
- An API with a rate limiter
- A week of request logs with per-key counts
- A backend capacity number (max requests per second)
Steps
1. Profile the traffic
Compute the per-key request rate distribution from the logs and identify the 99th percentile of legitimate callers.
Expected state: A distribution and a 99th-percentile rate per key.
2. Set the limit
Set the per-key limit above the 99th percentile but below the point where the aggregate exceeds backend capacity.
Expected state: A limit that protects the backend and rejects under 1% of legitimate traffic.
3. Deploy and watch
Deploy the limit, monitor the rejection rate and the backend latency, and confirm the rejection rate is under 1%.
Expected state: The rejection rate is under 1% and the backend latency is stable.
4. Adjust for campaigns
If a known campaign raises a key rate, raise its limit temporarily rather than rejecting the burst.
Expected state: Campaign traffic is served, not rejected.
Decision points
Per-key or global limit?
- Per-key for fairness
- Global for simplicity
Per-key, because a global limit lets one noisy caller starve the rest.
Reject or queue over-limit requests?
- Reject with 429
- Queue with backpressure
Reject, because a queue adds latency and memory; a 429 tells the client to retry.
Expected outcome
The rate limit protects the backend from abuse and rejects fewer than 1% of legitimate requests, with a process for temporary adjustments during campaigns.
Verification
Confirm the rejection rate is under 1% over a day of traffic, the backend latency is stable, and a deliberate burst over the limit is rejected with 429.
Cleanup
Keep the limit and the dashboard; remove the temporary campaign adjustments when the campaign ends.
Profile the real traffic
A rate limit that is set by guess rejects legitimate callers or fails to protect the backend. The right limit comes from the traffic distribution: the 99th percentile of per-key rates is the legitimate ceiling, and the backend capacity is the aggregate ceiling.
awk '{print $1}' requests.log | sort | uniq -c | sort -rn | percentile 99Set the limit between the two ceilings
The per-key limit goes above the 99th percentile so legitimate callers are not rejected, and the aggregate of all per-key limits stays below the backend capacity so the backend is not overwhelmed. If the two ceilings conflict, the backend needs more capacity, not a tighter limit.
LIMIT=$((P99 * 1.2)) # 20% headroom above the 99th percentileDeploy and watch the rejection rate
Deploy the limit and watch the rejection rate. A rejection rate over 1% means the limit is too tight or the traffic shifted; a rejection rate of 0% with rising backend latency means the limit is too loose.
Verification
The rejection rate is under 1% over a day, the backend latency is stable, and a deliberate burst over the limit gets a 429. Campaign adjustments are temporary and reverted when the campaign ends.