Capacity Forecast Workflow
Collect load and saturation metrics over a season, fit a trend with a growth model, and translate the forecast into a headroom request with a review cadence.
Objective
Predict when the service will hit its saturation limit and request headroom in time to provision before the cliff.
Environment
- Six weeks of load and saturation metrics in a time-series store
- A growth model that fits the seasonal pattern
- A provisioning lead time in weeks
Steps
1. Collect the season
Pull six weeks of request rate, CPU, memory, and queue depth at one-minute resolution, and tag by the seasonal driver like weekday or campaign.
Expected state: A dataset with enough samples to see the weekly pattern and the peak day, not only the average.
2. Fit the trend
Fit a model that captures both the seasonal cycle and the long-term growth, and project the saturation date from the current headroom.
Expected state: A forecast with a saturation date and a confidence band, not a single point estimate.
3. Translate to a headroom request
Convert the forecast into a provisioning request with the quantity, the date, and the cost, and route it to the capacity review.
Expected state: A ticket with a date that precedes the saturation date by the provisioning lead time.
4. Review on a cadence
Re-run the forecast every two weeks with fresh data, and adjust the headroom request if the saturation date moves.
Expected state: The forecast tracks reality, and the headroom request is updated before the cliff, not after.
Decision points
Which metric defines saturation?
- CPU at 80 percent
- Queue depth above an SLO threshold
Pick the metric that breaks the SLO first; CPU at 80 percent is a proxy, queue depth is the symptom.
How much lead time do we need?
- Twice the provisioning lead time
- Exactly the lead time
Twice the lead time, so a provisioning slip does not land you on the cliff.
Expected outcome
A headroom request lands before the saturation date with enough lead time to provision, and the forecast is re-run on a cadence so it does not go stale.
Verification
Confirm the projected saturation date is in the future, the headroom ticket precedes it by the lead time, and the forecast error stays within the confidence band.
Cleanup
Archive the model artifacts, record the actual saturation date when it arrives, and use it to calibrate the next forecast.
See the season, not the average
A month of averages hides the peak day. Pull six weeks at one-minute resolution and tag by the seasonal driver. The forecast needs the weekly pattern and the peak, not a flat line.
query.range('rate(http_requests_total[1m])', '-6w', 'now', '1m')Fit, then project
Fit a model that captures the cycle and the growth, and project the saturation date from the current headroom. Report a confidence band, not a single date, so the reader knows how much to trust the number.
Translate to a ticket
The forecast is useless if it does not become a provisioning request. Convert the saturation date into a ticket that precedes it by twice the lead time, route it to the capacity review, and re-run every two weeks.
Verification
The saturation date is in the future, the headroom ticket precedes it by twice the lead time, and the forecast error stays within the band. When the real saturation date arrives, record it and calibrate the next model.