Feature Flag Rollout Workflow
Ship a change behind a flag off, ramp the percentage in steps while watching the SLO, and kill the flag instantly if the error rate rises, then clean up once stable.
Objective
Release a change to a growing percentage of traffic with an instant kill switch and an SLO gate at every step.
Environment
- A feature flag service that supports percentage rollouts
- A live SLO dashboard that updates within seconds
- A way to kill the flag instantly from a command or a button
Steps
1. Ship off
Deploy the change with the flag off for everyone, and confirm the app behaves as before.
Expected state: The new code is in production but inactive; the baseline SLO is unchanged.
2. Ramp in steps
Turn the flag on for one percent, then five, twenty, fifty, and one hundred, watching the SLO at each step for a full window.
Expected state: Each step holds the SLO; a step that breaches triggers a kill.
3. Kill on breach
If the error rate rises above the threshold at any step, kill the flag instantly and investigate.
Expected state: The flag is off within seconds, the error rate recovers, and the incident is open.
4. Clean up
Once the flag is at one hundred percent for a full window, remove the flag and the old code path.
Expected state: The flag is gone, the old code path is removed, and the change is permanent.
Decision points
How big is the first step?
- One percent for a day
- Ten percent for an hour
One percent for a day, so a bad change affects few users and you see the weekly pattern.
When to remove the flag?
- Right at one hundred percent
- After a full window at one hundred
After a full window, so a weekly batch path that only runs at one hundred has a clean cycle.
Expected outcome
The change reaches one hundred percent with the SLO held at every step, or it is killed instantly at the first sign of trouble.
Verification
Confirm the flag reached one hundred percent or was killed, the SLO held at every completed step, and the flag is removed after a clean window.
Cleanup
Remove the flag and the old code path, archive the rollout artifacts, and record the rollout in the changelog.
Off, then ramp
Deploy the change with the flag off. The new code is in production but inactive. Ramp in steps: one, five, twenty, fifty, one hundred. Watch the SLO at each step for a full window.
flag set new-checkout --percent 1Kill on breach
If the error rate rises at any step, kill the flag instantly. A kill is one command, not a deploy. Investigate from the safety of the off state, not under fire.
Clean up
Once the flag is at one hundred for a full window, remove the flag and the old code path. A flag that stays forever is technical debt. The cleanup is a separate deploy so the rollback stays available during the window.
Verification
The flag reached one hundred or was killed, the SLO held at every completed step, and the flag is removed after a clean window. Start at one percent for a day so a bad change affects few users and the weekly pattern is visible.