Hotfix Deployment Workflow

Branch from production, make the minimal fix, run the targeted test suite, ship straight to production, and cherry-pick back to main so the fix is not lost on the next release.

intermediate~30 minhotfixreleaseincident

Objective

Ship a minimal fix to production in minutes without bypassing tests, and ensure the fix lands in main so it is not lost.

Environment

A production branch that can be branched from
A targeted test suite that runs in under five minutes
A deploy pipeline that can ship a single commit

Steps

1. Branch from production

Create a hotfix branch from the production commit, not from main, so the fix applies to what is running.

Expected state: A branch that shares the exact code in production, ready for the fix.

2. Make the minimal fix

Write the smallest change that fixes the issue, add a regression test, and run the targeted suite.

Expected state: The fix is small, the regression test passes, and the targeted suite is green.

3. Ship to production

Deploy the hotfix branch through the pipeline, watch the SLO, and confirm the issue is resolved.

Expected state: Production serves the fix, the SLO recovers, and the issue is closed.

4. Cherry-pick to main

Cherry-pick the hotfix commit to main so the next release includes it, and resolve any conflict by hand.

Expected state: main contains the fix, and the next release will not regress it.

Decision points

Fix forward or roll back?

Fix forward if the fix is clear
Roll back if the cause is unknown

Roll back if the cause is unknown; a guess shipped under pressure is a second incident.

Run the full suite?

Targeted suite only
Full suite

Targeted suite for speed; the full suite runs on main after the cherry-pick.

Expected outcome

The fix is in production in minutes, the SLO recovers, and the fix is in main so the next release does not regress it.

Verification

Confirm production serves the fix, the SLO recovers, the regression test passes, and main contains the cherry-picked commit.

Cleanup

Delete the hotfix branch, link the commit in the incident, and add the regression test to the main suite.

Branch from what is running

A hotfix branch from main includes unreleased changes that are not in production. Branch from the production commit so the fix applies to what is actually running.

git checkout -b hotfix/issue-1234 $PRODUCTION_SHA1

Minimal fix, targeted tests

Write the smallest change that fixes the issue and add a regression test. Run the targeted suite, not the full suite; speed matters and the full suite runs on main after the cherry-pick.

Cherry-pick back

Ship the hotfix to production, then cherry-pick the commit to main. A fix that lives only on the hotfix branch is lost on the next release. Resolve any conflict by hand and run the full suite on main.

Verification

Production serves the fix, the SLO recovers, the regression test passes, and main contains the cherry-picked commit. Roll back if the cause is unknown; a guess under pressure is a second incident. Delete the hotfix branch after the cherry-pick lands.