ML Model Deployment Gate

Validate machine-learning artifacts against explicit gates before promotion, covering model quality, inference latency, and data drift.

advanced~50 minmlmlopsreliability

Objective

Promote a validated model to production only after quality, latency, and drift gates pass with recorded evidence.

Environment

A working ml environment with access to the target system and its tooling
Configuration and credentials for the ml workflow installed and verified
A rollback or undo path documented before any change is applied

Steps

1. Evaluate offline quality

Run evaluation on a held-out test set and compare metrics against the champion model threshold.

Expected state: Candidate quality meets or exceeds the configured threshold with a recorded report.

2. Shadow-traffic latency check

Route a small slice of production traffic to the candidate and compare p99 latency and error rate against the baseline.

Expected state: Candidate latency and error rate stay within the release budget over the observation window.

3. Check data drift

Compare the incoming feature distribution against the training distribution using a drift metric and documented threshold.

Expected state: Drift stays below the alert threshold or is explicitly accepted with a recorded reason.

4. Promote or hold

Promote the candidate to production if every gate passes; otherwise hold and document the failing metric and next action.

Expected state: A promotion decision is recorded with the gate results and a rollback pointer.

Decision points

Which gate should block promotion?

Model quality below threshold or latency regression beyond budget
Data drift above threshold only

Block on quality or latency first; drift above threshold warrants investigation but may be accepted with sign-off.

Expected outcome

Promote a validated model to production only after quality, latency, and drift gates pass with recorded evidence. The change is verified and a rollback path is confirmed.

Verification

Promote a validated model to production only after quality, latency, and drift gates pass with recorded evidence. Compare before and after metrics and confirm the expected state is reached.

Cleanup

Archive the run artifacts, remove temporary credentials, and update the runbook with any changes.

Overview

Validate machine-learning artifacts against explicit gates before promotion, covering model quality, inference latency, and data drift. The goal is to promote a validated model to production only after quality, latency, and drift gates pass with recorded evidence.

kubectl rollout status deployment/model-server -n prod

Steps

Evaluate offline quality. Run evaluation on a held-out test set and compare metrics against the champion model threshold. Expected: Candidate quality meets or exceeds the configured threshold with a recorded report.
Shadow-traffic latency check. Route a small slice of production traffic to the candidate and compare p99 latency and error rate against the baseline. Expected: Candidate latency and error rate stay within the release budget over the observation window.
Check data drift. Compare the incoming feature distribution against the training distribution using a drift metric and documented threshold. Expected: Drift stays below the alert threshold or is explicitly accepted with a recorded reason.
Promote or hold. Promote the candidate to production if every gate passes; otherwise hold and document the failing metric and next action. Expected: A promotion decision is recorded with the gate results and a rollback pointer.

Verification

Confirm the outcome by checking: promote a validated model to production only after quality, latency, and drift gates pass with recorded evidence. Run the verification command and compare the result against the expected state. If any check fails, stop and investigate before continuing.

Decision points

Which gate should block promotion?
- Model quality below threshold or latency regression beyond budget
- Data drift above threshold only
- Recommendation: Block on quality or latency first; drift above threshold warrants investigation but may be accepted with sign-off.