Review Checklist Enforcement Workflow

Encode the checklist that reviewers repeat into automation where possible and structured prompts where not, keeping humans for judgment.

beginner~25 mincode-reviewautomationquality

Objective

Free reviewer attention for design and correctness judgment by automating the mechanical checks and structuring the judgment ones.

Environment

CI with lint, type-check, and custom-check capability
A PR template supporting a structured checklist
A recurring source of review findings to mine

Steps

1. Mine repeated review comments

Collect the review findings from recent months and classify them: mechanical (catchable by tooling) versus judgment (needs a human).

Expected state: A classified findings list exists with frequency per item.

2. Automate the mechanical class

Encode the top mechanical findings as CI checks: lint rules, forbidden imports, generated-code drift checks, size guards.

Expected state: Automated checks fail PRs for the violations reviewers used to type.

3. Prompt the judgment class

Turn the frequent judgment findings into structured review prompts in the PR template: questions about error paths, concurrency, and test strength.

Expected state: Reviewers answer named questions instead of remembering to ask them.

4. Retire comments the checks replaced

Reviewers stop repeating what automation catches, and the checklist version notes which comment classes moved to tooling.

Expected state: The comment-class frequency shifts toward design and correctness topics.

5. Re-mine quarterly

Repeat the findings mining quarterly, promoting new repeated comments into automation or prompts, keeping the checklist alive.

Expected state: The checklist evolves with the codebase's failure patterns.

Expected outcome

Review conversations dominated by design and correctness, with mechanical findings caught by CI and judgment prompts ensuring coverage.

Verification

Mechanical comment classes disappear from review threads; the quarterly mining shows the new dominant classes being addressed.

Every repeated comment is an automation candidate

Reviewers typing the same thirty comments per week are a lint rule waiting to be written. Mining review history, automating the mechanical class, and prompting the judgment class turns the checklist from memory burden into machinery.

mined findings (6 months, by frequency):
  MECHANICAL (54% of comments):
    unused import / dead code        -> lint rule
    missing error handling on await  -> custom check
    test missing for new branch      -> coverage gate
    forbidden cross-layer import     -> eslint boundary
  JUDGMENT (46%):
    error path semantics
    concurrency assumptions
    test assertion strength

PR template prompts (judgment):
  [ ] walked the failure path: what happens
      when the dependency returns 503?
  [ ] concurrency: any shared mutable state?

result: mechanical comments -86%, review
threads now 71% design/correctness

The PR template is the judgment checklist

Judgment questions cannot be automated, but they can be impossible to skip. Structured prompts in the template — failure path walked, concurrency considered, assertions strong — convert reviewer experience into coverage, without depending on any individual’s memory that day.

Quarterly re-mining keeps it current

The failure patterns of a codebase evolve: new architecture introduces new repeated comments, and old automated rules decay into noise. Repeating the mining quarterly keeps the split between tooling and prompts aligned with what actually recurs.