阅读中文版本

Build a Content Moderation Service

A complete moderation service: pre-publish screening, rule and model scoring, review queue, appeals, and false-positive handling tests.

moderation-service/
├── src/pipeline.ts
├── src/rules.ts
├── src/scoring.ts
├── src/review-queue.ts
├── tests/moderation.test.ts
└── package.json

const hit = rules.find((r) => r.test(content));

const action = score > blockAt ? 'block' : score > reviewAt ? 'review' : 'allow';

await db.insertDecision({ itemId, action, reason, score });

await db.enqueue({ itemId, priority, status: 'pending' });

await db.reopen(decisionId, appealReason);

Project overview

Moderation is one of the few features where the cost of a mistake is asymmetric: letting abuse through damages users, and blocking legitimate content damages them just as much. This build makes every decision explainable so neither error happens silently.

You will run cheap rules before an expensive score, produce an explicit action of allow, review, or block with a reason, route borderline items to a human queue, and record each decision so an appeal can reopen it with full context.

The pipeline is ordered by cost rather than by sophistication: regex and blocklists catch the obvious cases quickly, and only what survives reaches the model. That ordering is what keeps moderation affordable at volume.

The point of the decision record is not compliance theater; it is the ability to answer why a specific piece of content was treated a specific way months later, when a reviewer has forgotten the context and the model has changed.

Each stage leaves a working moderation path: rules alone already handle the obvious cases, the score adds judgment, the queue adds a human, and the appeal path closes the loop.

The service is aimed at teams that have a rule list but no consistent record of why anything was decided. It turns moderation from a set of ad hoc checks into a pipeline with an audit trail, without pretending a threshold can replace human judgment.

What you will build

The finished service offers five capabilities, each with a test that drives real content through the pipeline:

• Rule stage
• Scoring stage
• Decision record
• Review queue
• Appeal path

A rule stage handles the cheap, certain cases; a scoring stage handles the gray area with a numeric signal; a decision record captures the reason; a review queue gives humans the borderline items; and an appeal path lets a block be re-examined rather than becoming final.

The design keeps policy out of code. Thresholds, blocklists, and rule definitions live in configuration, so a policy change is an edit reviewed by the people who own the policy rather than a deploy that drags engineering into every decision.

Architecture and data model

A pipeline runs cheap rules first, then a scoring model, producing an action of allow, review, or block with a reason. Reviewable items enter a queue; every decision, including the matched rule or score, is stored for audit and appeal.

Content flows through a rule stage and then a scoring stage, each producing a signal. A decision assembler combines the signals into an action and a reason, and the action determines whether the item publishes, enters the review queue, or is refused.

Every decision is stored with the inputs that produced it: the matched rule or the score, the model version, and the resulting action. That record is what makes an appeal a review of evidence rather than a second opinion.

The review queue is an ordinary table with a status, so items can be claimed, decided, and time-boxed. Treating it as data rather than a separate service keeps the workflow inspectable and avoids a second system that drifts out of sync.

Rules and thresholds are loaded from configuration at boot and carried on each decision, so a threshold change is auditable. Storing the version that applied is the difference between explaining a past call and guessing at it.

Escalation is the composition rule: a rule match can force a block, a high score can force review, and an ambiguous signal falls through to the default action. Making the precedence explicit avoids the bug where a soft signal silently overrides a hard one.

Prerequisites and dependencies

You need storage for submissions and decisions, and a place to keep the rule and threshold configuration.

• Node.js 20 LTS or later
• PostgreSQL 15+
• A moderation policy definition

The dependencies keep the pipeline, storage, and the review workflow in one deployable service:

• pg (^8.12.0) — Decisions and review queue
• express (^4.21.0) — Screen and review endpoints

pnpm init
pnpm add pg express
pnpm add -D typescript tsx vitest

Validate rule syntax at boot so a malformed pattern fails startup rather than silently matching nothing, which is the moderation equivalent of a check that never runs. Keep thresholds in version control so a policy change is a diff.

Directory structure

Separate the rule stage, the scoring stage, decision assembly, the review queue, and the appeal path so each can be tested with fixed inputs:

moderation-service/
├── src/pipeline.ts
├── src/rules.ts
├── src/scoring.ts
├── src/review-queue.ts
├── tests/moderation.test.ts
└── package.json

Phase 1: Rule stage

Run fast deterministic rules before any expensive scoring.

const hit = rules.find((r) => r.test(content));

Cheap rules catch obvious cases and save model calls.

Run the rules first and let them return a decisive action when they are confident. A cheap certainty here saves an expensive model call and keeps the scoring model reserved for the cases that actually need judgment.

Phase 2: Scoring stage

Score borderline content and map thresholds to actions.

const action = score > blockAt ? 'block' : score > reviewAt ? 'review' : 'allow';

A single score with thresholds is easier to tune than many rules.

The scoring stage returns a number, not a verdict; mapping that number to an action is the threshold policy’s job. Keeping score and action separate means a threshold can be tuned without retraining or redeploying the model.

Phase 3: Decision record

Store the action, reason, and evidence for every item.

await db.insertDecision({ itemId, action, reason, score });

Appeals need a traceable decision, not just a verdict.

Assemble the final action from the signals and record the reason alongside it. A decision without a reason is unauditable, and an unauditable decision is one you cannot defend when a creator appeals.

Phase 4: Review queue

Enqueue reviewable items with priority and assign to a reviewer.

await db.enqueue({ itemId, priority, status: 'pending' });

Human review must not lose items.

Route review items into a queue with a status and a deadline. A queue that is merely a filtered view of everything is not a workflow; giving items a claimable state is what makes human review actually happen.

Phase 5: Appeal path

Let a user appeal a block and reopen the decision.

await db.reopen(decisionId, appealReason);

False positives need a correction path.

Reopening a blocked decision restores the original inputs and the reviewer notes, so the second look starts from evidence. An appeal that starts from a blank slate is a coin flip with extra steps.

Testing

Feed clearly benign, clearly abusive, and borderline content and assert allow, block, and review actions. Appeal a block and assert the decision is reopened.

Feed clearly benign, clearly abusive, and borderline content and assert the allow, block, and review actions respectively. The borderline case is the one that proves the threshold policy rather than the model.

Appeal a block and assert the decision is reopened with its original reason intact, then change a threshold from configuration and assert the action changes for the same input without a code change.

Assert that a decision row records the rule id or model version that produced it, because a record that omits the cause cannot answer the question an appeal poses.

Test a malformed rule at startup and assert the service refuses to boot, which is far better than a pattern that silently never matches and quietly lets abuse through.

Add a regression fixture for every nuance you tune, because moderation policies drift and the only thing that keeps a decision reproducible is a test that asserts it on a fixed input.

pnpm vitest run

Deployment and operations

Keep thresholds in configuration so policy changes do not require code deploys. Log model versions so decisions can be explained after a model update.

Keep thresholds in configuration and log the model version with every decision, so a policy or model change can be correlated with a shift in outcomes. A decision that cannot be explained after a model update is a support ticket waiting to happen.

Run review as an asynchronous workflow decoupled from the publish path, so a backlog in human review never blocks legitimate content that the rules already cleared.

Back up decision records for the retention your policy requires, and never delete the reason along with the content, since the reason is the part an appeal needs.

Release threshold changes as reviewed configuration tracked like code, so a change can be rolled back and its effect measured against the decisions it touched.

Ship the rule set and thresholds as a versioned bundle separate from the model, so a rule change can be evaluated independently of a model change and the two effects are never confused in an incident review.

Expose a manual override that records who overrode a decision and why, so the human path is audited with the same rigor as the automated one.

Security considerations

Treat all content as hostile: escape it in review UIs, cap sizes, and never execute model output. Restrict review endpoints to trained staff with audit logs.

Treat submitted content as hostile: escape it in any reviewer UI, because a moderation console is exactly where an attacker would try to store a script for the human who reads it next.

Do not log the full content of blocked items indefinitely; log the decision and a hash, and keep the raw content only for the retention your policy allows. A moderation store is otherwise a convenient archive of everything users tried to hide.

Protect the review queue with its own authorization, since the ability to overturn a block is as sensitive as the ability to block. The reviewer role is privileged and should not be implied by ordinary access.

Keep blocklists and model endpoints in configuration and secrets, and never let the pipeline fail open on an error; an unreachable model should route to review, not auto-approve.

Rate-limit submissions per user and per IP before moderation, since the pipeline is not cheap and an attacker can otherwise turn moderation itself into the denial-of-service surface.

Observability

Track action distribution, review queue depth, and appeal rate. Alert on a sudden shift in the block rate, which indicates a threshold or model drift.

Track the action mix over time: allow, review, and block. A sudden shift in the mix is usually a policy change or a new abuse pattern, and either way it is the signal to look.

Chart the review queue depth and the age of the oldest item, because a growing backlog silently converts review into an indefinite delay, which looks like a block to the user.

Measure the score distribution for allowed and blocked items separately; overlapping distributions mean the threshold is doing work the model cannot justify, which is where appeals originate.

Alert on pipeline errors that force a fallback action, since a moderation system that quietly defaults to allow during an outage is worse than one that fails loudly.

Track appeal outcomes, because the rate at which blocks are overturned is the clearest available measure of whether the threshold is calibrated to real human judgment.

Troubleshooting

High false-positive rate. Thresholds were tuned on one community; recalibrate per audience and let appeals inform the tuning.

Review queue grows without bound. Priorities were not set, so reviewers could not drain the important items first.

When a decision surprises a reviewer, check the applied threshold version first, then the matched rule, then the model score. The decision record names all three, so this order resolves nearly every surprise without re-running anything.

If the review queue stops moving, look at the oldest item and its status; a stuck item usually means a claim that was never released rather than a workflow that is genuinely overloaded.

Limitations and next steps

• No image or video moderation.
• The scoring model is a placeholder interface, not trained here.
• No multi-language policy tuning.

The example ships rules and a scoring interface but not a trained model, so the score stage is a plug point with a deterministic stub until a real classifier is wired in.

Thresholds are global rather than per-category or per-language, so one policy applies across all content. Per-category thresholds are the natural extension once the mix is understood.

The review workflow has no escalation or SLA timers; items wait until a human claims them. Adding deadlines and reassignment is straightforward but beyond the scope of this build.

There is no adversarial feedback loop that learns from overturned appeals, so threshold tuning remains a manual, data-informed process rather than an automatic one.

The pipeline evaluates text only; image and video moderation would need a separate stage with different latency and cost characteristics, which this build does not attempt.