Audit Every Feature Flag Change

Record who toggled which flag, when, and from where in an append-only table, and alert on production changes to high-risk flags.

~35 minfeature-flagsauditgovernance

Task

Create an auditable trail of feature flag changes without depending solely on a vendor's retention window.

When to use this

A checkout flag is disabled during an incident and nobody remembers who did it or whether it was restored. The vendor keeps ninety days of audit history, but the internal review needs a permanent record joined with deploy data.

Prerequisites

A feature flag service that can deliver change webhooks
PostgreSQL with an append-only audit table
An HTTPS endpoint that verifies webhook signatures

Steps

1. Create an append-only audit table and remove update and delete privileges from the application role.

CREATE TABLE flag_audit (
  id bigserial PRIMARY KEY,
  flag_key text NOT NULL,
  environment text NOT NULL,
  actor text NOT NULL,
  action text NOT NULL,
  before_value jsonb,
  after_value jsonb,
  occurred_at timestamptz NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now()
);

CREATE UNIQUE INDEX flag_audit_dedupe
  ON flag_audit (flag_key, environment, actor, occurred_at);
REVOKE UPDATE, DELETE ON flag_audit FROM app_rw;

Expected output: The table accepts inserts only and a repeated webhook delivery is rejected by the unique index.

2. Verify the webhook signature before writing a row, then insert the event with its original timestamp.

import crypto from 'node:crypto';

function verify(rawBody, signature, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(rawBody)
    .digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(expected), Buffer.from(signature)
  );
}

Expected output: Requests with a valid signature are persisted; requests with a bad or missing signature return 401 and write nothing.

3. Alert when a high-risk flag changes in production outside a deploy window.

SELECT flag_key, actor, action, occurred_at
FROM flag_audit
WHERE environment = 'production'
  AND flag_key IN ('checkout_v2', 'payments_kill_switch')
  AND occurred_at > now() - interval '5 minutes';

Expected output: A production toggle produces one row and triggers the audit alert within the polling interval.

4. Reconcile daily against the vendor audit API to detect dropped webhook deliveries.

node scripts/reconcile-flag-audit.mjs --since 2026-10-20 --until 2026-10-21

Expected output: The script reports zero missing events or lists the event ids that must be backfilled.

Troubleshooting

Problem

Duplicate rows appear after a webhook retry

Solution

Keep the unique index on flag, environment, actor, and timestamp, and treat a duplicate key error as success.

Problem

The actor is unknown or shows the API token name

Solution

Require changes through the UI or a named service account, and map token ids to owners in the flag service.

Limitations

Webhook delivery can lag or drop; daily reconciliation is required for completeness.
The audit table needs its own backup, retention, and access policy, or it becomes a single point of tampering.

Flags are production configuration

A feature flag can turn off checkout, change pricing, or expose a half-finished feature. That makes flag changes production changes and deserving of the same audit trail as deploys. Vendor audit logs help, but retention is finite and export is often manual. An internal append-only table gives you a durable record you can join with deploy timestamps and incident ids.

Record the right fields

Capture the flag key, environment, actor, action, before and after values, and the event time from the vendor, not just the receive time. The original timestamp lets you order events correctly when deliveries are delayed. A unique index on the natural key makes webhook retries idempotent. Revoke UPDATE and DELETE from the application role so the application can only append.

CREATE UNIQUE INDEX flag_audit_dedupe
  ON flag_audit (flag_key, environment, actor, occurred_at);

Verify and alert

Validate the HMAC signature over the raw request body with a constant-time comparison before parsing JSON. Reject unsigned requests. For high-risk flags, query recent production changes and alert the owning channel, including the actor and new value. This catches both mistakes and unauthorized changes.

Reconcile and retain

Webhooks are at-least-once and can be lost during outages. A daily job compares vendor events with stored rows and backfills gaps. Retain audit rows for at least the compliance period, store them in a separate schema with restricted access, and include them in backups. Review the trail monthly for changes made outside deploy windows.