Add a Feature Flag Kill Switch with Fail-Safe Defaults
Ship a database-backed kill switch that disables a risky feature without redeploying, with a bounded cache, safe defaults, and an audit trail for every toggle.
Task
Implement a database-backed feature kill switch with a fail-safe default and an append-only audit log for every toggle.
When to use this
A new recommendation path starts returning errors or slow responses, and rolling back the deploy takes longer than disabling the feature.
Prerequisites
- A service with PostgreSQL access and an authenticated admin route
- An existing authorization middleware that sets req.user
- A staging environment where the flag table can be renamed during verification
Steps
1. Create the flag table, an append-only audit table, and seed the feature so both enabled and disabled paths can be exercised.
psql "$DATABASE_URL" <<'SQL'
CREATE TABLE IF NOT EXISTS feature_flags (
name text PRIMARY KEY,
enabled boolean NOT NULL DEFAULT false,
updated_by text NOT NULL,
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS feature_flag_audit (
id bigserial PRIMARY KEY,
name text NOT NULL,
enabled boolean NOT NULL,
changed_by text NOT NULL,
changed_at timestamptz NOT NULL DEFAULT now()
);
INSERT INTO feature_flags (name, enabled, updated_by)
VALUES ('recommendations_v2', true, 'bootstrap')
ON CONFLICT (name) DO NOTHING;
SQL
Expected output: psql prints CREATE TABLE twice and INSERT 0 1; the recommendations_v2 row exists and is enabled.
2. Read flags through a five-second cache and fall back to a hard-coded safe default whenever the database is unavailable or the row is missing.
import pg from 'pg';
const pool = new pg.Pool({
connectionString: process.env.DATABASE_URL,
max: 5,
connectionTimeoutMillis: 2_000,
query_timeout: 2_000,
});
const DEFAULTS = new Map<string, boolean>([
['recommendations_v2', false],
['new_checkout', false],
]);
const TTL_MS = 5_000;
const cache = new Map<string, { enabled: boolean; at: number }>();
export async function isEnabled(name: string): Promise<boolean> {
const fallback = DEFAULTS.get(name) ?? false;
const hit = cache.get(name);
if (hit && Date.now() - hit.at < TTL_MS) return hit.enabled;
try {
const { rows } = await pool.query<{ enabled: boolean }>(
'SELECT enabled FROM feature_flags WHERE name = $1',
[name],
);
const enabled = rows[0]?.enabled ?? fallback;
cache.set(name, { enabled, at: Date.now() });
return enabled;
} catch {
cache.set(name, { enabled: fallback, at: Date.now() });
return fallback;
}
}
export function invalidateFlag(name: string): void {
cache.delete(name);
}
export { pool as flagsPool };
Expected output: With the database reachable, isEnabled returns the stored value; with the table missing or the connection failing, it returns false for recommendations_v2.
3. Add an admin-only endpoint that updates the flag and writes the audit row in the same transaction.
import express from 'express';
import { flagsPool, invalidateFlag } from './flags.js';
export const adminRouter = express.Router();
adminRouter.post('/flags/:name', express.json(), async (req, res, next) => {
const enabled = req.body?.enabled;
if (typeof enabled !== 'boolean') {
res.status(400).json({ error: 'enabled must be boolean' });
return;
}
const client = await flagsPool.connect();
try {
await client.query('BEGIN');
const { rowCount } = await client.query(
`UPDATE feature_flags
SET enabled = $2, updated_by = $3, updated_at = now()
WHERE name = $1`,
[req.params.name, enabled, req.user.id],
);
if (rowCount === 0) {
await client.query('ROLLBACK');
res.status(404).json({ error: 'unknown flag' });
return;
}
await client.query(
`INSERT INTO feature_flag_audit (name, enabled, changed_by)
VALUES ($1, $2, $3)`,
[req.params.name, enabled, req.user.id],
);
await client.query('COMMIT');
invalidateFlag(req.params.name);
res.json({ name: req.params.name, enabled });
} catch (error) {
await client.query('ROLLBACK');
next(error);
} finally {
client.release();
}
});
Expected output: A valid toggle commits both rows and returns the new state; an unknown flag returns 404 and writes no audit row.
4. Mount the admin router behind authorization and gate the risky code path on the flag.
import express from 'express';
import { adminRouter } from './admin.js';
import { isEnabled } from './flags.js';
const app = express();
app.use(express.json());
// requireAdmin is the existing authorization middleware.
app.use('/admin', requireAdmin, adminRouter);
async function loadRecommendations(_userId: string, engine: 'v1' | 'v2') {
return engine === 'v2' ? ['item-a', 'item-b'] : ['item-a'];
}
app.get('/api/recommendations', async (req, res) => {
if (await isEnabled('recommendations_v2')) {
res.json({ engine: 'v2', items: await loadRecommendations(req.user.id, 'v2') });
return;
}
res.json({ engine: 'v1', items: await loadRecommendations(req.user.id, 'v1') });
});
Expected output: When the flag is false, the endpoint returns engine v1; when it is true, it returns engine v2 without a deploy.
5. Toggle the flag with an admin token, inspect the audit trail, then prove the fail-safe by making the flag table unreadable and waiting out the cache.
curl -s -X POST http://localhost:3000/admin/flags/recommendations_v2 \
-H "authorization: Bearer $ADMIN_TOKEN" \
-H 'content-type: application/json' \
-d '{"enabled":false}' | jq
psql "$DATABASE_URL" -c "SELECT name, enabled, changed_by, changed_at FROM feature_flag_audit ORDER BY changed_at DESC LIMIT 5;"
curl -s -X POST http://localhost:3000/admin/flags/recommendations_v2 \
-H "authorization: Bearer $ADMIN_TOKEN" \
-H 'content-type: application/json' \
-d '{"enabled":true}' >/dev/null
sleep 6
psql "$DATABASE_URL" -c "ALTER TABLE feature_flags RENAME TO feature_flags_offline;"
sleep 6
curl -s http://localhost:3000/api/recommendations \
-H "authorization: Bearer $USER_TOKEN" | jq -r '.engine'
psql "$DATABASE_URL" -c "ALTER TABLE feature_flags_offline RENAME TO feature_flags;"
Expected output: The first toggle returns enabled false and adds an audit row; after the table is renamed and the cache expires, the endpoint returns engine v1.
Troubleshooting
Problem
A flag is disabled but some instances still serve the feature
Solution
Each process caches the flag for up to five seconds. Check the pod clock and wait one TTL; for faster propagation, publish invalidation through Redis pub/sub.
Problem
A database blip disables a feature that was on
Solution
This is the intended fail-safe for risky features. For low-risk flags, set the default to true and make sure the fallback is tested separately.
Problem
The admin endpoint returns 404 for a flag that exists in another environment
Solution
Flags are per database. Seed the row in every environment and keep the same defaults in code so a missing row still behaves predictably.
Limitations
- The five-second cache means a kill switch is not instantaneous across a fleet; size the TTL to the incident tolerance.
- This is a boolean kill switch, not a percentage rollout or user-targeting system.
- If PostgreSQL is unavailable, every flag falls back to its default, so defaults must encode the safe state.
A kill switch is an operational control
A feature flag that only selects a code path is not a kill switch. A kill switch needs three properties: a safe default that applies when the flag source is unavailable, a bounded propagation time so operators know when the change is effective, and an audit trail that answers who changed what and when. The database row is the source of truth, the in-process cache is a latency optimization, and the hard-coded default is the last line of defense.
Fail-safe does not always mean false. For a new recommendation engine, false is safe. For a flag that controls whether a read-only status page is available, false may cause an outage. Define the default per flag, document it next to the flag name, and test the database-unavailable path in staging. A generic default of false is convenient but not automatically correct.
Update and audit atomically
The admin endpoint updates the flag and inserts the audit row in one transaction. If the audit insert fails, the flag update rolls back; there is no state where a toggle happened without a record. The endpoint rejects non-boolean payloads and unknown flags instead of silently creating a new row, because a typo in a flag name should not create a second, permanently off feature. Authorization belongs in front of the router, and the route should be reachable only from the admin network or an authenticated control plane.
Invalidating the local cache after the commit makes the toggling instance immediate. Other instances still wait for the TTL unless a pub/sub channel broadcasts the invalidation. Five seconds is a reasonable compromise for many services, but it must be an explicit decision: during an incident, five seconds can feel long if every request is failing.
Prove the fallback
The verification sequence toggles the flag off, checks the audit row, toggles it back on, waits out the cache, renames the flag table, and requests the feature again. The response must be v1. Renaming only the flag table avoids breaking authentication and other queries, so the test isolates the flag lookup failure. Rename it back immediately; leaving it offline makes the admin endpoint return errors and prevents further toggles.
Also test the happy path from a clean process. Start the service, request the feature, toggle it, and request again. The first response should use the cached or database value; the second should reflect the admin change within the TTL. If the second response is stale, the cache invalidation is not wired to the same key or the admin route is writing to a different database.
Pitfalls
Do not store flag state only in environment variables; changing one requires a deploy, which defeats the purpose. Do not use the audit table as a general application log with unbounded text; keep it small and append-only. Do not let any admin user toggle a flag without authentication and network restriction. And do not assume a kill switch fixes a dependency outage: disabling a feature may reduce load, but the dependency still needs its own timeout, circuit breaker, and alerting.
Example
import pg from 'pg';
const pool = new pg.Pool({
connectionString: process.env.DATABASE_URL,
max: 5,
connectionTimeoutMillis: 2_000,
query_timeout: 2_000,
});
const DEFAULTS = new Map<string, boolean>([
['recommendations_v2', false],
['new_checkout', false],
]);
const TTL_MS = 5_000;
const cache = new Map<string, { enabled: boolean; at: number }>();
export async function isEnabled(name: string): Promise<boolean> {
const fallback = DEFAULTS.get(name) ?? false;
const hit = cache.get(name);
if (hit && Date.now() - hit.at < TTL_MS) return hit.enabled;
try {
const { rows } = await pool.query<{ enabled: boolean }>(
'SELECT enabled FROM feature_flags WHERE name = $1',
[name],
);
const enabled = rows[0]?.enabled ?? fallback;
cache.set(name, { enabled, at: Date.now() });
return enabled;
} catch {
cache.set(name, { enabled: fallback, at: Date.now() });
return fallback;
}
}
export function invalidateFlag(name: string): void {
cache.delete(name);
}
export { pool as flagsPool };