Recipes
Step-by-step solutions for real development tasks.
699 published
Page 18 of 20
Run Post-Deploy Smoke Checks with Automatic Rollback
Verify the new version with real requests, retry briefly, and roll back automatically when the checks keep failing.
Add a Blameless Postmortem Template
Ship a postmortem template with timeline, contributing factors, and owned action items, and enforce required sections in CI.
Expose Application Metrics for Prometheus Scraping
Add a token-protected /metrics endpoint with default Node.js metrics, a low-cardinality request histogram, and a Prometheus scrape job that reports the target up.
Add Property-Based Tests for Business Rules
Generate inputs with fast-check, assert invariants that must always hold, and let shrinking reduce a failure to the smallest reproducer.
Set a Query Budget per Request
Count SQL statements per request, fail tests that exceed the budget, and expose the count so N+1 regressions surface before users do.
Detect Query Plan Regressions in CI
Capture EXPLAIN output for critical queries as golden files and fail CI when a plan introduces sequential scans or a much higher cost.
Alert on Queue Backlog Age, Not Depth
Page on ApproximateAgeOfOldestMessage with a CloudWatch alarm so a stalled consumer is detected before the queue depth explodes.
Return Standard Rate Limit Headers
Expose RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset on every response and keep 429 bodies consistent so clients can back off.
Gate Reads on Replica Lag
Measure replay lag on the replica, route to the primary above a threshold, and alert before stale reads become visible to users.
Automate Dependency Updates with Renovate
Configure Renovate to group non-major updates, pin GitHub Actions to commit SHAs, require passing CI, and cap the number of concurrent pull requests.
Enforce a Request Body Size Limit
Cap JSON and raw payloads at the parser, reject oversized requests with a stable 413, and prove the limit before a single byte is buffered.
Add a Request Priority Queue
Route interactive traffic ahead of batch work with separate queues, weighted workers, and aging so latency-sensitive requests keep their SLO.
Enable Response Compression Safely
Compress JSON and HTML with gzip, skip pre-compressed assets, keep Vary correct, and measure the transfer win instead of assuming it.
Run a Quarterly Restore Drill
Restore the latest production backup into an isolated database, validate row counts and checksums, and record how long it took.
Rehearse a Rollback Before You Need It
Time a real rollback in staging, verify the old version still handles new data, and write the exact commands into the release runbook.
Fail the Deploy When a Rollout Stalls
Gate the pipeline on kubectl rollout status with a timeout, inspect the stuck ReplicaSet, and roll back instead of hanging.
Check Runbook Links in CI
Run lychee against runbook Markdown on a schedule so dead dashboards, wiki pages, and pager links are fixed before an incident.
Gate Deploys on Schema Compatibility
Run a Schema Registry compatibility check in CI before merging producer changes so backward-incompatible Avro updates never reach consumers.
Schedule Automatic Database Secret Rotation
Enable Secrets Manager rotation on a 30-day schedule, make consumers reload credentials, and prove the old password stops working.
Scan Commits for Secrets Before They Reach main
Run gitleaks on every push and pull request, rotate anything it finds, and block merges until the history is clean.
Declare Service Ownership in a Catalog
Add a catalog-info.yaml with owner, lifecycle, and on-call reference, and validate that every production service has an owner in CI.
Harden Session Cookies Against Theft and CSRF
Set HttpOnly, Secure, SameSite, and __Host- cookie attributes, rotate session IDs on login, and verify that browsers reject unsafe cookie writes.
Run a Monthly SLO Review
Pull error budget burn from Prometheus, compare it with the SLO target, and decide whether to ship features or invest in reliability.
Keep Unique Constraints Working With Soft Deletes
Swap a plain unique constraint for a partial unique index so active rows stay unique while deleted rows stop reserving the value forever.
Version Static Assets for Long-Lived Caching
Use content-hashed filenames with immutable caching, keep HTML revalidated, and verify a deploy never serves mismatched assets.
Partition a Large Table by Time Range
Convert a growing events table into monthly partitions, keep indexes attached, and verify that queries scan only the partitions they need.
Test Tenant Isolation End to End
Attempt cross-tenant reads and writes with another tenant's ids, assert 404, and prove row-level security fails closed without context.
Detect Terraform Drift on a Schedule
Run a read-only plan nightly, use detailed exit codes, and alert on real drift instead of waiting for the next apply to fail.
Enable Terraform State Locking and Unlock Safely
Lock remote state with a conditional write or DynamoDB table, and force-unlock only after confirming no apply is still running.
Isolate Test Fixtures Between Tests
Give each worker its own database from a migrated template, roll back per-test writes, and stop depending on execution order.
Run Integration Tests With Testcontainers
Start a disposable PostgreSQL container per test run, apply the real migrations, and tear it down so tests never share a database.
Inject Timeouts at the Dependency Boundary
Make a downstream call hang in a controlled way, then assert the timeout path returns 504, cancels the request, and releases resources.
Monitor TLS Certificate Expiry
Probe TLS endpoints with blackbox_exporter and alert when a certificate expires within 21 days or the chain fails validation.
Assert Spans in Trace Tests
Use an in-memory span exporter to assert span names, parent-child links, attributes, and error status for a single request.
Retry Transactions on Serialization Failures
Retry the whole unit of work on SQLSTATE 40001 and 40P01 with bounded backoff, and never retry errors that will fail again.
Test Webhook Signature Verification
Sign payloads the way the provider does, then assert that tampering, replay, and timestamp skew are rejected before any handler runs.