Recipes
Step-by-step solutions for real development tasks.
699 published
Page 16 of 20
Isolate Downstreams with Bulkhead Pools
Give each downstream dependency its own bounded connection or concurrency pool so one slow service cannot consume every shared slot.
Invalidate CDN Cache by Tag
Attach cache tags to origin responses and purge a whole tag group on deploy or data change instead of enumerating every URL and query string.
Gate a Canary on Error Rate and Latency
Compare canary metrics with a Prometheus query, abort automatically on regression, and fail closed when the canary has no traffic.
Review Cluster Capacity Headroom
Compare allocatable CPU and memory with requests, simulate losing one node, and record the headroom needed before peak season.
Purge CDN Cache After Every Deploy
Invalidate only the paths that changed, wait for completion, and verify that the new HTML is served from the edge.
Serve Stale Content on Origin Errors
Configure stale-if-error and stale-while-revalidate so a caching proxy keeps serving a recent copy when the origin returns 5xx or times out.
Alert Before a TLS Certificate Expires
Monitor cert-manager renewal status and expiry with a Prometheus rule, so a failed renewal pages someone before the outage.
Run a Controlled Latency Chaos Experiment
Inject 500ms of latency into one dependency with Toxiproxy during business hours, observe SLO burn, and verify timeouts and fallbacks.
Inject a Clock Instead of Calling Date.now
Pass a clock through the domain code, freeze time in tests, and assert expiry exactly at the TTL boundary instead of sleeping.
Add an AWS Budget Alert for Monthly Cloud Spend
Provision a monthly AWS Cost Budget with actual and forecasted thresholds that publish to an SNS topic, so spend anomalies reach the team before the invoice.
Detect and Report CloudFormation Stack Drift
Run drift detection on a schedule, parse resource-level results, and alert on the specific resources changed outside the stack.
Upgrade a Kubernetes Cluster One Minor Version at a Time
Check version skew and removed APIs, upgrade the control plane before nodes, and validate workloads on a canary node pool.
Alert on Runtime Config Drift from Git
Diff live ConfigMaps against the manifests in Git, alert on differences, and stop emergency edits from becoming permanent.
Drain Connections Before Pod Termination
Combine a preStop sleep, a long enough grace period, and readiness gating so in-flight requests finish before Kubernetes removes a pod from service.
Catch Connection Pool Leaks in Tests
Push concurrency past the pool size and assert every acquired client is returned, so a missing release fails in CI instead of hanging production.
Add Container Health Probes That Match Reality
Separate startup, liveness, and readiness probes so slow boots are not killed and a broken dependency does not restart-loop every pod.
Harden a Container with a Read-Only Root Filesystem
Run a Kubernetes container with a read-only root filesystem, drop Linux capabilities, and mount only the writable paths the workload actually needs.
Create a Cloud Cost Anomaly Alert
Enable AWS Cost Anomaly Detection with a dollar threshold and route alerts to the owning team's on-call channel.
Block Deploys That Miss Cost Allocation Tags
Validate required cost tags on infrastructure plans and block resources that would be invisible to the billing report.
Monitor Cron Jobs with Dead-Man's-Switch Heartbeats
Send a heartbeat only after a scheduled backup succeeds, and alert when the heartbeat is late, so silent cron failures are detected within the expected window.
Build a Tenant Data Deletion Job
Delete tenant rows across tables in dependency order, remove object storage prefixes, and write a completion receipt with counts.
Add a Data Integrity Reconciliation Check
Compare row counts and content fingerprints between primary and replica on a schedule, and alert on drift before it reaches a report.
Test Data Masking Before It Ships
Assert that emails, card numbers, and tokens are masked in logs, exports, and fixtures, and that raw values never reach shared storage.
Route Read Queries to a PostgreSQL Replica Safely
Split primary writes from replica reads with explicit consistency rules, request-level pinning, and a replication lag guard so stale reads never leak into user-visible paths.
Validate Pooled Database Connections
Combine keepalives, idle timeouts, lifetime limits, and a checkout probe so dead sockets are replaced instead of failing user requests.
Gate Deploys on Backward-Compatible Migrations
Lint migrations for destructive changes before they run, and require expand and contract releases so old and new code can coexist.
Enforce Database Query Timeouts
Set statement_timeout, lock_timeout, and idle_in_transaction_session_timeout per role and transaction so runaway queries cannot exhaust the pool.
Build a Dead Letter Queue Dashboard
Export dead-letter depth and message age to Prometheus, add a Grafana panel, and give operators a documented replay command.
Freeze Dependency Updates Before a Release
Pin installs with a frozen lockfile and pause automated update pull requests during the release window, then unfreeze deliberately.
Block Vulnerable Dependencies at Pull Request Time
Run GitHub dependency review on every pull request and fail when a new dependency brings a known vulnerability or a denied license.
Fail Over DNS to a Standby Region
Create health-checked primary and secondary records, verify the failover with a forced health check, and account for TTL lag.
Verify DNS TTL Before a Migration
Inspect TTLs across authoritative and public resolvers, lower them ahead of a cutover, and confirm propagation before switching traffic.
Inject an Error Rate Into a Dependency
Return a controlled share of 500s from a fake dependency, then assert retry limits, circuit breaking, and graceful degradation.
Add ETag Revalidation to Read Endpoints
Derive stable entity tags, answer If-None-Match with 304, and keep validators correct when compression or CDNs transform the bytes.
Audit Every Feature Flag Change
Record who toggled which flag, when, and from where in an append-only table, and alert on production changes to high-risk flags.
Add a Feature Flag Kill Switch with Fail-Safe Defaults
Ship a database-backed kill switch that disables a risky feature without redeploying, with a bounded cache, safe defaults, and an audit trail for every toggle.