DNS Resolution Incident Workflow
Distinguish DNS failure from service failure, verify resolution from multiple vantage points, and mitigate with TTL-aware record changes.
Objective
Diagnose and mitigate DNS-layer failures quickly, avoiding both false service rollbacks and record changes that propagate too slowly to help.
Environment
- dig or nslookup available on jump hosts and locally
- Access to the DNS provider console or zone files
- Knowledge of current TTLs on critical records
Steps
1. Test resolution from multiple vantage points
Query the failing name from inside the cluster, from a public resolver, and from a different region to scope whether failure is local, provider-wide, or global.
Expected state: A matrix of answers (or SERVFAIL/timeouts) exists naming the scope before any change.
2. Separate DNS from service failure
Connect to the resolved IP directly (curl --resolve) to prove the service itself is healthy while the name is not.
Expected state: Direct-IP access works while the name fails, confirming the DNS layer as the fault domain.
3. Check propagation and caches
Compare authoritative answers against local resolver caches, and verify whether stale or negative-cache entries (SOA minimum) are extending the outage.
Expected state: The incident distinguishes authoritative failure from cached-negative extension with TTL evidence.
4. Mitigate within TTL reality
Apply the least-global fix: adjust the failing record, or fail over to a secondary zone or provider, acknowledging current TTLs in the plan.
Expected state: The mitigation accounts for how long caches will hold old or negative answers.
5. Verify end-to-end and post-fix
Confirm resolution from the original failing vantage points, then review TTL policy on critical records (low TTL for failover-eligible names).
Expected state: Resolution works everywhere tested, and a TTL policy change is filed if the TTL prolonged the incident.
Expected outcome
DNS incidents scoped by vantage-point evidence, mitigated with TTL-aware changes, and closed with a TTL policy that shortens the next one.
Verification
The incident record contains the vantage-point matrix, the direct-IP service proof, and the TTL propagation timeline; post-incident, failover-eligible records carry low TTLs.
The name is not the service
When a name stops resolving, the reflex is to roll back the service — which fixes nothing when the service is healthy behind a broken resolver chain. Testing from multiple vantage points and connecting by IP separates the layers in minutes.
# scope the failure
for vp in cluster jumphost 8.8.8.8 1.1.1.1; do
echo "== $vp =="
dig +time=2 +tries=1 @$vp api.example.com A +short
dig +time=2 +tries=1 @$vp api.example.com | grep -E "status|SERVER"
done
# cluster: SERVFAIL (resolver 10.0.0.53)
# public: 203.0.113.7 (fine -> provider/local scope)
# prove the service itself is healthy
curl --resolve api.example.com:443:203.0.113.7 \
https://api.example.com/health # 200 OK
# authoritative vs cache
dig @ns1.example.com api.example.com +short # 203.0.113.7
dig api.example.com @10.0.0.53 +short # (negative-cached)Negative caching is the invisible multiplier
A failed lookup gets cached too: resolvers hold SERVFAIL for the SOA minimum, often minutes. A DNS record fixed at the provider still fails for users until negative entries expire, so the mitigation plan must include that propagation window.
TTLs are set for the next incident
A five-minute TTL costs slightly more query volume every day and buys a five-minute failover when DNS matters most. Records that participate in failover belong at low TTL; records that never change can stay long. That decision is the incident’s real postmortem action.