Health Checks: Signals That Actually Reflect Service Health
Health checks decide whether traffic reaches an instance; shallow checks miss broken dependencies, while deep checks amplify outages across the fleet.
Definition
A health check is an endpoint or probe that reports whether an instance can serve traffic. Liveness checks detect unrecoverable states, readiness checks gate traffic, and deep dependency checks verify that the instance can complete real work.
Context and scope
Load balancers, orchestrators, and service meshes all consult health signals, so a badly designed check can remove healthy capacity or keep broken instances in rotation.
Common misconceptions
- A 200 from the process means healthy — it may still fail every request that touches the database
- Deeper checks are always better — checks that call every dependency turn one slow backend into a fleet-wide failure
- Health checks replace monitoring — they are a routing signal, not an observability strategy
Anti-patterns
- Sharing one endpoint for liveness and readiness
- Checking dependencies in liveness probes so a database blip restarts every pod
- Setting probe timeouts shorter than normal p99 latency
Related concepts
Authoritative source
What a check actually reports
A health check is a routing signal, not a diagnosis. It answers one question: should this instance receive traffic right now? That answer can come from the process itself, from a dependency check, or from the orchestrator’s view of the container, and each source fails in a different way.
Three kinds of checks
Liveness asks whether the process is wedged and should be restarted. Readiness asks whether it can serve traffic now. Startup checks grant slow processes extra time before liveness begins. Deep checks that call databases or upstream APIs verify real capability but couple your routing decision to someone else’s availability.
readinessProbe:
httpGet: { path: /ready, port: 8080 }
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet: { path: /live, port: 8080 }
periodSeconds: 10
failureThreshold: 6Designing checks that do not amplify failure
A check should verify what the instance controls: its event loop, its worker pool, its ability to reach the local cache. For example, readiness may confirm the database pool still has a usable connection, but liveness must not fail because the database is briefly unreachable, or every replica restarts and doubles the outage. Timeouts belong below the probe interval so a slow check does not look like a hung process.
Health checks are not monitoring
A passing probe says nothing about latency, error rate, or correctness. Services can be healthy and useless. Compared to monitoring, which measures real behaviour over time, a probe is a single deliberate question asked on a schedule. Keep probes cheap and stable, and measure the rest with SLIs. When a probe depends on a downstream service, that service’s incident becomes yours, at fleet scale.
Choosing thresholds
Set thresholds from observed recovery time, not from defaults. Short periods catch failures quickly but produce flapping; long periods tolerate blips but delay removal. Compare against your deployment cadence: a probe that needs two minutes to notice a bad instance makes rolling deploys slow and risky.