Add a Health Check Endpoint

Expose a liveness and readiness endpoint that checks critical dependencies with a timeout, so orchestrators can restart or remove unhealthy instances.

~20 minoperationshttpkubernetes

Task

Add health endpoints that distinguish a live process from one ready to serve traffic, without leaking secrets or blocking on slow dependencies.

When to use this

A service runs behind Kubernetes or a load balancer. Without probes, traffic reaches instances that cannot reach the database or are still starting up.

Prerequisites

An HTTP service with a router
Access to the dependencies that determine readiness
Knowledge of the orchestrator's probe configuration

Steps

1. Add a liveness endpoint that returns 200 when the process can handle requests. It must not depend on external services, or a database outage will restart every instance.

app.get('/healthz', (_req, res) => res.status(200).json({ status: 'ok' }));

Expected output: GET /healthz returns 200 without touching the database.

2. Add a readiness endpoint that checks dependencies with a short timeout and returns 503 when they are unavailable.

app.get('/readyz', async (_req, res) => {
  try {
    await Promise.race([
      db.query('SELECT 1'),
      new Promise((_, reject) => setTimeout(() => reject(new Error('timeout')), 1000)),
    ]);
    res.status(200).json({ status: 'ready' });
  } catch {
    res.status(503).json({ status: 'not-ready' });
  }
});

Expected output: GET /readyz returns 200 when the dependency responds and 503 when it times out.

3. Configure the orchestrator to use the correct probe for each purpose. Use a longer startup probe for slow boots and a readiness probe to remove instances from rotation.

Expected output: The pod receives traffic only after readiness passes and is not restarted by a dependency outage.

4. Verify the behavior by stopping the dependency in a test environment and observing readiness fail while liveness stays healthy.

curl -i http://localhost:3000/readyz

Expected output: HTTP/1.1 503 while the dependency is down; 200 after recovery.

Troubleshooting

Problem

Pods restart during a database outage

Solution

Liveness is checking the database. Keep liveness process-local; use readiness for dependency health.

Problem

Readiness flaps under load

Solution

Increase the timeout slightly, cache the check for a few seconds, and avoid unbounded queries.

Limitations

Health checks are not authorization; protect detailed diagnostics behind admin access.
A readiness check can only verify reachability, not full query correctness.

Liveness versus readiness

Liveness answers “should this process be restarted?” It should check only the process and its event loop. Readiness answers “should this instance receive traffic?” It can check the database, cache, and required downstream services.

app.get('/healthz', (_req, res) => res.sendStatus(200));

app.get('/readyz', async (_req, res) => {
  const ok = await withTimeout(checkDatabase(), 1000);
  res.status(ok ? 200 : 503).json({ ready: ok });
});

Verify

/healthz returns 200 while the database is down.
/readyz returns 503 while the database is down and 200 after recovery.
The load balancer stops routing to an unready instance.
The orchestrator does not restart healthy processes during a dependency outage.

Keep it cheap

Probes run frequently. Use a lightweight query, cache the result for a few seconds, and never expose connection strings, stack traces, or user data in the response. If readiness depends on a third party, consider a degraded mode rather than failing all traffic.