Add a gRPC Health Check Probe

Expose the standard grpc.health.v1 service, wire Kubernetes probes through grpc-health-probe, and flip to NOT_SERVING during shutdown.

~35 mingrpckuberneteshealth-check

Task

Give Kubernetes an accurate readiness and liveness signal for a gRPC service instead of probing a sidecar HTTP port that does not reflect gRPC state.

When to use this

A payments service speaks gRPC on port 50051. The existing TCP probe only proves the socket is open, so a pod with a broken database pool keeps receiving traffic, and rolling updates fail because probes fire before the gRPC server is listening.

Prerequisites

A gRPC server written in Go, Java, Python, or Node.js
Kubernetes 1.24 or newer with kubectl access
The grpc-health-probe binary available in the container image
grpcurl installed locally for manual verification

Steps

1. Register the standard health service and report the serving state of each gRPC service name.

import (
  "google.golang.org/grpc"
  "google.golang.org/grpc/health"
  healthpb "google.golang.org/grpc/health/grpc_health_v1"
)

func registerHealth(srv *grpc.Server) *health.Server {
  hs := health.NewServer()
  healthpb.RegisterHealthServer(srv, hs)
  hs.SetServingStatus("payments.v1.Payments", healthpb.HealthCheckResponse_SERVING)
  return hs
}

Expected output: grpc.health.v1.Health/Check answers SERVING for payments.v1.Payments.

2. Replace the TCP probes with exec probes that call the standard health service.

readinessProbe:
  exec:
    command: ["/bin/grpc_health_probe", "-addr=:50051", "-service=payments.v1.Payments"]
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 2
livenessProbe:
  exec:
    command: ["/bin/grpc_health_probe", "-addr=:50051"]
  periodSeconds: 15
  failureThreshold: 3
startupProbe:
  exec:
    command: ["/bin/grpc_health_probe", "-addr=:50051"]
  failureThreshold: 30
  periodSeconds: 2

Expected output: Pods become Ready only after the gRPC server answers SERVING, and slow starts are covered by the startup probe.

3. Call the health endpoint directly with grpcurl to confirm the reported status.

grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check

Expected output: {"status": "SERVING"}

4. Mark the service NOT_SERVING during graceful shutdown so load balancers stop sending new RPCs before the listener closes.

hs.SetServingStatus("payments.v1.Payments", healthpb.HealthCheckResponse_NOT_SERVING)
time.Sleep(10 * time.Second) // let clients observe the change
srv.GracefulStop()

Expected output: Readiness fails first and the pod is removed from endpoints before the process exits.

Troubleshooting

Problem

The probe logs connection refused during startup

Solution

Use the startupProbe shown above; liveness should not begin until the gRPC listener is accepting connections.

Problem

grpc-health-probe returns NOT_FOUND for the service name

Solution

The service string must match exactly what was registered; register the empty service name as an overall server health signal.

Limitations

The health service reports only the states you explicitly set; it does not inspect database health automatically.
grpc-health-probe is a separate binary that must be copied into the image, while native Kubernetes gRPC probes are still newer.

Why TCP probes are not enough

A TCP probe succeeds as soon as the socket accepts connections, which can happen before interceptors, database pools, and migrations are ready. A gRPC service also has no HTTP endpoint to probe unless you add one. The standard grpc.health.v1.Health service solves both problems: it is language-neutral, supported by grpc-health-probe, and can report per-service status.

Wire readiness and liveness

Readiness should call the specific service name. Liveness should only prove the process and gRPC runtime can answer, so it uses the empty service name. The startup probe gives a slow image up to 60 seconds without liveness killing it. Keep timeoutSeconds short so a wedged server fails the probe instead of hanging the kubelet.

readinessProbe:
  exec:
    command: ["/bin/grpc_health_probe", "-addr=:50051", "-service=payments.v1.Payments"]
  periodSeconds: 10

Verify and drain

grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check should return SERVING. During shutdown, set the status to NOT_SERVING, wait for endpoint propagation, then call GracefulStop. This removes the pod from load balancing before in-flight RPCs are cut, which prevents UNAVAILABLE bursts during rollouts.

Keep it honest

Update the serving status when a critical dependency fails, and set it back when recovered. Do not make the health check run a heavy query on every probe; a cached flag updated by a background check is enough. If the service uses TLS, add -tls and the correct server name to the probe arguments.