Golden Signals
Golden signals are latency, traffic, errors, and saturation — four metrics that give a fast, actionable view of any service's health.
Definition
The four golden signals are latency, traffic, errors, and saturation. Latency is how long requests take, ideally split into successful and failed requests. Traffic is the demand on the system, such as requests per second. Errors are the rate of failed requests, including explicit failures and wrong content. Saturation is how full a constrained resource is, such as CPU, memory, connections, or queue depth.
Context and scope
Golden signals come from Google's SRE practice and are used for dashboards and alerts on request-serving systems. They are a starting point, not a complete observability model; business metrics and dependency health may be needed too.
Common misconceptions
- Four metrics are enough for every system — batch jobs, queues, and storage need additional signals
- Average latency is sufficient — the tail often determines user experience and timeout behavior
- High CPU means the service is unhealthy — saturation matters when it affects latency or errors, not in isolation
Anti-patterns
- Alerting on raw CPU without a user-facing symptom
- Mixing successful and failed request latency in one average
- Monitoring traffic but not queue depth or connection pool saturation
Related concepts
Related examples
Authoritative source
Four questions
When a service is in trouble, the first questions are: Is it slow? Is it receiving traffic? Is it failing? Is it running out of something? The golden signals map directly to those questions. A dashboard organized around them helps an on-call engineer orient before diving into traces or logs.
# Latency: p95 by route
histogram_quantile(0.95, sum by (route, le) (rate(http_request_duration_seconds_bucket[5m])))
# Errors: 5xx ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
# Saturation: connection pool in use
db_pool_in_use / db_pool_maxSplit latency and errors
An average that mixes fast failures with slow successes hides both. Track latency separately for successful and failed requests, and use percentiles rather than only the mean. A service with a 50 ms average and a 4-second p99 may feel broken to the users in the tail.
Saturation predicts the cliff
CPU at 90% may be fine for a batch worker and fatal for a latency-sensitive API. Queue depth, thread pool usage, connection pool usage, and memory pressure are often better saturation signals than CPU alone. Alert on the resource that actually limits throughput, and test the behavior at the limit.
When to use it
Use golden signals as the default dashboard and alerting frame for request-serving services. The advantage is a small, actionable set that covers most incidents; the drawback is that it does not explain causes or business impact. Compared to exhaustive metrics, golden signals reduce noise but need traces, logs, and business metrics to complete the picture.