Capture gRPC DEADLINE_EXCEEDED Traces
Instrument a gRPC client to surface deadline budgets and server spans so DEADLINE_EXCEEDED can be traced to the slow dependency.
Task
Add OpenTelemetry spans around gRPC calls and read the deadline budget to locate the slow dependency behind DEADLINE_EXCEEDED.
When to use this
A mobile backend intermittently fails with DEADLINE_EXCEEDED but logs show only the client status code, not which downstream call consumed the budget.
Prerequisites
- A gRPC service reachable from staging
- OpenTelemetry SDK wired into the client
- A trace backend that renders spans
Steps
1. Attach a deadline to every call and record it as a span attribute.
ctx, cancel := context.WithTimeout(ctx, 300*time.Millisecond)
defer cancel()
span.SetAttributes(attribute.Int64("rpc.deadline_ms", 300))
resp, err := client.GetFeed(ctx, req)Expected output: The client fails fast at 300 ms; the trace shows the configured deadline on the call span.
2. Emit a client-side event when the deadline fires so the exact point is captured.
if status.Code(err) == codes.DeadlineExceeded {
span.AddEvent("deadline exceeded", trace.WithAttributes(attribute.Int("elapsed_ms", elapsed)))
}Expected output: The trace timeline marks where the deadline fired relative to downstream spans.
Troubleshooting
Problem
Spans are nested but no child is slow
Solution
Check for retries inside the call; each attempt needs its own span or they collapse into one budget.
Problem
Deadline propagates as cancelled on the server
Solution
Confirm the server honours the incoming deadline and returns DEADLINE_EXCEEDED rather than masking it.
Limitations
- Requires tracing infrastructure; without it only the client status is visible.
- Sampling can hide rare slow calls; use tail-based sampling for deadlines.
- Does not fix slow dependencies, only locates them.
What this recipe does
Add OpenTelemetry spans around gRPC calls and read the deadline budget to locate the slow dependency behind DEADLINE_EXCEEDED.
A mobile backend intermittently fails with DEADLINE_EXCEEDED but logs show only the client status code, not which downstream call consumed the budget.
Prerequisites
- A gRPC service reachable from staging
- OpenTelemetry SDK wired into the client
- A trace backend that renders spans
Procedure
- Attach a deadline to every call and record it as a span attribute.
ctx, cancel := context.WithTimeout(ctx, 300*time.Millisecond)
defer cancel()
span.SetAttributes(attribute.Int64("rpc.deadline_ms", 300))
resp, err := client.GetFeed(ctx, req)Expected result: The client fails fast at 300 ms; the trace shows the configured deadline on the call span.
- Emit a client-side event when the deadline fires so the exact point is captured.
if status.Code(err) == codes.DeadlineExceeded {
span.AddEvent("deadline exceeded", trace.WithAttributes(attribute.Int("elapsed_ms", elapsed)))
}Expected result: The trace timeline marks where the deadline fired relative to downstream spans.
How to confirm it worked
Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows a DEADLINE_EXCEEDED trace names the child span that consumed the budget and includes the configured deadline attribute.
Troubleshooting
Spans are nested but no child is slow — Check for retries inside the call; each attempt needs its own span or they collapse into one budget.
Deadline propagates as cancelled on the server — Confirm the server honours the incoming deadline and returns DEADLINE_EXCEEDED rather than masking it.
Limitations
- Requires tracing infrastructure; without it only the client status is visible.
- Sampling can hide rare slow calls; use tail-based sampling for deadlines.
- Does not fix slow dependencies, only locates them.