中文
devexamples
← Recipes

阅读中文版本

Propagate gRPC Deadlines in a Service Chain

Set a deadline on every gRPC call, forward the remaining time to downstream calls, and return DEADLINE_EXCEEDED instead of waiting forever.

~35 mingrpcreliabilitytimeouts

Task

Add deadline handling to gRPC clients and servers so slow chains fail fast and free resources.

When to use this

A gRPC call without a deadline pins a server thread and never returns, cascading into connection exhaustion when one dependency slows down.

Prerequisites

  • Generated gRPC stubs
  • A context or metadata carrier for the deadline

Steps

1. Set an explicit deadline on every outbound call from configuration.

Expected output: No call can run unbounded.

2. Read the inbound deadline and derive each downstream deadline from the remaining time.

Expected output: Budgets shrink as the chain deepens.

3. Return DEADLINE_EXCEEDED promptly when the context is done.

grpcurl -max-time 1 localhost:50051 svc.Health/Check

Expected output: Callers receive a clear status rather than a hang.

4. Cancel in-flight work with the context when the deadline passes.

Expected output: Canceled calls stop consuming resources.

Troubleshooting

Problem

Deadline too tight causes spurious failures

Solution

Set budgets from measured p99 latency with headroom, and tune per endpoint.

Problem

Deadline ignored by blocking work

Solution

Pass the context into every blocking operation and check for cancellation.

Limitations

  • A deadline is not a cancellation for non-cooperative code; blocking calls that ignore context will still hang.
  • Aggressive deadlines can increase error rates if they are not derived from real latency data.

Context

Add deadline handling to gRPC clients and servers so slow chains fail fast and free resources. A gRPC call without a deadline pins a server thread and never returns, cascading into connection exhaustion when one dependency slows down.

Steps

  1. Set an explicit deadline on every outbound call from configuration. Expected: No call can run unbounded.
  2. Read the inbound deadline and derive each downstream deadline from the remaining time. Expected: Budgets shrink as the chain deepens.
  3. Return DEADLINE_EXCEEDED promptly when the context is done. Expected: Callers receive a clear status rather than a hang.
  4. Cancel in-flight work with the context when the deadline passes. Expected: Canceled calls stop consuming resources.
ctx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
defer cancel()
resp, err := client.Get(ctx, req)
if status.Code(err) == codes.DeadlineExceeded {
	return nil, err
}

Verify

  • No call can run unbounded.
  • Budgets shrink as the chain deepens.
  • Callers receive a clear status rather than a hang.
  • Canceled calls stop consuming resources.

Limits

  • A deadline is not a cancellation for non-cooperative code; blocking calls that ignore context will still hang.
  • Aggressive deadlines can increase error rates if they are not derived from real latency data.