Propagate Request Deadlines Downstream

Pass an absolute deadline to every downstream call, budget the time between hops, and cancel work when the deadline passes instead of waiting indefinitely.

~35 minreliabilitytimeoutsdistributed-systems

Task

Add deadline propagation so a slow dependency cannot hold a request open past its budget.

When to use this

A request hangs because one downstream call has no timeout; the caller gives up but the server keeps working on an abandoned request.

Prerequisites

A request context object
Clients that accept per-call timeouts

Steps

1. Compute an absolute deadline at the edge from the overall timeout.

Expected output: Every handler knows when it must finish.

2. Derive each downstream timeout from the remaining budget, not a fixed value.

Expected output: Later hops automatically get less time.

3. Cancel in-flight work when the deadline passes and propagate the cancellation.

curl -m 2 https://api.example.com/slow-endpoint

Expected output: Abandoned work stops instead of consuming resources.

4. Return 504 with the deadline-exceeded message when the budget runs out.

Expected output: Callers get a consistent timeout error.

Troubleshooting

Problem

Deadline already passed on entry

Solution

Check the deadline at the start of the handler and short-circuit before doing expensive work.

Problem

Clock skew between hosts

Solution

Propagate an absolute deadline from a single source and compare using that host clock, not each hop remaining time alone.

Limitations

Absolute deadlines require roughly synchronized clocks; otherwise use hop-by-hop timeouts.
Deadlines are advisory unless every layer honors cancellation.

Context

Add deadline propagation so a slow dependency cannot hold a request open past its budget. A request hangs because one downstream call has no timeout; the caller gives up but the server keeps working on an abandoned request.

Steps

Compute an absolute deadline at the edge from the overall timeout. Expected: Every handler knows when it must finish.
Derive each downstream timeout from the remaining budget, not a fixed value. Expected: Later hops automatically get less time.
Cancel in-flight work when the deadline passes and propagate the cancellation. Expected: Abandoned work stops instead of consuming resources.
Return 504 with the deadline-exceeded message when the budget runs out. Expected: Callers get a consistent timeout error.

const deadline = ctx.deadline ?? Date.now() + budget;
const remaining = deadline - Date.now();
if (remaining <= 0) throw new DeadlineExceeded();
await fetch(url, { signal: AbortSignal.timeout(remaining) });

Verify

Every handler knows when it must finish.
Later hops automatically get less time.
Abandoned work stops instead of consuming resources.
Callers get a consistent timeout error.

Limits

Absolute deadlines require roughly synchronized clocks; otherwise use hop-by-hop timeouts.
Deadlines are advisory unless every layer honors cancellation.