Third-Party Dependency Outage Workflow

Prove the outage is external, degrade dependent features gracefully, and keep a decision clock on whether to wait or to invoke the backup path.

intermediate~25 minincidentdependenciesresilience

Objective

Handle an external dependency outage with evidence of externality, graceful degradation of dependent features, and a time-boxed escalation decision.

Environment

A critical third-party dependency with its own status page
Feature flags gating dependent features
A documented fallback or backup provider if one exists

Steps

1. Prove it is them, not us

Test the dependency from multiple networks and check its status page, while confirming your own egress and integration are healthy.

Expected state: Externality is evidenced, preventing a pointless rollback of your own stack.

2. Map your dependent features

List which user-facing features degrade when this dependency is down, and which have fallbacks, from the dependency map maintained in advance.

Expected state: A concrete list of impacted features with their fallback status exists.

3. Degrade deliberately

Flag off the dependent features or switch them to fallback mode, with user-facing messaging that explains the temporary unavailability.

Expected state: Users see honest degraded states instead of errors from a feature that cannot work.

4. Start the decision clock

Set a time box (based on business tolerance) at which waiting ends and the backup path or manual process starts, with the decision owner named.

Expected state: A visible countdown to the invoke-backup decision exists with an owner.

5. Recover and review dependency posture

When the provider recovers, ramp features back, verify queued interactions, and file the posture review: SLO, fallback readiness, and exit strategy.

Expected state: Full function returns, and a dependency posture review is filed with actions.

Expected outcome

External outages handled without self-inflicted rollbacks, with graceful degradation and a pre-decided escalation point instead of an open-ended wait.

Verification

The incident record shows the externality evidence, the degraded feature list with messaging, and the decision-clock timeline; the posture review produced concrete actions.

Their outage should not become your outage

A third-party failure tempts two mistakes: rolling back your own healthy stack, and staring at their status page indefinitely. Proving externality first, degrading deliberately second, and running a decision clock third keeps the incident bounded.

INC-3010  maps provider down  14:31Z
externality proof:
  provider status page: incident (14:20Z)
  our egress: OK (other https fine)
  sandbox direct curl: timeout from 3 networks
impacted features:
  map widget        -> fallback: static map image
  address autofill  -> flag OFF, plain input
  distance pricing  -> fallback: flat rate (bounded)
decision clock:
  tolerance: 90 min business
  16:01Z if still down -> manual pricing table
  provider recovers 15:44Z -> ramp back
posture review filed: single-provider, no SLA credit

Externality proof prevents friendly fire

The most common self-inflicted wound during dependency outages is rolling back your own last deploy, because the timing coincided. Three networks plus the provider status page plus your own egress health takes five minutes and prevents hours of misdirected work.

The decision clock ends the staring contest

Waiting for a third party is sometimes right, but only as a bounded decision: at ninety minutes, the backup path or manual process starts. Without the clock, teams watch status pages for hours while the business impact compounds silently.