← Workflows

阅读中文版本

Dependency Isolation Workflow

Isolate a failing dependency so its outage degrades only the features that need it instead of cascading across the system.

advanced~65 minincidentresiliencereliability

Objective

Contain a dependency failure so impact stays limited to dependent features.

Environment

  • Circuit breaker or bulkhead mechanisms
  • Feature to dependency mapping
  • Fallback behaviour definitions

Steps

1. Identify the blast radius

Determine which features depend on the failing dependency and how.

Expected state: The dependency usage map is known.

2. Isolate with a bulkhead or breaker

Limit concurrent calls and trip a breaker so failures do not exhaust shared resources.

Expected state: The dependency cannot consume the whole thread or connection pool.

3. Provide fallbacks where possible

Serve cached or degraded data for features that can tolerate it.

Expected state: Some features keep working during the outage.

4. Verify containment

Simulate the dependency failure and confirm only dependent features degrade.

Expected state: Unrelated features stay healthy.

Decision points

Fail fast or serve degraded data?

  • Degrade where safe, fail fast otherwise
  • Always fail fast

Serve degraded data where correctness allows; fail fast elsewhere.

Global or per-dependency limits?

  • Per-dependency bulkheads
  • One shared limit

Use per-dependency bulkheads so one slow dependency cannot starve others.

Expected outcome

A dependency outage degrades only the features that rely on it, protecting the rest of the system.

Verification

Confirm that simulating the dependency failure leaves unrelated features healthy and dependent ones degrade gracefully.

Cleanup

Document fallback behaviour and revisit bulkhead limits after the incident.

Overview

Contain a dependency failure so impact stays limited to dependent features.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

  • Circuit breaker or bulkhead mechanisms
  • Feature to dependency mapping
  • Fallback behaviour definitions

Procedure

1. Identify the blast radius

Determine which features depend on the failing dependency and how.

Expected state: The dependency usage map is known.

2. Isolate with a bulkhead or breaker

Limit concurrent calls and trip a breaker so failures do not exhaust shared resources.

Expected state: The dependency cannot consume the whole thread or connection pool.

3. Provide fallbacks where possible

Serve cached or degraded data for features that can tolerate it.

Expected state: Some features keep working during the outage.

4. Verify containment

Simulate the dependency failure and confirm only dependent features degrade.

Expected state: Unrelated features stay healthy.

if breaker.isOpen('payments') { return fallback() }

Verification

Confirm that simulating the dependency failure leaves unrelated features healthy and dependent ones degrade gracefully.

Decision points

Fail fast or serve degraded data?

  • Degrade where safe, fail fast otherwise
  • Always fail fast

Recommendation: Serve degraded data where correctness allows; fail fast elsewhere.

Global or per-dependency limits?

  • Per-dependency bulkheads
  • One shared limit

Recommendation: Use per-dependency bulkheads so one slow dependency cannot starve others.

Common failures

  • Shared pools let one dependency starve the whole service.
  • No fallback turns a partial outage into a total one.
  • Unknown dependency maps slow isolation decisions.