中文
devexamples
← Workflows

阅读中文版本

Metric Cardinality Audit

Tune observability: audit metric cardinality, configure trace sampling, and confirm logs are structured and retained within budget.

advanced~35 minobservabilitymonitoringcost

Objective

Keep observability signal high while controlling cost and cardinality within the configured budget.

Environment

  • A working observability environment with access to the target system and its tooling
  • Configuration and credentials for the observability workflow installed and verified
  • A rollback or undo path documented before any change is applied

Steps

1. Audit metric cardinality

Inspect the top series by label and identify high-cardinality labels that should be removed or bucketed.

Expected state: Series count drops after removing or bucketing the offending labels.

2. Configure trace sampling

Set tail sampling to keep all error traces and a representative sample of healthy traces within the volume budget.

Expected state: Error traces are retained and healthy-trace volume stays within budget.

3. Verify structured logs

Confirm every service emits JSON logs with a correlation id and the retention policy matches the compliance window.

Expected state: Logs are structured, correlated, and retained for the required window.

Decision points

What sampling rate should traces use?

  • Tail sampling that keeps all errors
  • A fixed 1 percent head sample

Use tail sampling to keep all errors and a representative healthy sample; fixed head sampling drops rare error traces.

Expected outcome

Keep observability signal high while controlling cost and cardinality within the configured budget. The change is verified and a rollback path is confirmed.

Verification

Keep observability signal high while controlling cost and cardinality within the configured budget. Compare before and after metrics and confirm the expected state is reached.

Cleanup

Archive the run artifacts, remove temporary credentials, and update the runbook with any changes.

Overview

Tune observability: audit metric cardinality, configure trace sampling, and confirm logs are structured and retained within budget. The goal is to keep observability signal high while controlling cost and cardinality within the configured budget.

promtool check rules rules/*.yaml

Steps

  1. Audit metric cardinality. Inspect the top series by label and identify high-cardinality labels that should be removed or bucketed. Expected: Series count drops after removing or bucketing the offending labels.

  2. Configure trace sampling. Set tail sampling to keep all error traces and a representative sample of healthy traces within the volume budget. Expected: Error traces are retained and healthy-trace volume stays within budget.

  3. Verify structured logs. Confirm every service emits JSON logs with a correlation id and the retention policy matches the compliance window. Expected: Logs are structured, correlated, and retained for the required window.

Verification

Confirm the outcome by checking: keep observability signal high while controlling cost and cardinality within the configured budget. Run the verification command and compare the result against the expected state. If any check fails, stop and investigate before continuing.

Decision points

  • What sampling rate should traces use?
    • Tail sampling that keeps all errors
    • A fixed 1 percent head sample
    • Recommendation: Use tail sampling to keep all errors and a representative healthy sample; fixed head sampling drops rare error traces.