以 Exemplar 关联指标与追踪

概念概述

以 Exemplar 关联指标与追踪（Exemplars Linking Metrics to Traces）指的是：Exemplar 把采样的追踪标识附加到指标数据点，使运维能从延迟尖峰直接跳到代表性请求追踪。

从更严格的定义看，An exemplar is a sampled trace identifier or other metadata attached to a metric observation, so an aggregated timeseries can be linked back to concrete requests. 该概念在工程实践中的价值，在于它能帮助团队用统一的语言描述一个具体的质量属性或设计约束，从而在评审与协作中减少歧义。

为什么需要它

It is used to shorten the path from a dashboard alert to a root cause trace without searching logs blindly. 它与 Operational Dashboard Design、Selecting Service Level Indicators、On-Call Rotation Design 等概念经常一起出现，用于解决相近的工程问题。理解它的适用场景，有助于在多种备选方案之间做出有依据的取舍，而不是凭直觉选择技术方案。

常见误解

• exemplars record every request rather than a sample
• they work without a tracing backend

需要避免的反模式

• attaching exemplars only to success paths and missing the slow ones
• sampling so sparsely that no exemplar represents the spike

实践要点

1. Metrics aggregate and lose identity, so an exemplar preserves a few concrete examples, typically a sampled trace identifier, alongside each bucket.
2. The value depends on sampling correlated with the tracing pipeline, so both systems must agree on the trace identifier and configure sampling that captures interesting requests.
3. Exemplars add cardinality and storage cost, so they are enabled selectively on metrics that matter for diagnosis.

参考来源

https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exemplars

代码示例

import promClient from 'prom-client';
import express from 'express';

const registry = new promClient.Registry();
promClient.collectDefaultMetrics({ register: registry, prefix: 'svc_' });

const ordersTotal = new promClient.Counter({
  name: 'orders_created_total',
  help: 'Total orders created',
  labelNames: ['tenant', 'channel'],
  registers: [registry],
});

const requestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Request latency distribution',
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5],
  labelNames: ['route', 'method'],
  registers: [registry],
});

async function handleCheckout(req, res) {
  const end = requestDuration.startTimer({ route: '/checkout', method: 'POST' });
  try {
    ordersTotal.inc({ tenant: 'acme', channel: 'web' });
    res.json({ ok: true });
  } finally {
    end();
  }
}

const app = express();
app.post('/checkout', handleCheckout);
app.get('/metrics', async (req, res) => {
  res.set('content-type', registry.contentType);
  res.end(await registry.metrics());
});

app.listen(3000);

use tracing::{debug, error, info, instrument, warn};
use tracing_subscriber::{fmt, layer::SubscriberExt, util::SubscriberInitExt, EnvFilter};

#[instrument(name = "order_flow", skip(payload))]
fn process_order(order_id: u64, payload: &str) -> u64 {
    debug!(bytes = payload.len(), "payload received");
    let total = payload.len() as u64 * order_id % 97;
    info!(total, "order total computed");
    total
}

fn init_tracing() {
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new("info,order_service=debug"));

    let is_prod = std::env::var("APP_ENV")
        .map(|v| v == "production")
        .unwrap_or(false);

    let registry = tracing_subscriber::registry().with(filter);

    if is_prod {
        registry
            .with(fmt::layer().json().flatten_event(true))
            .init();
    } else {
        registry
            .with(fmt::layer().pretty().with_target(false))
            .init();
    }
}

fn main() {
    init_tracing();

    let span = tracing::info_span!("startup", version = env!("CARGO_PKG_VERSION"));
    let _guard = span.enter();

    info!("service starting");
    warn!("cache warmup skipped");

    let total = process_order(42, "line-item;quantity=3");
    error!(total, "downstream settlement slow");

    drop(_guard);
    info!("shutdown complete");
}