指标样本（Exemplars）

概念定义

直方图说 p99 桶变大了；trace 说某个请求花了 900 毫秒。exemplar 就是两者的 join：桶条目携带 trace id，聚合与实例互相引用。

工作机制

测量期间，插桩可选择把样本——值、时间、trace 与 span id——记进桶元数据；存储按桶保持有界。仪表盘在桶上暴露样本链接，点击即在链路 UI 打开对应 trace——前提是该 trace 已被采样保留。

使用场景

面向用户端点的延迟直方图、队列滞后指标、以及一切“尖峰一出现就要找实例”的聚合。收益是诊断速度：从告警到代表性 trace 一次点击。

常见误区

exemplar 不是全量存储：每桶只保留有界的选择样本，聚合仍然精确。保留期错配会让链接腐烂——指标存数周而 trace 只存数天，就是死链。尾采样必须保住有趣的 trace，否则样本指向虚空；每桶样本存储会增大抓取载荷——把启用集中在诊断价值高的指标上。

代码示例

import promClient from 'prom-client';
import express from 'express';

const registry = new promClient.Registry();
promClient.collectDefaultMetrics({ register: registry, prefix: 'svc_' });

const ordersTotal = new promClient.Counter({
  name: 'orders_created_total',
  help: 'Total orders created',
  labelNames: ['tenant', 'channel'],
  registers: [registry],
});

const requestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Request latency distribution',
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5],
  labelNames: ['route', 'method'],
  registers: [registry],
});

async function handleCheckout(req, res) {
  const end = requestDuration.startTimer({ route: '/checkout', method: 'POST' });
  try {
    ordersTotal.inc({ tenant: 'acme', channel: 'web' });
    res.json({ ok: true });
  } finally {
    end();
  }
}

const app = express();
app.post('/checkout', handleCheckout);
app.get('/metrics', async (req, res) => {
  res.set('content-type', registry.contentType);
  res.end(await registry.metrics());
});

app.listen(3000);

use tracing::{debug, error, info, instrument, warn};
use tracing_subscriber::{fmt, layer::SubscriberExt, util::SubscriberInitExt, EnvFilter};

#[instrument(name = "order_flow", skip(payload))]
fn process_order(order_id: u64, payload: &str) -> u64 {
    debug!(bytes = payload.len(), "payload received");
    let total = payload.len() as u64 * order_id % 97;
    info!(total, "order total computed");
    total
}

fn init_tracing() {
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new("info,order_service=debug"));

    let is_prod = std::env::var("APP_ENV")
        .map(|v| v == "production")
        .unwrap_or(false);

    let registry = tracing_subscriber::registry().with(filter);

    if is_prod {
        registry
            .with(fmt::layer().json().flatten_event(true))
            .init();
    } else {
        registry
            .with(fmt::layer().pretty().with_target(false))
            .init();
    }
}

fn main() {
    init_tracing();

    let span = tracing::info_span!("startup", version = env!("CARGO_PKG_VERSION"));
    let _guard = span.enter();

    info!("service starting");
    warn!("cache warmup skipped");

    let total = process_order(42, "line-item;quantity=3");
    error!(total, "downstream settlement slow");

    drop(_guard);
    info!("shutdown complete");
}