Expose Application Metrics for Prometheus Scraping

1. Install prom-client in the service.

pnpm add prom-client

Expected output: prom-client is added to package.json and the lockfile.

2. Create one registry, collect default Node.js metrics, and define a request-duration histogram with bounded labels.

import client from 'prom-client';

export const register = new client.Registry();
client.collectDefaultMetrics({ register, prefix: 'nodejs_' });

export const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
  registers: [register],
});

Expected output: The registry exposes nodejs_ default metrics plus http_request_duration_seconds_bucket, _sum, and _count.

3. Instrument requests with the matched route template rather than the raw URL, and serve /metrics only to a bearer token.

import express from 'express';
import { httpDuration, register } from './metrics.js';

const METRICS_TOKEN = process.env.METRICS_TOKEN;
if (!METRICS_TOKEN) throw new Error('METRICS_TOKEN is required');

const app = express();

app.use((req, res, next) => {
  if (req.path === '/metrics') {
    next();
    return;
  }

  const end = httpDuration.startTimer();
  res.on('finish', () => {
    const route = req.route ? `${req.baseUrl}${req.route.path}` : 'unmatched';
    end({
      method: req.method,
      route,
      status: `${Math.floor(res.statusCode / 100)}xx`,
    });
  });
  next();
});

app.get('/healthz', (_req, res) => {
  res.json({ ok: true });
});

app.get('/metrics', async (req, res) => {
  if (req.get('authorization') !== `Bearer ${METRICS_TOKEN}`) {
    res.status(403).end();
    return;
  }
  res.set('content-type', register.contentType);
  res.end(await register.metrics());
});

app.listen(3000, () => console.log('listening on 3000'));

Expected output: GET /healthz records a 2xx histogram observation with route /healthz; GET /metrics without the bearer token returns 403.

4. Add a Prometheus scrape job that reads the bearer token from a file and targets the service DNS name.

scrape_configs:
  - job_name: orders-api
    scrape_interval: 15s
    metrics_path: /metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/orders-api-token
    static_configs:
      - targets: ["orders-api.default.svc.cluster.local:3000"]

Expected output: The Prometheus configuration contains one orders-api job with a 15-second interval, a bearer credential file, and the service target.

5. Validate and reload Prometheus, then confirm the target health is up with no scrape error.

promtool check config /etc/prometheus/prometheus.yml
curl -s -X POST http://localhost:9090/-/reload
curl -s http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | select(.labels.job == "orders-api") | [.health, .lastError] | @tsv'

Expected output: promtool reports SUCCESS, the reload returns 200, and the target row is up followed by an empty lastError.

6. Generate traffic, wait for a scrape, and query the p95 request duration from Prometheus.

for i in $(seq 1 20); do curl -s -o /dev/null http://localhost:3000/healthz; done
sleep 20
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))' \
  | jq -r '.data.result[0].value[1]'

Expected output: The query returns a numeric p95 such as 0.012 seconds; if the result is NaN, wait for another scrape interval and generate more requests.

7. Verify the endpoint rejects unauthenticated scrapes and returns the expected metric families when authorized.

curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3000/metrics
curl -s -H "authorization: Bearer $METRICS_TOKEN" http://localhost:3000/metrics \
  | grep '^http_request_duration_seconds_count'

Expected output: The first command prints 403 and the second prints http_request_duration_seconds_count lines with method, route, and status labels.

Problem

The Prometheus target is down

Solution

Read lastError, check the network policy and DNS name, and confirm the token file exists and matches METRICS_TOKEN. A 403 from the endpoint appears as a scrape failure.

Problem

Prometheus reports duplicate metric registration

Solution

Two modules create a Registry or the app clears the module cache. Export one registry instance and register metrics exactly once per process.

Problem

histogram_quantile returns NaN

Solution

There are no observations in the rate window yet. Generate traffic, wait at least two scrape intervals, and query again.

Problem

Cardinality grows without bound

Solution

The route label is using the raw URL or user input. Use the Express route template, and never put IDs, emails, or query strings in labels.