Observability Onboarding Workflow
Add the three pillars to a service in order, start with logs that answer who and what, add metrics that answer how much, and finish with traces that answer where.
Objective
Give a service enough observability to debug a production issue without SSH, by adding logs, metrics, and traces in priority order.
Environment
- A service with no structured logging
- A metrics scraper and a log pipeline already in place
- A trace collector reachable from the service
Steps
1. Add structured logs
Replace console.log with a structured logger that emits who, what, and a request id on every line.
Expected state: Every log line has a request id, a user id, and a field for the action.
2. Add RED metrics
Instrument the request path with rate, errors, and duration, and expose them on the metrics endpoint.
Expected state: A dashboard with request rate, error rate, and latency p99 for the service.
3. Add traces
Auto-instrument the HTTP and database calls so a slow request shows the span that took the time.
Expected state: A trace per request with spans for the handler, the database call, and any downstream.
4. Wire a runbook
Write a runbook that starts from each signal and ends at a fix, so the on-call can use the observability.
Expected state: A runbook with a path from alert to log to trace to fix.
Decision points
Which pillar first?
- Logs
- Metrics
Logs first; they answer who and what, which is the first question in any incident.
Trace sampling rate?
- 100 percent in staging
- 1 percent in production
1 percent in production, 100 in staging, so cost is controlled and staging is complete.
Expected outcome
A service with logs, metrics, and traces that let the on-call debug an issue without SSH, plus a runbook that ties them together.
Verification
Confirm a log line has a request id, the dashboard shows RED, a trace has spans, and the runbook path works end to end.
Cleanup
Archive the old console.log calls, tune the log level for production, and sample the traces to control cost.
Logs first
Logs answer who and what, the first question in any incident. Replace console.log with a structured logger that emits a request id, a user id, and the action on every line. A log without a request id cannot be correlated.
log.info({ reqId, userId, action: 'checkout' }, 'checkout started');Then metrics, then traces
Add RED metrics so the dashboard shows rate, errors, and duration. Add traces so a slow request shows the span that took the time. Traces are last because they are the most expensive and the least often needed in a first pass.
Wire a runbook
Observability without a runbook is noise. Write a path from alert to log to trace to fix, so the on-call can follow it under pressure. The runbook is what makes the signals useful.
Verification
A log line has a request id, the dashboard shows RED, a trace has spans, and the runbook path works end to end. Sample traces at 1 percent in production and 100 in staging to control cost.