Mine Access Logs for Status and Latency Anomalies

Turn nginx or app access logs into ranked tables of error rates, slow paths, and suspicious clients using only jq, awk, and sort.

~20 minapiobservabilityshell

Task

Analyze plain access logs with shell tools to rank endpoints by error rate and latency and identify the clients driving anomalies.

When to use this

Metrics show a small error-rate bump after a deploy, but averages hide which endpoint, which status, and which clients are affected.

Prerequisites

JSON-formatted access logs (or a convertible combined format)
jq installed for JSON logs
Log files reachable from your shell

Steps

1. Compute error rate per endpoint.

jq -r 'select(.status >= 500) | .uri' access.log | sort | uniq -c | sort -rn | head -15

Expected output: Endpoints are ranked by 5xx count, and one path usually towers over the rest.

2. Join counts with traffic to get a rate, not a raw number.

jq -r '(.status|tostring) + " " + .uri' access.log | sort | uniq -c | awk '{rate[$2]+=$1} END {for (u in rate) print rate[u], u}' | sort -rn | head

Expected output: Total requests per endpoint next to its error count separates a busy healthy path from a quiet broken one.

3. Find the slowest paths by request time.

jq -r '(.request_time|tostring) + " " + .uri' access.log | sort -rn | head -10

Expected output: The top latency entries show the exact slow URI shapes, including query strings that reveal the expensive parameters.

4. Rank clients by request volume to spot abuse.

jq -r '.client' access.log | sort | uniq -c | sort -rn | head -10

Expected output: A single address sending tens of thousands of requests per hour is either a scraper or a stuck retry loop.

5. Check for retry storms around the anomaly window.

jq -r 'select(.status >= 500) | .client' access.log | sort | uniq -c | sort -rn | head -5

Expected output: The same client appearing in most 5xx lines indicates a hot retry loop amplifying a small outage.

Troubleshooting

Problem

Logs are combined format, not JSON

Solution

Either switch nginx to a JSON log_format or parse with awk field positions; jq only pays off after JSON output exists.

Problem

URIs with ids make every row unique

Solution

Normalize the path before counting, for example strip numeric segments with sed so /orders/123 becomes /orders/ID.

Problem

request_time missing

Solution

The variable was not included in the log format; add $request_time and $upstream_response_time to the format and wait for fresh traffic.

Limitations

Access logs sample successful parsing only; requests rejected before logging are invisible.
Latency fields measure server-side time and exclude client network and rendering.

The log file is already a metrics store

You do not need a metrics platform to answer which endpoint broke and for whom. JSON access logs plus three shell tools produce ranked tables in seconds: error counts per path, slowest requests, and the noisiest clients. This is often enough to route an incident to the right owner before dashboards finish loading.

# Endpoints ranked by 5xx count
jq -r 'select(.status >= 500) | .uri' access.log \
  | sort | uniq -c | sort -rn | head -15

# Slowest requests with their query strings
jq -r '(.request_time|tostring) + " " + .uri' access.log \
  | sort -rn | head -10

Rates beat raw counts

A path with two hundred errors sounds broken until you see it serves two million requests an hour. Always join error counts with total traffic per endpoint before escalating, and normalize ids out of the URI so per-resource paths aggregate into one rankable line.

Look for the amplifier

During small outages the interesting line is a single client appearing in most of the 5xx entries: a retry loop without backoff turns one failure into hundreds of requests. Finding that client changes the incident from scale-up to a one-line client fix or a rate limit.