Diff a Live API Response Against Its OpenAPI Spec

Capture a real response, validate every field against the published OpenAPI schema, and list exact mismatches before clients discover them by breaking.

~30 minapiopenapicontract-testing

Task

Validate a live endpoint response against the OpenAPI document and enumerate every field-level mismatch with evidence.

When to use this

The spec says status is an enum of two values, but production quietly returns a third one added last quarter. Mobile clients built strictly from the spec now crash at runtime.

Prerequisites

curl and jq installed
The published OpenAPI document (openapi.json) accessible locally or by URL
A staging or production endpoint safe to query

Steps

1. Download the spec and resolve the schema for the target response.

curl -s https://staging.example.com/openapi.json -o openapi.json && jq -r '.paths["/api/checkout"].get.responses["200"].content["application/json"].schema.$ref' openapi.json

Expected output: The command prints a $ref such as #/components/schemas/CheckoutResponse, identifying the schema to validate against.

2. Capture a real response from the live endpoint.

curl -s "https://staging.example.com/api/checkout?orderId=demo" -H "accept: application/json" -o live.json && jq -r '.status' live.json

Expected output: The response saves to live.json; printing one field such as status shows pending, which the enum may or may not allow.

3. Compare observed fields against the schema definition.

jq -r '.components.schemas.CheckoutResponse.properties | keys[]' openapi.json | sort > /tmp/spec-keys; jq -r 'keys[]' live.json | sort > /tmp/live-keys; diff /tmp/spec-keys /tmp/live-keys

Expected output: The diff lists fields present in only one side, for example coupon applied in the live response but absent from the spec.

4. Validate enum values and types for the fields that exist on both sides.

ENUM=$(jq -r '.components.schemas.CheckoutResponse.properties.status.enum // [] | join(",")' openapi.json); LIVE=$(jq -r '.status' live.json); echo "spec enum: [$ENUM] live: $LIVE"; jq -r 'if (.items|type) == "array" then (.items|type) else (.status|type) end' live.json

Expected output: The enum set prints next to the live value, making any drift such as a third status visible in one line.

5. Turn the manual check into a repeatable script.

#!/usr/bin/env bash
# spec-diff.sh <endpoint> <schema-ref>
set -euo pipefail
ref="${2//#\/components\/schemas\/}"
spec_keys=$(jq -r --arg r "$ref" '.components.schemas[$r].properties | keys[]' openapi.json | sort)
live_keys=$(jq -r 'keys[]' live.json | sort)
comm -3 <(echo "$spec_keys") <(echo "$live_keys") || true

Expected output: Running the script in CI prints only genuine mismatches, so spec drift becomes a build failure instead of a support ticket.

Troubleshooting

Problem

The $ref points to a nested allOf schema

Solution

Resolve the chain first with jq recursion, or use a dedicated validator like schemathesis once the manual diff proves the spec file itself is sound.

Problem

Live response fields vary by feature flag

Solution

Query the endpoint with flags both on and off, and record both variants in the spec using oneOf or a documented nullable field.

Problem

Spec uses 3.0 nullable syntax, validator expects 3.1

Solution

Convert nullable: true into type: ["string", "null"] with a one-off jq transform, or pin the validator version to match the spec version.

Limitations

A single response cannot prove the spec correct; only that this response conforms, so sample several states including error responses.
Query parameters and headers are not covered by a body diff; validate them separately from the paths section.

The spec is a contract only if something enforces it

An OpenAPI document that nobody validates against production is documentation, not a contract. A five-minute curl-plus-jq diff converts it back into a contract by comparing real responses with declared schemas and printing every mismatch as evidence.

# Extract declared enum for a field
jq -r '.components.schemas.Order.properties.status.enum' openapi.json
# → ["pending", "shipped"]

# Read the value production actually returns
jq -r '.status' live.json
# → "partially_shipped"   <-- spec drift, mobile app crashes

Drift accumulates silently

Every bypassed code review, every hotfix shipped at 2 a.m., adds a field or value that the spec never learned about. Diffing live traffic quarterly, or per release in CI, keeps the document honest without demanding a full contract-testing framework on day one.

Escalate to generated tests

Once the manual diff is part of the routine, promote it to schemathesis or dredd, which fuzz endpoints against the whole schema. The manual technique remains valuable as the triage step that explains exactly which field diverged and how.