Add an OpenAPI Contract Test

Lint the OpenAPI document in CI, exercise every endpoint with Schemathesis against a test server, and fail on responses that violate the contract.

~35 minapitestingopenapi

Task

Detect breaking API changes and undocumented responses before a deploy, using the OpenAPI document as an executable contract.

When to use this

A field is renamed or a status code changes in code, but the OpenAPI file and the clients still describe the old response. Consumers break only after deployment.

Prerequisites

An OpenAPI 3.x document kept in the repository
A test server with seeded data and authentication for protected routes
Python 3.10+ or a container image for Schemathesis
CI permission to run the test suite and publish reports

Steps

1. Lint the OpenAPI document for structural errors and unresolved references.

npx @redocly/cli lint openapi.yaml

Expected output: The document passes lint with no errors and a documented ruleset.

2. Run property-based tests against a disposable server so generated requests exercise the documented schema.

pip install schemathesis && schemathesis run openapi.yaml --url http://localhost:3000 --checks all

Expected output: Every documented operation returns responses that match the schema, or the failing operation is reported.

3. Add the lint and contract run to CI before deployment, using a seeded test database and a fixed random seed for reproducibility.

schemathesis run openapi.yaml \
  --url http://localhost:3000 \
  --checks all \
  --seed 12345 \
  --hypothesis-max-examples 50

Expected output: A pull request that changes a response shape fails CI before merge.

4. Review failures as either an implementation bug or an outdated contract, and update the spec and clients together.

git diff -- openapi.yaml

Expected output: The spec change and the implementation change are reviewed in the same pull request.

Troubleshooting

Problem

Schemathesis generates invalid authentication

Solution

Configure the security scheme and a test token, or exclude protected operations from generation and test them separately.

Problem

The test finds unrelated schema drift

Solution

Fix the schema or mark the endpoint as undocumented only with an owner and a deadline; do not disable the check globally.

Limitations

Contract tests validate shape and status, not business correctness or authorization.
Generated tests can be slow; cap examples and run the full suite nightly.

The spec must be executable

An OpenAPI file that is generated once and never tested is documentation, not a contract. Linting catches unresolved references and malformed schemas. Property-based testing goes further: it generates requests from the spec, sends them to a real server, and checks that the responses match the documented status codes and shapes. A mismatch is either an implementation bug or a stale contract; both need attention.

npx @redocly/cli lint openapi.yaml
schemathesis run openapi.yaml --url http://localhost:3000 --checks all

Run against a disposable server

Use a seeded test database and a server built from the same commit as the spec. Running against production risks destructive generated requests and noisy data. Configure authentication for protected routes and exclude operations that require a third-party callback. Set a random seed so a failure can be reproduced, and cap the number of generated examples so the suite stays fast.

Treat drift as a review event

When the contract test fails, compare the implementation and the spec. If the implementation is correct, update the spec and regenerate clients. If the spec is correct, fix the implementation. Do not update the spec automatically from the implementation without review; that turns the contract into a description of whatever the server currently does, including its bugs.

Cover what generation cannot

Contract tests do not verify business rules, authorization boundaries, or idempotency. Keep targeted integration tests for those. Run the contract suite on every pull request and a larger randomized run nightly. Publish the failing request and response in the report so a reviewer can reproduce it without rerunning the whole suite.