Conformance Testing
Conformance testing checks an implementation against a written standard, so that compliance with the spec is a pass/fail property rather than a matter of interpretive reading by each implementer.
Definition
Conformance testing is the practice of running a standardized test suite — authored by the specification maintainers — against an implementation to verify it satisfies the mandatory requirements of the standard, producing a result that consumers can use to decide interoperability.
Context and scope
Applied in language standards, network protocols, file formats, and API specifications where multiple implementations must interoperate and the spec is the arbiter.
Common misconceptions
- Passing the conformance suite means full compliance — the suite covers mandatory behavior; optional features and edge cases may be untested
- Conformance is a one-time certification — implementations drift as the standard amends and the implementation evolves; conformance is a continuous property
Anti-patterns
- Treating the conformance suite as the spec, so passing the tests but violating an untested requirement is considered compliant
- Running only the conformance tests and skipping interoperability tests with other implementations
Related concepts
Authoritative source
What it is
Conformance testing is the practice of running a standardized test suite — authored by the specification maintainers — against an implementation to verify it satisfies the mandatory requirements of the standard, producing a result that consumers can use to decide interoperability. Applied in language standards, network protocols, file formats, and API specifications where multiple implementations must interoperate and the spec is the arbiter. The distinguishing question is what counts as evidence: a test that passes by accident is worse than no test, because it manufactures false confidence that propagates into release decisions.
How it works
Sound tests exercise a specific behavior contract, isolate the unit under test from environmental noise, and fail with a diagnostic that points at the cause rather than the symptom. Determinism is engineered, not assumed: clocks are fixed, randomness is seeded, and external state is replaced with doubles whose interactions are asserted, not merely stubbed. Coverage measures which behaviors were exercised, not which lines were visited.
When to apply and pitfalls
Apply where the cost of an undetected regression exceeds the cost of writing and maintaining the test. The recurring pitfall is testing the implementation rather than the behavior — such tests break on every refactor and teach nothing. Equally damaging are tests that assert only the happy path, leaving the error branches — where most production bugs live — unexercised.