Validate JSONL with Expectation Suites

Define expectation suites over a JSONL dataset to catch missing fields and range violations before loading a warehouse.

~35 mindataqualityvalidation

Task

Add data-quality expectations that assert schema and value ranges on a JSONL file.

When to use this

Bad records reach the warehouse because nothing validates ingestion files before loading.

Prerequisites

Great Expectations or an equivalent framework
A JSONL dataset
A target warehouse or script runner

Steps

1. Define expectations for required columns and ranges.

import great_expectations as gx
ctx = gx.get_context()
validator = ctx.sources.pandas_default.read_json('events.jsonl')
validator.expect_column_values_to_not_be_null('user_id')
validator.expect_column_values_to_be_between('amount', min_value=0)

Expected output: Each expectation produces a pass/fail result per column.

2. Fail the load when expectations do not pass.

python validate.py && load.py || echo 'quality gate failed, skipping load'

Expected output: The loader only runs when expectations pass, blocking bad data.

Troubleshooting

Problem

Expectations too strict

Solution

Start with critical checks and add more as data stabilises.

Problem

Validation slow on huge files

Solution

Validate a sample or push critical checks into the warehouse load step.

Limitations

Expectations check the checked columns only.
Not a substitute for schema enforcement in the warehouse.
Adds a processing stage to the pipeline.

What this recipe does

Add data-quality expectations that assert schema and value ranges on a JSONL file.

Bad records reach the warehouse because nothing validates ingestion files before loading.

Prerequisites

Great Expectations or an equivalent framework
A JSONL dataset
A target warehouse or script runner

Procedure

Define expectations for required columns and ranges.

import great_expectations as gx
ctx = gx.get_context()
validator = ctx.sources.pandas_default.read_json('events.jsonl')
validator.expect_column_values_to_not_be_null('user_id')
validator.expect_column_values_to_be_between('amount', min_value=0)

Expected result: Each expectation produces a pass/fail result per column.

Fail the load when expectations do not pass.

python validate.py && load.py || echo 'quality gate failed, skipping load'

Expected result: The loader only runs when expectations pass, blocking bad data.

How to confirm it worked

Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows an intentionally corrupt record fails the gate and the load is skipped as designed.

Troubleshooting

Expectations too strict — Start with critical checks and add more as data stabilises.

Validation slow on huge files — Validate a sample or push critical checks into the warehouse load step.

Limitations

Expectations check the checked columns only.
Not a substitute for schema enforcement in the warehouse.
Adds a processing stage to the pipeline.