Merge Multiple Parquet Files Safely

Consolidate many small Parquet files into fewer larger ones to cut metadata overhead and improve scan efficiency.

~30 mindataparquetstorage

Task

Compact many small Parquet files into a smaller number of reasonably sized files.

When to use this

A streaming job produced thousands of tiny Parquet files, and queries pay metadata overhead for each file.

Prerequisites

pyarrow or duckdb
A directory of small Parquet files
Temporary space for the output

Steps

1. Read the small files and rewrite as a consolidated dataset.

import pyarrow.dataset as ds
import pyarrow.parquet as pq
dataset = ds.dataset('small_files/', format='parquet')
pq.write_to_dataset(dataset, root_path='merged.parquet', compression='zstd')

Expected output: The merged dataset has far fewer, larger files with the same rows.

2. Verify row counts match.

python -c "import pyarrow.parquet as pq; print(pq.ParquetDataset('small_files').read().num_rows, pq.ParquetDataset('merged.parquet').read().num_rows)"

Expected output: Both row counts are equal, proving no data was lost.

Troubleshooting

Problem

Schemas differ across files

Solution

Unify the schema first; a merge fails or silently coerces when types conflict.

Problem

Out of memory

Solution

Merge in batches by partition instead of all at once.

Limitations

Rewriting doubles disk usage temporarily.
Does not deduplicate rows.
Schema differences must be resolved first.

What this recipe does

Compact many small Parquet files into a smaller number of reasonably sized files.

A streaming job produced thousands of tiny Parquet files, and queries pay metadata overhead for each file.

Prerequisites

pyarrow or duckdb
A directory of small Parquet files
Temporary space for the output

Procedure

Read the small files and rewrite as a consolidated dataset.

import pyarrow.dataset as ds
import pyarrow.parquet as pq
dataset = ds.dataset('small_files/', format='parquet')
pq.write_to_dataset(dataset, root_path='merged.parquet', compression='zstd')

Expected result: The merged dataset has far fewer, larger files with the same rows.

Verify row counts match.

python -c "import pyarrow.parquet as pq; print(pq.ParquetDataset('small_files').read().num_rows, pq.ParquetDataset('merged.parquet').read().num_rows)"

Expected result: Both row counts are equal, proving no data was lost.

How to confirm it worked

Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows the merged dataset has equal row counts, far fewer files, and identical query results.

Troubleshooting

Schemas differ across files — Unify the schema first; a merge fails or silently coerces when types conflict.

Out of memory — Merge in batches by partition instead of all at once.

Limitations

Rewriting doubles disk usage temporarily.
Does not deduplicate rows.
Schema differences must be resolved first.