阅读中文版本

Write Partitioned Parquet Datasets

Write Parquet partitioned by a low-cardinality column so downstream queries prune partitions and scan far less data.

import pandas as pd
df = pd.read_csv('events.csv')
df['event_date'] = pd.to_datetime(df['event_date'])
df.to_parquet('events.parquet', partition_cols=['event_date'], compression='zstd')

find events.parquet -name '*.parquet' | head
du -sh events.parquet

What this recipe does

Write a dataset partitioned on a date column for efficient downstream pruning.

Analytical queries scan the entire dataset because everything sits in one large file.

Prerequisites

• Python with pyarrow or pandas
• A dataset with a date or category column
• Awareness of target partition size

Procedure

1. Write partitioned Parquet with a sensible row-group size.

import pandas as pd
df = pd.read_csv('events.csv')
df['event_date'] = pd.to_datetime(df['event_date'])
df.to_parquet('events.parquet', partition_cols=['event_date'], compression='zstd')

Expected result: Files are organised into event_date=YYYY-MM-DD directories for pruning.

2. Confirm the partition layout and sizes.

find events.parquet -name '*.parquet' | head
du -sh events.parquet

Expected result: Each partition has a reasonable file size, avoiding tiny-file overhead.

How to confirm it worked

Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows a query filtered on the partition column touches only matching directories and scans much less data.

Troubleshooting

Too many tiny partitions — Partition on a coarser granularity such as month when daily partitions are too small.

High-cardinality partition key — Avoid partitioning on unique ids; it creates millions of tiny files.

Limitations

• Over-partitioning causes small-file overhead.
• Does not optimise sort order within partitions.
• Partition pruning depends on query filters.