Parquet in Python: reading, writing and optimization

Parquet in Production: How to Read, Write, and Optimize Data for Analytics

Parquet is more than just a storage format—it’s an architectural optimization tool. Its columnar structure, built-in read-time filtering, dictionary encoding, and row group metadata reduce I/O by orders of magnitude and shrink data sizes up to 140× compared to CSV. For data engineers and analysts, this means not only saving disk space but also lowering latency for interactive queries, reducing memory pressure, and achieving predictable performance. This article provides practical guidance on working with Parquet in Python at the middle/senior level—from basic reading via pandas to manual control over schema, partitioning, and compression.

Why Parquet Is Replacing CSV in Modern Data Pipelines

CSV is a text-based format without semantics: each row is processed as a single block, delimiters are parsed on the fly, and types are inferred heuristically. Parquet, on the other hand, is binary, strictly typed, and columnar, with embedded metadata. Key differences include:

• Columnar Storage: Each column is stored separately, allowing you to read only the required fields (e.g., year and indicator_value) while ignoring the rest—without parsing the entire row.
• Row Groups: Data is divided into blocks (default ~128 MB), each containing statistics (min, max, null_count). When filtering year == 2023, the engine skips entire row groups without loading them into memory.
• Dictionary Encoding: For string columns with high repetition (regions, categories), Parquet either automatically or explicitly stores a dictionary of unique values and references to them—so instead of “Moscow Region” × 500,000 records, it uses code 1 × 500,000.
• Type Safety: In CSV, NULL in a numeric field turns the entire column into float64; in Parquet, types are rigidly defined in the schema and validated upon writing.

The difference in practice is enormous: a 576 MB cancer incidence dataset shrinks to 4 MB in Parquet (compression ratio ×144), and the time to read a filtered subset drops from 12–18 seconds to under 300 ms.

Reading: From pandas to Fine-Grained Control with pyarrow

Basic Reading via pandas

For a quick start, pd.read_parquet() is sufficient:

import pandas as pd

df = pd.read_parquet(
    "data_zis_109_v20260126.parquet",
    columns=["object_name", "year", "indicator_value"],
    filters=[("year", "=", 2023)]
)

Importantly, filtering happens at the read stage, not after the entire DataFrame is loaded—this is critical for files larger than 1 GB.

Direct Access via pyarrow: Control Over Schema and Metadata

When precise control is needed, use pyarrow.parquet directly:

import pyarrow.parquet as pq

# Without loading the data—just schema and metadata
schema = pq.read_schema("data_zis_109_v20260126.parquet")
meta = pq.read_metadata("data_zis_109_v20260126.parquet")
print(f"Rows: {meta.num_rows:,}, Columns: {meta.num_columns}")

# Reading with Filtering and Column Selection
table = pq.read_table(
    "data_zis_109_v20260126.parquet",
    columns=["object_name", "object_oktmo", "year", "indicator_value"],
    filters=[("year", "=", 2023)]
)

df = table.to_pandas()  # Convert to pandas if necessary

For extremely large files, stream reading in batches is recommended:

pf = pq.ParquetFile("data_zis_109_v20260126.parquet")
for batch in pf.iter_batches(batch_size=100_000):
    df_batch = batch.to_pandas()
    # Process the batch

Writing: From Simple to Production-Grade Serialization

Challenges of Automatic Serialization

df.to_parquet() works, but hides risks:

• Implicit type determination: int64 with NULL → float64, datetime64 → string.
• Lack of dictionary encoding for highly cardinal string columns.
• Inappropriate row group size (default 64 MB), which reduces filtering efficiency.

Explicit Schema and Controlled Encoding

import pyarrow as pa
import pyarrow.parquet as pq

schema = pa.schema([
    pa.field("object_name", pa.string()),
    pa.field("object_oktmo", pa.dictionary(pa.int32(), pa.string())),
    pa.field("year", pa.int32()),
    pa.field("indicator_value", pa.float64())
])

table = pa.Table.from_pandas(df, schema=schema, preserve_index=False)

pq.write_table(
    table,
    "data_zis_new_version.parquet",
    compression="zstd",
    row_group_size=500_000,
    version="2.6",
    write_statistics=True
)

Why zstd? Compression is 30–50% better than snappy, with comparable decompression speed. It strikes the optimal balance for analytical workloads.

Partitioning and Working with File Sets

If the data is logically split (by year, region), use write_to_dataset():

pq.write_to_dataset(
    table,
    root_path="data_zis_partitioned/",
    partition_cols=["year"]
)

Reading then becomes efficient:

import pyarrow.dataset as ds
dataset = ds.dataset("data_zis_partitioned/", format="parquet", partitioning="hive")

table = dataset.to_table(filter=ds.field("year") == 2023)

Partitioning is justified for datasets over 1 GB and when the partition key has fewer than 1,000 distinct values. Avoid partitioning by high-cardinality fields (e.g., indicator_value)—it creates thousands of small files and hurts performance.

What Matters

• Parquet does not replace a database: it’s ideal for append-only, read-heavy analytics scenarios but doesn’t support transactions, UPDATEs, or indexes.
• Dictionary encoding delivers maximum benefits when cardinality is below 10⁴ and repetition exceeds 30%. For columns with unique values (UUIDs, timestamps), it’s useless.
• Row group size affects filtering accuracy: too small leads to excessive metadata; too large makes skipping unnecessary data inefficient. The optimal range is 100–500 thousand rows.
• Statistics (write_statistics=True) are mandatory for production. Without them, filtering behaves like a full scan.
• Always validate the schema before writing: pa.Table.from_pandas(df, schema=schema) will throw an error if types don’t match—this protects against silent corruption.

— Editorial Team