What is Apache Parquet? The Complete Guide to Columnar Storage

Introduction: The Paradigm Shift to Columnar Storage

In traditional relational databases (OLTP), data is typically arranged in a row-oriented format. When a database records a new customer transaction, all fields for that specific row (such as user_id, timestamp, item_name, and price) are written contiguously on physical disk sectors. While this is optimal for single-record inserts and point lookups, it creates massive I/O bottlenecks during analytical workloads (OLAP).

Analytical queries rarely need all columns. If you run a query like "SELECT AVG(price) FROM orders WHERE date >= 2026-01-01", a row-oriented format forces the disk to scan every single byte of every irrelevant column (descriptions, shipping addresses, customer names) just to extract the price and date fields.

Apache Parquet solves this by pivoting data 90 degrees into a columnar storage model. In Parquet, all values for column A are stored together, followed by all values for column B. This architectural difference allows query engines to skip unreferenced columns entirely (a technique known as column pruning), reducing disk I/O by 80% to 95% on typical analytical datasets.

Internal Architecture: Row Groups, Column Chunks, and Pages

A single Apache Parquet file is organized hierarchically into three distinct layers: Row Groups, Column Chunks, and Pages.

1. Row Groups: A logical horizontal partition of data containing a fixed number of rows (typically between 128 MB and 512 MB). Dividing a file into multiple Row Groups enables parallel processing by distributed computing engines like Spark, Trino, and DuckDB.

2. Column Chunks: Within each Row Group, the data for a specific column is stored as an isolated block of bytes known as a Column Chunk. Each Column Chunk contains rich statistical metadata including minimum values, maximum values, and null counts.

3. Pages: Column Chunks are further subdivided into Pages (typically 1 MB in size). A page is the smallest indivisible unit of compression and encoding in Parquet. Pages can be Data Pages (containing row values) or Dictionary Pages (containing frequency lookup tables).

-- DuckDB can inspect Parquet internal layout directly:
SELECT 
  row_group_id, 
  column_id, 
  total_uncompressed_size, 
  total_compressed_size, 
  encodings
FROM parquet_metadata('sales_data.parquet')
LIMIT 5;

Advanced Encodings & Compression Algorithms

Because identical data types are clustered together in columnar storage, Parquet achieves unprecedented compression ratios through domain-specific encoding techniques applied prior to standard byte compression:

• Dictionary Encoding: If a column contains repeated categorical strings (e.g., country codes like "US", "DE", "JP"), Parquet creates a small integer dictionary lookup table and replaces string values with 1-byte integer IDs.

• Run-Length Encoding (RLE) and Bit-Packing: Sequences of repeating numbers or booleans are compressed into count-value pairs. Consecutive sequences of True/False values can be packed into single bits.

• Delta Encoding: Timestamps and monotonically increasing sequence IDs are encoded as numeric deltas between consecutive rows, collapsing multi-byte integers into tiny offsets.

Once encoded, pages are passed through modern lossless compression codecs such as Snappy (optimized for ultra-fast decompression) or Zstandard / Zstd (offering the highest compression ratios with balanced CPU utilization).

Statistics and Predicate Pushdown

Every Parquet file includes a File Metadata Footer written at the very end of the file. This footer contains the exact byte offsets of all Row Groups alongside column-level statistics (min and max values).

When an analytical engine executes a filter such as "WHERE age > 65", it first reads the lightweight footer. If a Row Group reports "max_value: 42" for the age column, the query engine skips reading that entire Row Group from disk or network storage. In cloud environments like AWS S3 or Snowflake, this predicate pushdown eliminates immense data transfer costs.

Frequently Asked Questions