Deep Dive into Parquet Encodings: RLE, Bit-Packing, Dictionary, and Delta Compression

The Two-Stage Compression Pipeline in Columnar Formats

A common misconception in data engineering is that Parquet compact file size is primarily due to generic compression algorithms like Snappy, Gzip, or Zstandard.

In reality, generic byte compression is merely the final step in a two-stage pipeline. The true foundation of Parquet efficiency is domain-specific Columnar Encoding. Because all values in a column share the exact same data type, Parquet applies mathematical encodings that exploit data distribution and cardinality, drastically collapsing data volume before passing bytes to a general-purpose compressor.

Dictionary Encoding: Eliminating String Redundancy

When a column contains repetitive values (such as state abbreviations, country codes, or product category names), storing the full string repeatedly wastes massive disk and memory bandwidth.

Dictionary Encoding builds a small dictionary table containing each unique string once, assigning each an integer index (0, 1, 2...). The column data is then stored simply as a sequence of tiny bit-packed integer keys. If the number of distinct values exceeds a preset threshold (typically 40,000 unique values per row group), the encoder automatically falls back to PLAIN encoding.

Run-Length Encoding (RLE) and Bit-Packing

Run-Length Encoding (RLE) replaces consecutive sequences of identical values with a single count-value pair. For example, the boolean sequence [True, True, True, True, True] is encoded simply as (5, True).

Bit-Packing eliminates unused bits in integer representations. If an integer column has a maximum value of 3, standard 32-bit or 64-bit integer allocations waste 30 to 62 bits per row. Bit-packing stores each value using exactly 2 bits (since 2 bits can represent 0, 1, 2, and 3), packing 16 values into a single 32-bit word.

Delta Encoding: Monotonically Increasing Data and Timestamps

Timestamp columns and auto-incrementing database primary keys often consume substantial storage. Delta Encoding stores only the differences (deltas) between consecutive values rather than the full multi-byte numbers.

In a sequence like [1000000, 1000002, 1000005, 1000008], storing the deltas [0, 2, 3, 3] compresses multi-byte numbers down to single bytes, enabling dramatic space reduction.

Frequently Asked Questions