How to Inspect Parquet Metadata, Schema, and Row Groups

Why Inspecting Metadata Matters in Data Pipelines

In production data lakes, silent data corruptions and schema drift cause significant downtime. A single upstream service changing a column type from INT64 to DOUBLE can cause downstream Spark or Snowflake ingestion jobs to fail.

Inspecting Parquet metadata allows data engineers to verify file health, validate row counts, confirm compression codecs, and ensure that min/max statistics are correctly calculated for query optimization.

Anatomy of the Parquet File Footer

Unlike CSV files where headers are located at the beginning, a Parquet file locates its schema and metadata at the very end of the file. The last 4 bytes of every valid Parquet file must equal the ASCII string "PAR1" (the magic number).

Immediately preceding the final magic number is a 4-byte integer defining the length of the FileMetaData block, serialized using the Apache Thrift protocol. The metadata footer includes:

• Schema Definition: Hierarchical tree of field names, physical data types (BOOLEAN, INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY), and logical annotations (UTF8, TIMESTAMP, DECIMAL, DATE).

• Row Group Array: Pointers to byte offsets, uncompressed sizes, compressed sizes, and record counts for each row partition.

• Key-Value Custom Metadata: Arbitrary key-value pairs frequently used by libraries (e.g., pandas metadata, GeoParquet spatial projections).

Inspecting Metadata with SQL and TableView

With TableView.dev, inspecting schema and metadata takes zero command-line configuration. Simply open the file, navigate to the Schema tab, and view all column definitions, nullability, and physical data types instantly.

For CLI users, DuckDB provides built-in metadata inspection functions that extract row group statistics programmatically:

-- Inspect column schemas and types:
SELECT * FROM parquet_schema('production_orders.parquet');

-- Inspect row groups, codecs, and compression metrics:
SELECT 
  row_group_id, 
  column_id, 
  file_offset, 
  total_uncompressed_size / total_compressed_size AS compression_ratio
FROM parquet_metadata('production_orders.parquet');

Frequently Asked Questions