Parquet Metadata Explained: Footer, Row Groups, Statistics

In the previous posts, we’ve covered how Parquet optimizes storage through columnar storage, compression, and encoding. Now, let’s explore another essential feature that sets Parquet apart: metadata. Metadata in Parquet plays a crucial role in improving data efficiency, enabling faster queries and optimized storage. In this post, we’ll dive into the different types of metadata stored in Parquet files, how metadata improves query performance, and best practices for leveraging metadata in your data pipelines.

Reviewed October 15, 2026 against the Apache Parquet file format docs and the page index spec.

What is Metadata in Parquet?

In Parquet, metadata refers to information about the data stored within the file. This information includes things like the structure of the file (schema), statistics about the data, compression details, and more. Metadata is stored at various levels in a Parquet file: file-level, row group-level, and column-level.

By storing rich metadata alongside the actual data, Parquet allows query engines to make decisions about which data to read, which rows to skip, and how to optimize query execution without scanning the entire dataset.

Types of Metadata in Parquet

Parquet files store metadata at three levels:

File-level metadata: Information about the overall file, such as schema and version information.
Row group-level metadata: Statistics about subsets of rows (row groups), such as row count, column sizes, and compression.
Column-level metadata: Detailed statistics about individual columns, such as minimum and maximum values, null counts, and data types.

Let’s take a closer look at each type of metadata and how it improves performance.

1. File-Level Metadata

File-level metadata describes the structure of the entire Parquet file. This includes:

Schema: The schema defines the structure of the data, including column names, data types, and the hierarchical structure of nested fields.
Number of row groups: This specifies how many row groups are stored in the file. Each row group contains data for a specific range of rows.
Version information: This indicates which version of the Parquet format was used to write the file, ensuring compatibility with different readers.

File-level metadata is stored in the footer of the Parquet file, which means it is read first when opening the file. Query engines can use this information to understand the overall structure of the data and determine how to process it efficiently.

2. Row Group-Level Metadata

Parquet files are divided into row groups, and each row group contains a horizontal partition of the data (i.e., a subset of rows). Row group-level metadata provides summary information about the rows contained in each row group, including:

Row count: The number of rows stored in each row group.
Column chunk sizes: The size of each column chunk within the row group, which is useful for estimating the cost of reading specific columns.
Compression and encoding details: Information about the compression algorithm and encoding technique used for each column in the row group.

This metadata allows query engines to skip entire row groups if they’re irrelevant to the query. For example, if a query is filtering for rows where a specific column’s value falls within a certain range, the engine can skip row groups where the column’s values do not meet the filter criteria.

3. Column-Level Metadata (Statistics)

Perhaps the most powerful type of metadata in Parquet is column-level statistics. These statistics provide detailed information about the values stored in each column and include:

Minimum and Maximum Values: The minimum and maximum values for each column. This allows query engines to quickly eliminate irrelevant data by skipping over row groups or pages that do not match query conditions.
Null Counts: The number of null values in each column, which helps optimize queries that filter based on null values.
Distinct Count: Some implementations may include distinct count metadata for columns, which can help in estimating cardinality for query optimization.

Row group statistics live in the column chunk metadata in the footer. Page-level min/max values live in the optional page index (the ColumnIndex and OffsetIndex structures), which writers place near the footer. Older writers put page statistics inside each page header instead, and the Parquet spec now recommends the page index over those header statistics. Together they give query engines fine-grained control over which data to read and which data to skip.

4. File Footer

The file footer is where the file-level, row group and column chunk metadata is stored. A Parquet file ends with the serialized footer, a 4-byte footer length and the 4-byte magic number PAR1. When a query engine opens a Parquet file, it reads those last bytes, then the footer, to understand the file structure, row group layout, and column statistics. This enables query optimization before even touching the actual data. The page index and bloom filters are stored separately near the footer, and their offsets are recorded in the footer, so readers that do not need them never pay to read them.

How Metadata Improves Query Performance

One of the biggest advantages of Parquet’s rich metadata is its ability to enable predicate pushdown. Predicate pushdown is the process of applying filter conditions (predicates) as early as possible in the query execution process to minimize the amount of data that needs to be read.

For example, consider a query that filters for rows where the value in the Age column is greater than 30. With Parquet’s column-level metadata, the query engine can use the min/max statistics to skip entire row groups or pages where the Age column’s maximum value is less than or equal to 30. This significantly reduces the amount of data read from disk, resulting in faster query execution.

Other Ways Metadata Optimizes Queries

Column Pruning: Since Parquet is a columnar format, queries that only require specific columns can skip over irrelevant columns. The metadata helps identify which columns are needed for the query and ensures that only those columns are read.
Row Group Skipping: If a query involves filtering based on a column’s value, Parquet’s row group-level metadata allows the query engine to skip entire row groups that do not match the filter condition. This reduces the number of rows that need to be scanned.
Page Skipping: In addition to skipping row groups, metadata stored at the page level (within row groups) allows fine-grained control, letting the query engine skip pages that do not match query conditions.

Best Practices for Leveraging Metadata in Parquet

To maximize the benefits of Parquet’s metadata, consider these best practices:

Choose Appropriate Row Group Size: Row groups are the primary unit of parallelism and skipping in Parquet. For large datasets, selecting the right row group size is important to balance between performance and memory usage. Smaller row groups allow more precise skipping, but can increase the overhead of metadata management.
Enable Statistics Collection: Ensure that Parquet writers are configured to collect column statistics, as this enables features like predicate pushdown and page skipping. Most modern processing frameworks (like Apache Spark, Dremio, and Hive) enable statistics collection by default.
Optimize for Query Patterns: If your workload involves frequent filtering on specific columns, consider sorting the data based on those columns. This can make min/max statistics more effective for skipping irrelevant data.
Keep the Footer Lean: The footer itself is not compressed, and it grows with the number of row groups and columns. Very wide tables with many small row groups produce large footers that slow down every file open. Prefer fewer, larger row groups, and consider turning off statistics for long string columns you never filter on.
Utilize Data Catalogs: When managing a large number of Parquet files, table formats like Apache Iceberg, tracked in a catalog such as Apache Polaris or Hive Metastore, add table-level metadata on top of Parquet’s file-level metadata, including file-level statistics that let engines skip whole files before opening a single footer, making query optimization more effective.

Frequently Asked Questions

Where is metadata stored in a Parquet file?

The main metadata (the FileMetaData structure) sits in the footer at the end of the file, followed by a 4-byte footer length and the 4-byte magic number PAR1. Readers seek to the end of the file, read the length, then read the footer. Each data page also carries a small page header used to decode it.

What metadata does a Parquet file contain?

The footer holds the schema, the number of rows, the list of row groups, optional key/value metadata, and for every column chunk its encodings, compression codec, sizes, offsets and statistics such as min, max and null count. Optional structures include the page index, bloom filters and size statistics.

How do Parquet statistics enable predicate pushdown?

An engine compares a query filter such as age > 30 with each row group’s min and max for that column. If the maximum is 30 or lower, the whole row group is skipped without being read. The page index applies the same idea to individual pages.

What is the Parquet page index?

The page index is a pair of optional structures, ColumnIndex and OffsetIndex, stored near the footer. ColumnIndex holds per-page min/max values and null information, and OffsetIndex maps rows to page locations, so readers can skip pages without scanning every page header.

How can I inspect Parquet metadata?

In Python, pyarrow.parquet.ParquetFile(path).metadata shows the row groups, columns and statistics, and DuckDB’s parquet_metadata() function returns the same details as a table. Both read only the footer, so they are fast even on large files.

Conclusion

Metadata is one of the most powerful features of Parquet, enabling efficient storage and fast queries through predicate pushdown, column pruning, and row group skipping. By leveraging the rich metadata stored in Parquet files, you can drastically improve query performance and reduce the amount of data that needs to be read from disk.

In the next post, we’ll explore reading and writing Parquet files in Python, using libraries like PyArrow and FastParquet to demonstrate how to work with Parquet files programmatically.

Stay tuned for part 8: Reading and Writing Parquet Files in Python.