What Is Parquet: The Columnar Format Explained

You've probably met Parquet without realizing it. A dashboard query runs against files in object storage, the SQL looks harmless, and yet a seemingly simple aggregation spends most of its time reading data that the query never needs. The same dataset might work smoothly in DuckDB or Spark, then behave differently in another engine because a logical type, statistic, or newer feature isn't interpreted the same way.

So, what is Parquet? Apache Parquet is an open-source, column-oriented data file format designed for efficient storage and retrieval, especially for analytical workloads. It isn't a database and it doesn't replace a table format such as Iceberg or Delta Lake. It's the physical file layer underneath many data lakes, warehouses, analytical exports, and machine learning pipelines.

Table of Contents

• Why Parquet Exists and Where It Fits

  • Where Parquet Fits in a Modern Data Stack

• How Parquet Came to Be

• Inside a Parquet File

  • Row groups are the floors

  • Column chunks are the shelves

  • Pages are the smallest practical read unit

• Encoding and Compression Working Together

  • Common Parquet Encodings and Codecs at a Glance

• Predicate Pushdown and Why Queries Get Faster

  • What the reader checks

• Parquet vs CSV, JSON, and ORC

  • Parquet vs CSV vs JSON vs ORC

• Interoperability, Security, and Real-World Trade-Offs

  • Engine support is a compatibility question

  • The costs behind the convenience

• Practical Tips for Analytics Workloads

  • A practical layout checklist

  • Governance and AI considerations

Why Parquet Exists and Where It Fits

An analyst runs:

SELECT region, AVG(revenue)
FROM sales
GROUP BY region;

SELECT region, AVG(revenue)
FROM sales
GROUP BY region;

SELECT region, AVG(revenue)
FROM sales
GROUP BY region;

On a large CSV, the query engine typically parses rows across the file and inspects fields that the calculation never uses. Storage reads and CPU time go toward unrelated columns. Parquet groups values by column, allowing the reader to focus on region and revenue instead.

The key idea is physical organization. A row-oriented file keeps each record's fields together, while a columnar file groups values from the same column. Columns commonly contain similar data types and repeated values, giving encoding and compression methods a more predictable pattern. Apache describes Parquet as an open-source, column-oriented data file format for efficient storage and retrieval in its official documentation.

That design suits several common workloads:

• Cloud data lakes: Files in Amazon S3 and similar object stores can be queried without loading every field into a database.

• Analytical fact tables: Spark, Trino, and Presto can scan selected columns from large event or transaction datasets.

• Operational exports: Teams can extract application data and publish an analytical copy without retaining a text-heavy interchange format.

• Machine learning features: Training pipelines and feature stores can exchange typed, compressed files instead of repeatedly parsing CSV or JSON.

Parquet also has a specific place in a data lake stack. A data lake holds files and metadata. Table formats add transactions, snapshots, partition tracking, and schema management. This comparison of data lakes and data marts places Parquet within that broader architecture.

Where Parquet Fits in a Modern Data Stack

Parquet sits underneath table formats such as Iceberg and Delta Lake, providing the physical file layer rather than managing transactions or snapshots. That separation explains why a Parquet file can work across several engines, while table-level behavior still depends on the surrounding metadata and query system.

The practical mental model is simple: Parquet makes analytical file scans selective and economical. Its benefits depend on how files are laid out, how values move through encoding and compression, how filters reach the reader, and whether the chosen engine supports the relevant features consistently.

How Parquet Came to Be

Parquet emerged from a practical Hadoop problem: analytical queries often needed a few fields from very wide records. Twitter and Cloudera developed it as an open-source project, influenced by Google's Dremel work on columnar processing for nested data. The first release arrived on 13 March 2013, followed by Parquet 1.0 in July 2013, as recorded in the Apache Parquet history.

The design target was selective reading. If a query requested customer_id and revenue from a wide event record, the reader could avoid parsing unrelated fields. Dremel's influence also pointed Parquet toward nested analytical data, not only flat spreadsheet-like tables.

ORC developed in the Hive ecosystem around similar goals, giving Hadoop users another serious columnar choice. Parquet's role grew as more tools adopted compatible readers and writers. The format has also continued to evolve. One example is BYTE_STREAM_SPLIT, approved in the 2.8.0 release line in 2019, which added a standardized capability rather than leaving the format unchanged.

The project became a top-level Apache Software Foundation project on 27 April 2015. That governance supports an ecosystem with many independent implementations instead of a single vendor-controlled reader. Spark, Trino, Presto, Hive, ClickHouse, DuckDB, Polars, BigQuery, Snowflake, Athena, and Redshift are among the engines and services listed in the format ecosystem documentation. Compatibility still depends on which features each engine implements and how it handles schemas, encodings, and metadata.

That matters at ingestion time. A data ingestion pipeline can write analytical batches as Parquet, while a table format adds transactions and snapshots and a query engine decides how to scan the files.

Historical lesson: Parquet succeeded by matching a common analytical access pattern, then spreading across engines that could read selected columns efficiently.

Its history explains the division of responsibilities: Parquet handles the physical file, encoding, and compression layers, while surrounding systems provide table management and query behavior.

Inside a Parquet File

A query needs only a few fields from a large table, yet a row-oriented file may make it read every field. Parquet avoids that waste through a nested physical layout. A useful analogy is a library building: the Parquet file is the building, a row group is a floor, a column chunk is a shelf, and a page is a stack of books on that shelf. Each level has a specific job.

A Parquet file starts with a 4-byte magic header, PAR1. Its data is organized into row groups. Every row group has exactly one column chunk per column, and each column chunk contains one or more pages. This hierarchy gives readers the structure needed to select columns and skip irrelevant data, as described in Apache's Parquet concepts documentation.

Row groups are the floors

A row group contains a horizontal slice of table rows. For columns such as event_date, region, and revenue, one row group holds a portion of all three. Their row ranges remain aligned, so an engine can evaluate the same records while reading only the columns required by a query.

Writers commonly target row groups in the broad range of 128 MB to 1 GB. The appropriate setting depends on the engine, storage system, workload, and number of columns. Larger groups can reduce metadata overhead, while smaller groups provide finer-grained skipping. Treat row-group sizing as a workload decision rather than a universal constant.

Column chunks are the shelves

Inside a row group, each column has its own column chunk. The region chunk stores encoded region values, while the revenue chunk stores encoded revenue values. A query that needs those fields can avoid reading an unrelated customer_notes chunk.

The footer records each chunk's location and metadata. It also describes the schema and column metadata paths, allowing a reader to interpret nested fields and locate relevant bytes without scanning from the beginning.

Pages are the smallest practical read unit

Column chunks are divided into pages containing encoded values and page metadata. A common default page size is 1 MB, though writers and engines may choose other settings. Pages provide a finer boundary for statistics, compression, and selective reads.

The file ends with a footer containing the information needed to decode the data, followed by a final PAR1 marker. A reader can seek to the end, read the footer, identify the schema and row-group locations, then issue targeted reads for selected chunks and pages. This structure makes Parquet binary, self-describing, and splittable for distributed processing.

Reader's mental model: Start at the footer, identify useful row groups and columns, then read targeted chunks and pages.

Parquet carries the structural information required to read itself, so a separate catalog is not required. A catalog can still make discovery easier. Strong metadata management helps at table scale, especially when many files and schema versions must be tracked. File portability also depends on readers supporting the features used by the writer, not merely on both tools calling themselves Parquet-compatible.

Encoding and Compression Working Together

People often use “encoding” and “compression” as if they mean the same thing. In Parquet, they're separate stages in a pipeline:

1. Encoding reshapes values to expose patterns.

2. Compression reduces the resulting byte stream.

Consider a country column containing repeated strings. Dictionary encoding can assign each distinct value a small integer identifier, then store those identifiers instead of repeating full strings throughout the page. Run-length encoding can represent a sequence of identical values compactly. Delta encoding stores differences between consecutive values, which can be effective when numbers are ordered or change gradually. Bit-packing stores small integers using only the bits they require.

These techniques don't all suit every column. Dictionary encoding is a natural fit for repeated categorical values, while delta encoding needs a useful sequence pattern. The writer chooses an encoding for the column and page, and the metadata records that choice.

Only after this reshaping does a page-level codec compress the bytes. Apache's documentation describes codecs such as Snappy, GZIP, LZ4, and ZSTD, while PyArrow also supports Brotli and uncompressed output in its Parquet configuration documentation. The format documentation explains that page contents are compressed as-is, without additional framing or padding, which keeps decompression behavior straightforward.

Common Parquet Encodings and Codecs at a Glance

Encoding and codec settings live in different parts of the file metadata. The encoding choice belongs with the page and column information, while the compression codec is recorded in the relevant page metadata. Writers such as Apache Spark, DuckDB, and PyArrow may make different defaults, so two valid Parquet files can have different performance profiles.

A common tuning mistake is selecting an aggressive compression level without measuring the workload. Once a column has already been made highly regular through encoding, a slower setting such as ZSTD level 19 may add CPU cost without producing a useful reduction in size. The right choice depends on whether the bottleneck is storage, network transfer, CPU, or latency.

Predicate Pushdown and Why Queries Get Faster

A dashboard query filters January sales, groups results by region, and sums revenue. A Parquet reader may answer it without decoding most of the file. The mechanism is called predicate pushdown, or more precisely, filtering and pruning at the storage layer.

SELECT region, SUM(revenue)
FROM sales
WHERE event_date BETWEEN '2025-01-01' AND '2025-01-31'
GROUP BY region;

SELECT region, SUM(revenue)
FROM sales
WHERE event_date BETWEEN '2025-01-01' AND '2025-01-31'
GROUP BY region;

SELECT region, SUM(revenue)
FROM sales
WHERE event_date BETWEEN '2025-01-01' AND '2025-01-31'
GROUP BY region;

The engine first checks the minimum and maximum event_date recorded for each row group. If a row group lies entirely outside January, the reader can skip it without opening its data pages. For a row group that overlaps the date range, page-level statistics can remove pages whose ranges still cannot match.

The reader then requests only the event_date, region, and revenue pages needed for the query. Other columns share the same rows, but they do not need to be decoded. Parquet's file-level and column-level structure provides the metadata layout that makes this selective access possible.

What the reader checks

A typical pruning sequence includes:

• Row-group statistics: Compare the filter with each column chunk's recorded range.

• Page statistics: Inspect finer-grained minimum and maximum values within candidate row groups.

• Dictionary information: Resolve some categorical filters from dictionary page values.

• Bloom filters: Check probabilistic membership information when the file contains it and the reader supports it.

• Column selection: Decode only projected columns required for the result.

File layout determines how much work pruning can avoid. If values are scattered across every page, min/max ranges overlap and the engine has little to skip. Sorting by a frequently filtered column can cluster similar values, although the useful arrangement depends on the writer and table design.

Pushdown is also an engine behavior, not a promise that every predicate is handled identically. Readers may restrict pruning for nested types or interpret logical types differently. Dictionary and Bloom filter support can vary as well. For broader guidance on optimizing SQL queries, connect the storage behavior to the query plan.

The operational rule is simple: Parquet does not make every query fast. It makes selective queries cheaper when file layout and metadata support pruning.

Parquet vs CSV, JSON, and ORC

The right format depends on the job. CSV and JSON work well as interchange formats because people can inspect them, applications can produce them easily, and APIs commonly accept them. Repeated analytical scans expose their limits: values are stored as text, and readers usually parse records one by one.

Parquet makes a different trade-off. It stores typed data in a binary, compressed, column-oriented layout, so an engine can read only the columns needed by a query. That makes it a strong default for analytical storage, while making it inconvenient to open in a text editor or send directly to a consumer that expects JSON.

The encoding pipeline matters here. Parquet can encode similar values together, apply dictionary or run-length techniques where appropriate, and then compress the resulting pages. CSV and JSON usually leave more parsing and type inference to the reader. ORC follows a similar columnar model and is the closest comparison, especially in Hive-oriented environments.

Parquet has direct support across tools including Spark, Trino, Presto, Hive, ClickHouse, DuckDB, Polars, BigQuery, Snowflake, Athena, and Redshift, as described in the Apache format documentation. The shared core improves portability, though it does not guarantee identical behavior for every logical type, nested structure, or metadata feature.

Parquet vs CSV vs JSON vs ORC

Schema evolution still requires coordination. Parquet records schema information in each file, while a table system or catalog usually reconciles versions across a collection of files. Adding a nullable field is generally easier than changing an existing field's meaning or physical representation. Logical types can also produce different results when readers support them unevenly.

Use CSV or JSON when humans, APIs, or simple ingestion tools need the output. Choose ORC when Hive compatibility and an ORC-centered stack determine the format. Choose Parquet for modern analytical lakes and cross-engine data exchange, then test the specific readers, writers, and logical types in your environment.

Interoperability, Security, and Real-World Trade-Offs

Parquet's broad adoption can create a misleading impression that every engine reads every file identically. Spark, DuckDB, Trino, BigQuery, Athena, and Polars generally agree on the core layout, but they can diverge on nested types, logical types such as UUID and JSON, decimal precision, page indexes, and the meaning of statistics in the footer.

That creates a practical failure mode: one writer produces a valid file, while another reader lacks support for the particular feature used. The file may open successfully in one tool and fail, downgrade a type, or ignore an optimization in another. Apache's implementation status page makes this unevenness explicit. It shows different feature sets across implementations, including Parquet Java reaching 1.16.0 in September 2025, while Arrow Rust, Hyparquet, DuckDB, and Polars track different feature versions.

Engine support is a compatibility question

Security deserves the same attention as interoperability. In April 2025, the critical deserialization vulnerability CVE-2025-30065 affected the parquet-java library. Security guidance warned teams to upgrade to Parquet 1.15.1 and avoid untrusted files, as recorded in the Apache implementation and security guidance.

That changes the answer to “Can I safely read any Parquet file in any tool?” No. Treat files from untrusted uploads or external parties as hostile input. Patch the relevant library, isolate readers where appropriate, avoid accepting remote schemas blindly, and review optional metadata and page-index behavior before enabling aggressive features. A sandboxed reader can reduce blast radius, but it doesn't replace dependency management.

The costs behind the convenience

Parquet also has non-security trade-offs:

• Frequent updates: Immutable analytical files can create write amplification when records change often.

• Small files: Many tiny objects increase listing, open, and metadata overhead in object storage.

• Wide schemas: Footer metadata can grow substantially when files contain many columns and row groups.

• Debugging: Corrupted binary files are harder to inspect than malformed CSV.

• Engine gaps: A feature supported by one implementation may be ignored or rejected by another.

Teams also need to account for data residency requirements when selecting storage locations, readers, and processing paths. Parquet improves physical storage and access patterns, but it doesn't by itself solve governance, isolation, or regulatory control.

Practical Tips for Analytics Workloads

A reliable Parquet setup starts with file layout rather than codec tweaking. For many cloud analytics workloads, target row groups around 128 MB to 256 MB and pages near 1 MB, as shown in this Parquet optimization playbook. These are starting points, not guarantees. Test with the engine, object store, and query shapes your team uses.

A practical layout checklist

• Choose row groups deliberately: Balance targeted reads against metadata overhead and object-store request behavior.

• Keep pages moderate: A page size near 1 MB is a sensible baseline for compression and page-level filtering.

• Sort useful filter columns: event_time and user_id can benefit when queries filter selectively and values cluster within pages.

• Avoid indiscriminate sorting: Sorting adds write work, so prioritize columns that appear in important predicates.

• Partition coarsely: Date or category partitions are often easier to manage than high-cardinality keys that create excessive directory counts.

• Compact small files: A table with many tiny Parquet objects can perform poorly even when each file is individually well formed.

Compression should follow the workload. Snappy is a reasonable speed-oriented choice. ZSTD is useful when storage reduction matters and CPU capacity is available. GZIP can remain appropriate for legacy compatibility, but it's rarely the first choice for interactive analytics when faster codecs are supported.

Parquet isn't a good answer for every dataset. A tiny reference table under a few megabytes may be easier to distribute as CSV or store in a small database. The format earns its keep when repeated analytical scans, column selection, compression, and distributed processing outweigh the cost of binary tooling and file management.

Governance and AI considerations

Enforce schemas at write time, record the writer and reader versions, and test representative files across the engines your consumers use. Use Iceberg or Delta Lake when you need table-level transactions, snapshots, and managed schema evolution. Parquet remains a useful physical interchange layer between training pipelines and feature stores, but it shouldn't be mistaken for a complete governance system.

For teams operating many Parquet datasets, observability should cover more than file existence. Monitor schema changes, delivery timing, record validation, and business metrics so a technically readable file doesn't become analytically unreliable. digna provides in-database data quality, timeliness, anomaly, validation, schema tracking, and platform observability capabilities that can be applied while data remains in the customer's environment.

Working rule: Optimize the file layout for the filters your users run, then validate the result in every engine that matters.

If your lake contains Parquet files but you can't easily track schema drift, late arrivals, or changing data behavior, visit digna to see how in-database observability can monitor those risks inside your own environment. Start with the module that matches your immediate reliability problem, then expand as your analytics and AI workloads grow.