If your lake already stores Parquet files, why doesn't that automatically give you a table?

That gap trips up a lot of teams. They have files, folders, partitions, and SQL engines that can read them. But they don't yet have a reliable answer to basic production questions like: Which files make up the current version of this table? What happens if two jobs write at once? Can I roll back a bad load without rebuilding data by hand?

Table of Contents

• What Is an Open Table Format

• The Metadata Layer That Changes Everything

  • Six things the metadata layer changes

• Apache Iceberg, Delta Lake, and Apache Hudi Compared

  • Iceberg vs Delta Lake vs Hudi at a Glance

  • Where the designs differ

  • What those choices mean in production

  • A practical way to choose

• How a Query Actually Reaches the Data

  • First, the engine asks for the table's current state

  • Then the table format supplies the file map

  • Planning happens before scanning

• Real Benefits and the Trade-offs You Will Hit

  • Open Table Format Benefits and Trade-offs at a Glance

  • Each benefit creates a new responsibility

  • The hidden cost is maintenance work

• Practical Implementation and Migration Considerations

  • Start with the engines you already run

  • Choose the catalog like it is infrastructure

  • Migration is usually less glamorous than the announcement

  • A practical default

• What Open Table Formats Mean for Data Observability

  • Metadata becomes telemetry

  • Observability gets closer to the write path

• Where to Go From Here

What Is an Open Table Format

If your lake already stores Parquet files, why do production pipelines still break on something as simple as "what is the current table"?

Teams running concurrent Spark writes to a partitioned Parquet prefix often learn the answer the hard way. Files exist, but table state is unclear. One engine may read a partially updated partition, another may miss newly added files, and an on-call engineer ends up inspecting object storage paths by hand to work out what "current" even means.

An open table format solves that problem at the metadata layer. It defines how a table tracks its files, versions, schema changes, and write operations so multiple engines can treat data in object storage like a real table instead of a loose directory convention.

That distinction matters because the core decision is architectural, not about choosing a new file type. The data files are often still Parquet or ORC. If you want a refresher on the storage layer underneath, this Parquet file format explainer covers that foundation. The open table format sits above those files and records the rules for table state.

A useful analogy is a warehouse with labeled inventory bins. Parquet files are the boxes on the floor. An open table format is the inventory system that says which boxes belong to the current shipment, which ones were replaced, who updated the records, and what the warehouse looked like before the last bad change. Without that inventory system, every reader has to guess from what it can see in storage.

In practice, that metadata architecture gives you table behavior that plain files do not provide cleanly on their own. You get ACID transactions, schema evolution, and time travel because the format keeps a consistent record of table state and changes over time. That is why Apache Hudi, Apache Iceberg, and Delta Lake are usually discussed together. They are different ways to manage the same underlying problem.

The operational outcomes are the part engineers feel. Schema drift becomes easier to control because the table has an authoritative schema history. Query cost drops because engines can use metadata to avoid scanning every file just to discover what exists. Observability improves because you can inspect snapshots, commits, and file-level changes instead of treating a storage prefix like a black box.

So the shortest accurate definition is this: an open table format is an open specification for managing table metadata and table behavior on top of files in object storage. The files hold the data. The format defines how systems agree on what the table is.

The Metadata Layer That Changes Everything

Why do two engines sometimes read the same lake and return different answers?

The root cause is often not the files themselves. It is the absence of a shared, authoritative record of table state. That is why open table formats are best understood as a metadata architecture choice. The files still matter, but production behavior usually depends on how the metadata layer defines truth.

A library catalog is a useful model here. The books on the shelves are your Parquet files. The catalog decides which books belong to the collection, which edition is current, which books were removed, and what the collection looked like yesterday. Without that catalog, every reader walks the shelves and makes its own guess.

That distinction shows up fast in operations. Query cost rises when engines have to list directories and inspect files just to discover the table. Schema drift gets harder to contain when no system owns schema history. Observability stays weak when a storage prefix is the only thing you can inspect.

For broader context on how teams organize and govern metadata, digna's metadata management guide covers the surrounding discipline.

Six things the metadata layer changes

1. Snapshots create a consistent table view
   Readers do not interpret "whatever files exist right now" as the table. They read from a defined snapshot, which gives every engine the same answer about current state at that moment.

2. File tracking makes planning cheaper
   Instead of scanning a whole directory tree, the engine can consult metadata that already lists relevant data files and their stats. The practical effect is lower query cost and more predictable planning, especially as tables grow.

3. Transactions prevent partial visibility
   A write becomes visible when the commit succeeds. If a job fails halfway through, readers stay on the previous valid snapshot rather than seeing a mix of old and new files.

4. Schema evolution becomes explicit
   Column additions, renames, and type changes are recorded as table metadata changes. That gives you a source of truth for schema history instead of relying on each engine to infer structure from files.

5. Partitioning becomes a table rule
   Directory layout stops carrying so much meaning. The table metadata can describe partition logic directly, which makes layout changes easier to manage over time.

6. History becomes inspectable
   Older snapshots remain part of the table record. That helps with rollback, debugging, audit questions, and simple questions like "what changed between these two runs?"

A concrete failure mode makes this easier to see.

A batch pipeline writes a new day of data into object storage. One query engine starts reading while the write is still in progress. Another engine reads a few minutes later, after more files have landed. Both teams say they queried "the same table." They did not. They queried different file listings at different moments.

With an open table format, the commit publishes a new table state only when it is complete. Until then, readers continue to use the prior snapshot. That is the operational shift. You stop treating folder listing as the contract and start treating metadata as the contract.

Practical rule: If folder contents define your table at read time, your table behavior is still based on storage conventions.

Apache Iceberg makes that contract especially clear. Its table specification defines table metadata as a JSON object that tracks schema, partition specs, properties, and snapshots, with snapshots recorded in the table metadata itself, as described in the Iceberg table specification.

That is why this layer changes so much in production. It determines how systems agree on table truth, and that decision affects correctness, cost, and how quickly your team can explain a bad result.

Apache Iceberg, Delta Lake, and Apache Hudi Compared

If open table formats are really a metadata architecture choice, not just a file-format choice, then the comparison changes. You are not picking between three ways to store Parquet. You are picking three ways to define table truth, publish changes, and coordinate readers and writers in production.

That framing matters because the operational pain shows up in different places. One format may reduce cross-engine friction. Another may fit a Spark-centered platform more naturally. A third may make frequent updates and CDC pipelines easier to operate. The practical question is simple: where do you want the metadata system to carry the heaviest load?

Iceberg vs Delta Lake vs Hudi at a Glance

The origin story is less important than the design center. Each format answers the same question differently: how should a table record changes, expose current state, and let compute engines agree on what they are reading?

Where the designs differ

Apache Iceberg centers the table around a metadata tree. That usually appeals to teams running multiple engines because the table contract is defined as a portable specification rather than by one processing stack's behavior. In practice, that can mean fewer surprises when Spark, Trino, Flink, and other engines need to read the same data without inventing separate conventions.

Delta Lake centers the table around a transaction log. For teams already standardized on Spark and Databricks, that often feels natural because the write path, governance controls, and operational tooling line up closely. The metadata choice shows up in daily work. Schema changes, rollback, and table maintenance can feel more integrated when the broader platform already assumes Delta semantics.

Apache Hudi centers much more of the design on write-heavy operation. If your pain is not "many engines must agree" but "records change constantly and downstream consumers need incremental movement," Hudi often enters the conversation early. Its architecture reflects that bias. The Hudi technical spec explains that metadata is stored in an internally managed table under the Hudi technical spec .hoodie/metadata path, and that this metadata table is itself a Hudi table using merge-on-read.

A useful analogy is a warehouse inventory system. Iceberg is designed so many departments can trust the same catalog record. Delta is designed so the warehouse and its operating system work tightly together. Hudi is designed for a warehouse where items are constantly being corrected, replaced, or restocked, and the inventory system needs to keep up with that churn.

What those choices mean in production

Teams usually feel the difference first.

If schema drift is a recurring problem, the metadata model matters because schema evolution is not just a feature checkbox. It affects whether downstream jobs fail loudly, read mixed assumptions, or require engine-specific handling.

If query cost keeps rising, metadata layout matters because the table has to help the engine avoid unnecessary files. A cleaner planning path usually means fewer files scanned and fewer expensive surprises at runtime.

If observability gaps are the issue, the format choice matters because the metadata history determines how easily your team can answer basic operational questions. Which snapshot introduced the bad records? Which commit changed partition behavior? Which writer published incompatible files?

That is also why table-format decisions and quality controls end up connected. In Databricks-heavy environments, Databricks data quality practices often need to be designed alongside Delta table behavior, not treated as a separate layer added later.

A practical way to choose

Use Iceberg if cross-engine interoperability is a first-order requirement and you want the table spec itself to stay as independent as possible from one vendor runtime.

Use Delta Lake if your platform already runs heavily on Spark or Databricks and you value tight integration across the write path, governance, and day-to-day operations.

Use Hudi if frequent updates, CDC ingestion, and incremental consumption define the workload more than broad engine portability.

No format wins every category. Each one places the metadata contract in a slightly different role. That design choice shapes how you handle correctness, cost, and incident debugging once the table is in production.

How a Query Actually Reaches the Data

Why does the same SQL query feel cheap on one table and painfully expensive on another, even when both point to the same object store?

The answer usually sits in metadata, not in Parquet itself.

SELECT * FROM sales WHERE order_date = current_date

A query like this does not start by scanning folders. It starts by asking, "What does this table mean right now?" That is the key mental model. An open table format is a metadata architecture for answering that question reliably.

First, the engine asks for the table's current state

The query engine resolves sales through a catalog such as Hive Metastore, AWS Glue, Unity Catalog, or Nessie. The catalog acts like a receptionist with the current address, not like the warehouse holding every box. Its job is to point the engine to the current table metadata entry.

That detail has real operational consequences. If the catalog points to stale metadata, the engine plans against stale table state. If an ingestion job drops files directly into storage without a valid commit, those files may exist physically but remain invisible logically.

For a wider architecture view, this data system architecture overview helps frame where the catalog sits relative to storage and compute.

Then the table format supplies the file map

Once the engine has the metadata entry, the open table format takes over:

• Iceberg reads table metadata, manifests, and the active snapshot.

• Delta Lake reads the transaction log plus any checkpointed state.

• Hudi reads its timeline and table metadata to determine the current view.

Different mechanics, same purpose. The engine needs an authoritative file map before it reads any data files.

That is the practical shift from a plain data lake to an open table format. Without this metadata layer, the engine often has to infer table state from folders and file names. With it, table state is declared explicitly.

Planning happens before scanning

Query cost starts to diverge.

After the engine learns which files belong to the current snapshot, it can narrow the work using partition data, file statistics, and commit metadata. Instead of opening every file under a path and figuring things out late, it can rule out irrelevant files during planning.

That changes production behavior in ways data engineers notice quickly:

• Lower query cost because fewer files need to be opened

• Less schema drift confusion because the read path follows committed table metadata, not whatever files happened to land in storage

• Better incident debugging because you can inspect snapshots, commits, and file membership directly

• Fewer observability blind spots because table state is recorded as metadata, not hidden in folder conventions

If a dashboard suddenly looks wrong, the debugging question becomes more precise. Which snapshot did the engine read? Which commit added those files? Which metadata change altered partition or schema behavior?

That is why open table formats are better understood as a control plane for tables. The query reaches data only after metadata defines what the table is, which files belong to it, and which files can be ignored.

Real Benefits and the Trade-offs You Will Hit

The first sprint after adopting an open table format usually feels good. Writes become more reliable. Table changes stop feeling like folder surgery. Teams regain confidence that a table means one coherent thing at a point in time.

Then production reality shows up.

Open Table Format Benefits and Trade-offs at a Glance

Each benefit creates a new responsibility

Reliable writes remove a class of lake failures, but they also create a new outage boundary around the metadata path and catalog.

Time travel helps when someone publishes bad data, but historical snapshots don't manage themselves. If you never expire old snapshots or clean metadata, storage and planning overhead grow.

Schema evolution reduces brute-force rewrites, but it doesn't remove the need for data contracts. A renamed column can still break a downstream consumer that expected the old name.

For teams comparing lakehouse patterns more broadly, this data lake vs data mart comparison is useful because it highlights how storage flexibility and curated consumption serve different operational goals.

The hidden cost is maintenance work

This is the part many architecture diagrams skip. Self-managed open tables still need compaction, snapshot expiration, metadata cleanup, and ongoing maintenance. Without that work, performance degrades and storage costs grow, as discussed in the CDO Magazine analysis of open tables and interoperability.

The same piece makes another important point. Open table formats improve portability, but they don't automatically solve reliability or observability, especially in fast-changing, multi-engine environments.

Operational reality: Open table formats reduce file chaos. They don't remove the need for platform discipline.

There's also a governance catch. Teams often celebrate that the table format is open, while leaving the catalog, access policies, masking rules, and audit behavior under one vendor's control. Recent commentary argues that if those layers remain vendor-controlled, the architecture isn't fully open in practice, even if the files and table format are, as explored in Onehouse's perspective on open table formats and open lakehouse architecture.

Practical Implementation and Migration Considerations

Migration decisions go more smoothly when you make them in the order operations teams face them.

Start with the engines you already run

If Spark dominates and your team already depends on Delta-native workflows, Delta may be the least disruptive path.

If your environment is multi-engine, Iceberg often fits the operating model more cleanly.

If your hardest requirement is frequent upserts from CDC and streaming sources, Hudi deserves serious evaluation.

Choose the catalog like it is infrastructure

A lot of teams treat the catalog as an implementation detail. It isn't. It becomes part of your control plane.

Use a simple catalog if simplicity is the priority. Use a cloud-integrated or platform-governed catalog if security, lineage, and centralized policy matter more. If you expect multiple engines across clouds, choose a catalog strategy that won't trap governance in one runtime.

Migration is usually less glamorous than the announcement

Common migration patterns include:

• Rewrite into new tables: Cleanest semantics, highest up-front movement

• Dual writes for a period: Safer cutover, more temporary complexity

• Clone or convert where possible: Faster, but validate assumptions carefully

• Phased domain rollout: Better for large estates with different workload types

The quiet risk isn't usually the file conversion itself. It's forgetting to stand up day-one maintenance.

You need jobs or managed services for compaction, retention, and cleanup before the first important table goes live. If you skip that, the platform starts drifting immediately.

A practical default

A reasonable default is simple:

• Pick Iceberg for multi-engine SQL and governance-oriented lakehouse design.

• Pick Delta Lake if your platform already leans heavily on Databricks and Spark.

• Pick Hudi when the write path is dominated by CDC, upserts, and streaming correction patterns.

And if you're instrumenting table health from day one, one option is digna, which runs in the customer's environment and monitors schema changes, timeliness, anomalies, and record validation across lakes, warehouses, and pipelines.

What Open Table Formats Mean for Data Observability

Open table formats don't just improve storage semantics. They also expose a metadata surface that observability tools can read directly.

Metadata becomes telemetry

Iceberg snapshots, Delta transaction logs, and Hudi commit metadata all create machine-readable evidence of table change. That matters because many quality and reliability checks don't need to scan full datasets first.

A platform can inspect commit history and ask:

• Did a schema change arrive unexpectedly?

• Did today's partition land later than usual?

• Did a write publish far fewer files than the pipeline normally produces?

• Did the table stop advancing even though upstream jobs ran?

Observability gets closer to the write path

That changes the operating model for monitoring. Instead of relying only on downstream query failures or dashboard complaints, teams can detect issues at the point where table state changes.

When metadata tells you the table changed, observability can test the change before users discover the break.

This is one reason open table formats matter beyond storage and query speed. They create a shared control surface that engines, governance systems, and observability tooling can all read.

The caveat is important. Metadata helps expose signals. It doesn't interpret them for you. Teams still need rules, baselines, and incident workflows that connect table change to business impact.

Where to Go From Here

An open table format is not the finish line. It's the foundation for a better-behaved lakehouse.

Before calling the architecture production-ready, verify a few things:

• Catalog resilience: Make sure the catalog is highly available enough to become part of your control plane.

• Maintenance jobs: Confirm compaction, snapshot expiration, and metadata cleanup are scheduled from day one.

• Retention policy: Align time-travel history with compliance and rollback needs.

• Engine behavior: Check that engines read table metadata directly rather than relying on directory listing shortcuts.

• Observability wiring: Monitor commit and snapshot signals, not just query outputs.

If you're evaluating Iceberg on an existing lake, start with one table that already hurts from schema drift or multi-engine access.

If you're layering Delta into a Spark-heavy shop, begin where transaction safety matters more than portability debates.

If you're testing Hudi for CDC-heavy workloads, choose a table where upserts and incremental consumption are already the source of operational pain.

The useful question isn't just what is an open table format. It's whether your team is ready to operate the metadata architecture that comes with it.

digna gives data teams an in-environment platform for data quality and observability, which fits naturally with open table formats because so much of the useful signal lives in commits, schema changes, and freshness metadata. If you're turning raw lake files into governed, production-facing tables, visit digna to see how it monitors schema drift, timeliness, anomalies, and validation without moving your data out of your environment.

The metadata layer makes a query planner’s job easier; making that same metadata answer operational questions is what data platform observability adds on top.

Frequently asked questions

What does the metadata layer actually store?

Schema and its history, the list of data files making up each version of the table, partition definitions, and per-file statistics such as value ranges and null counts. That last part is what lets a query planner discard files before it ever opens them.

How does a query reach the data?

The engine asks the catalog for the current metadata pointer, reads the manifest to get the file list with statistics, discards files whose statistics cannot satisfy the filter, and only then opens the remaining Parquet files. Most of the speed comes from the files it never opens at all.

Does an open table format improve data quality?

It improves consistency, not correctness. The format guarantees every engine sees the same committed version of the table with the same schema; it makes no claim about whether the values are accurate, complete or fresh. Those stay separate checks layered on top of the format.

What can you observe from table metadata?

Commit frequency shows whether loads are arriving on schedule, snapshot diffs show which files a change touched, and schema history shows when a column's type moved. These signals catch a class of problem earlier than row-level tests, because they surface before anyone queries the data.

Is there a cost to adopting one?

Yes, though it is operational rather than licensing. You inherit snapshot expiry, file compaction and catalog availability as things that must be maintained, and query planning gains a metadata round trip. For small, rarely changing datasets, plain Parquet with a stable layout can still be the right answer.