It's Tuesday morning. A Spark job has picked up a Parquet file while another process was still writing it. A downstream dbt model reads an incomplete batch, joins it again during a retry, and doubles part of the revenue total. The team doesn't find the problem in the pipeline logs. Finance finds it in a report.

This is the kind of failure that makes a raw data lake feel less like a database and more like a shared folder with excellent throughput. The files may be durable, cheap, and open, but the lake still needs a reliable way to define which files belong to a table, which version readers should see, and how writers publish changes safely. That's the role of an open table format.

Table of Contents

• The Problem an Open Table Format Solves

• How an Open Table Format Works Under the Hood

  • Publishing a new table state

  • Choosing a write layout

• What an Open Table Format Actually Buys You

• Iceberg, Delta Lake, and Hudi Compared

• How Open Table Formats Fit Modern Data Platforms

• Limits and Trade-offs Most Articles Skip

  • Operational work doesn't disappear

• Choosing and Adopting an Open Table Format

  • Pilot

  • Dual-write validation

  • Cutover

  • Decommissioning

The Problem an Open Table Format Solves

A raw data lake usually stores files in object storage, often using formats such as Parquet, ORC, or Avro. The storage system knows that the files exist, but it doesn't automatically know that a particular collection of files represents one logical table, that a new batch is complete, or that a reader should see either the old state or the new state, never a mixture of both.

Directory conventions try to fill that gap. Teams create folders for dates, regions, tenants, or ingestion runs, then ask processing engines to infer table structure from paths and file contents. That approach works until multiple writers, retries, updates, deletes, schema changes, and concurrent readers arrive at the same time.

Practical rule: A folder of files is storage. A table needs a contract for identity, state, and change.

An open table format is a specification and metadata layer that sits above raw data files and below query or processing engines. It describes those files as a versioned, transactional table, including the schema, partitioning rules, active files, and committed snapshots. The data remains in open storage, while the table metadata provides the coordination that raw directories lack. Databricks' overview of open table formats describes this layer as the mechanism that adds capabilities such as ACID transactions, schema evolution, time travel, and row-level updates or deletes to files in object storage.

The format also separates the logical table from one engine's private storage assumptions. Compatible clients can let Spark, Trino, Flink, Snowflake, BigQuery, and DuckDB work with the same underlying table, although exact feature support depends on the engine, connector, catalog, and table format version. That engine neutrality is the important part. A platform team can use Spark for transformation, Trino for interactive SQL, Flink for streaming, and another engine for BI without creating a separate physical copy for every workload.

This is why an open table format belongs in a broader data platform architecture. It doesn't replace the platform. It supplies a dependable table state that the rest of the platform can discover, query, monitor, and govern.

Historically, open table formats emerged from the limitations of Hive-style, directory-based table handling. Apache Hudi began at Uber in 2016, Apache Iceberg originated at Netflix around 2017, and Delta Lake was introduced by Databricks in 2017 and open-sourced in 2019. These milestones marked a move toward file-level metadata management, ACID behavior, schema evolution, and safer updates on cloud object storage. A history of open table formats places those projects at the center of the lakehouse's evolution.

How an Open Table Format Works Under the Hood

An open table format becomes easier to understand when you separate the table into layers. Think of a freezer filled with ice cubes.

The data files are the ice cubes. They contain the actual records, usually in a columnar format such as Parquet. A table may contain many files, and those files can be distributed across object storage.

Partitioning is the tray. It groups files according to a layout that can help query planning, such as a date or another transform of a column. The important distinction is that a modern table format can manage this layout as table metadata instead of forcing every user to understand the physical directory structure.

Manifest files are inventory lists. Each manifest records which data files belong to a particular part of the table and includes information that helps an engine decide which files it can skip. A query for a narrow date range doesn't need to inspect every cube in the freezer if the inventory identifies the relevant tray.

The metadata layer is the binder. It tracks the table schema, partition specification, current snapshot, and the references to manifest lists. A snapshot is one consistent view of the whole binder, not merely a timestamp attached to an arbitrary folder.

Apache Iceberg's model illustrates this layered approach. It organizes tables through immutable snapshots and manifest files. Each commit creates a new point-in-time view while preserving earlier versions for time travel and rollback, with metadata JSON, manifest lists, and manifest files forming the commonly described hierarchy. This Apache Iceberg metadata cheat sheet outlines those components.

Publishing a new table state

A writer doesn't normally overwrite the currently published snapshot in place. It stages new or replacement files, creates updated metadata and manifests, and then commits a new table state through the catalog or table protocol. The final publication step changes the table's current metadata reference atomically.

Readers that start before the commit continue using the earlier snapshot. Readers that start after the commit use the new one. That separation prevents a query from seeing half of a batch because files appeared in object storage at different moments.

A catalog coordinates table identity and discovery. It may use a REST interface, a Hive-style service, or another catalog implementation. The catalog answers questions such as where the table metadata lives and which metadata version is current. It's the control point that stops every engine from inventing its own interpretation of the table.

Manifest lists support planning-time pruning, while metadata stores the schema and partition specification. Hidden partitioning goes a step further by allowing users to query logical columns without writing filters against physical folder names. That means a table can evolve its partition strategy without forcing every analyst to rewrite SQL around storage paths.

Choosing a write layout

Copy-on-write and merge-on-read represent different trade-offs.

With copy-on-write, an update rewrites affected data files. Reads stay simpler because the latest table state is already materialized in columnar files, but frequent updates can create more write work.

With merge-on-read, new changes can be stored separately and merged with base files during reads or later compaction. Writes can stay more responsive for update-heavy or streaming workloads, but readers and maintenance processes carry more responsibility.

If you're still getting comfortable with the underlying files, this explanation of Parquet provides the lower-level context. Parquet stores records. The open table format explains how those records participate in a governed, versioned table.

What an Open Table Format Actually Buys You

The benefits are easier to evaluate as a matrix. Each capability solves a particular failure mode, but none of them guarantees that the business meaning of the data is correct.

ACID transactions are the foundation. Without them, teams often depend on a rename-and-pray pattern. A job writes into a temporary directory and hopes that a final rename will prevent readers from seeing an incomplete result. That approach becomes fragile when retries, multiple writers, object storage behavior, and independent query engines interact.

Time travel changes incident response. If a pipeline introduces bad values, an analyst can compare the current table with an earlier snapshot, rerun a query against the prior state, or restore a known-good version, provided the relevant metadata and files haven't been removed by retention procedures. The table's history becomes an operational artifact rather than an invisible sequence of file mutations.

Schema evolution addresses a different problem. Source systems change. A producer adds a field, renames a column, or changes the order of fields. A table format can record compatible structural changes without requiring every historical file to be rewritten immediately. That reduces migration friction, but the team still needs a contract for how downstream consumers interpret the change.

Performance improvements come from moving work from scan time to planning time. The engine can use partition metadata, file-level column statistics, and sort order to avoid reading irrelevant files. The result depends on how the table is written and maintained. Metadata can narrow the search, but it can't rescue a layout that creates excessive file churn or poor data locality.

The boundary matters:

A table can be transactionally correct and semantically wrong.

An atomic commit can preserve a complete batch containing incorrect revenue, duplicate customers, or an invalid status code. Open table formats provide structural consistency and historical state. Data validation, lineage, ownership, and business monitoring still require separate design.

Iceberg, Delta Lake, and Hudi Compared

Apache Iceberg, Delta Lake, and Apache Hudi are the three dominant open table format choices. They share the broad goal of bringing reliable table behavior to object storage, but their metadata models and workload priorities differ.

Iceberg's strength is its snapshot and manifest architecture. An engine can plan against metadata without listing every object-storage file, and hidden partitioning lets table administrators change physical organization without exposing every layout detail to SQL users.

Delta Lake uses a transaction log stored in the _delta_log/ directory. Operations appear as sequentially numbered JSON or Parquet files, and that log supports atomic commits, snapshot isolation, and reproducible table history. This comparison of Delta Lake, Iceberg, and Hudi explains the role of the log in Delta's design.

Hudi is built around record-level indexing and supports Copy On Write and Merge On Read, a combination that suits pipelines handling frequent upserts and deletes. AWS's open table format guidance describes those write modes and Hudi's focus on incremental processing and streaming CDC.

For a batch ETL workload, all three can be viable. For BI, the practical question is which query engines can read the table with the features your queries need, including pruning, snapshot reads, and schema behavior. For CDC streaming, write amplification, indexing, merge semantics, and compaction may matter more than a simple feature checklist.

Ecosystem support keeps changing across Spark, Trino, Flink, Snowflake, BigQuery, and other engines. That makes compatibility testing more valuable than assuming a logo on a support page guarantees identical behavior everywhere. Teams should validate writes, concurrent commits, schema changes, deletes, time travel, and failure recovery with their actual engines and catalogs. A Databricks data quality approach also needs to be evaluated alongside format selection, because a table's write protocol doesn't replace checks on the data it contains.

The practical choice is not “which format wins?” It's which write semantics, catalog model, maintenance process, and engine integrations match the pipeline you operate.

How Open Table Formats Fit Modern Data Platforms

A lakehouse usually has four functional layers. Object storage holds the files. The open table format manages table metadata and committed state. Processing and query engines read or write through compatible clients. Governance and observability systems inspect what happened and whether the resulting data is usable.

A typical flow starts when records land in S3 or ADLS. A writer creates or updates a table, publishes metadata, and registers the table with a catalog. Spark may transform the data, Flink may process a stream, Trino may serve interactive queries, and Snowflake, BigQuery, or Athena may provide additional consumption paths when their integrations support the table's protocol and features.

That separation makes one logical dataset available to several workloads. Analytics can query the table, a machine learning pipeline can derive features, and a streaming consumer can process fresh changes without forcing each team to maintain a separate copy. The trade-off is that every client must agree on table semantics, catalog access, supported features, and authorization behavior.

This is why the format is better understood as part of the lakehouse's control plane, not merely as a file-format choice. The control plane exposes signals that matter on an ordinary incident day:

• Commit patterns: Detect stalled writers, unusual commit frequency, or repeated failures.

• Metadata freshness: Identify tables whose catalog state or metadata updates lag behind expected delivery.

• Partition drift: Find changes in physical organization that alter query behavior.

• Schema events: Track added, removed, or modified columns before downstream consumers fail.

• Snapshot behavior: Relate data-quality incidents to the exact committed table state that readers consumed.

A platform such as digna can sit alongside this layer by monitoring metadata freshness, schema change events, timeliness, validation results, anomalies, and platform signals inside the customer's own environment. That approach uses table history as an operational signal source while keeping data-quality responsibility separate from the table protocol.

The distinction is useful. The format tells you which table state was committed. Observability tells you whether that state arrived on time, follows expected patterns, satisfies business rules, and remains safe for downstream use. More context on that relationship appears in how to maintain data quality in a lakehouse.

Limits and Trade-offs Most Articles Skip

An open table format solves the atomicity gap for a table. It doesn't turn an object-storage lake into a fully coordinated relational database.

The first limit is transaction scope. ACID guarantees remain table-scoped in the general model. If a pipeline updates a sales table and an inventory table, each table can commit safely while the overall business operation still becomes inconsistent between them. A multi-table transaction needs a coordinator or a platform feature designed for that purpose.

The second limit is semantic quality. A table may contain every expected file, a valid schema, and a clean snapshot while holding incorrect values. An open table format won't know that a refund is larger than its order, that a customer identifier violates a business rule, or that revenue suddenly reflects the wrong currency.

Operational work doesn't disappear

Table maintenance still needs engineering attention.

• Compaction: Merge-on-read layouts and update-heavy workloads may require compaction so readers don't repeatedly reconcile many change files.

• File sizing: Excessive small files increase planning and scanning overhead, even when metadata allows pruning.

• Partition tuning: A partition scheme that suited yesterday's access patterns may produce poor performance after workload or data distribution changes.

• Retention: Time travel depends on retaining the metadata and data files required by older snapshots. Cleanup policies must balance recovery needs with storage management.

• Concurrency: Multiple writers can conflict. The pipeline must handle retries, failed commits, and idempotency rather than assuming every write will succeed.

Governance also lives beyond the basic table format. Catalogs such as Unity Catalog, Glue, Polaris, and Nessie can manage discovery, permissions, lineage integrations, and coordination, but each introduces configuration, availability, compatibility, and upgrade responsibilities. Vendor differences around catalog specifications, REST interfaces, and hidden partitioning can complicate portability even when two systems claim support for the same underlying format.

Operational boundary: The format protects table state. Your platform still owns quality, lineage, access policy, incident response, and workload design.

These constraints aren't reasons to reject open table formats. They're the boundaries to document before production. A design that includes catalog operations, maintenance jobs, quality checks, monitoring, and rollback procedures will behave very differently from a design that treats the format as a drop-in replacement for a warehouse.

Choosing and Adopting an Open Table Format

Choose the format against the workload, not against a vendor slogan. Start with the engines that must read and write the table, then test the write patterns, catalog integration, failure behavior, maintenance path, and quality signals that matter in production.

A practical migration can fit into a 90-day plan without pretending that a format change is a single deployment.

Pilot

Select representative tables, including an append-heavy table, a table with schema changes, and a workload with updates or deletes. Measure query behavior, commit conflicts, metadata growth, recovery steps, and the effort required to validate records.

Dual-write validation

Write the same logical inputs through the legacy path and the new table path. Compare row counts, keys, null behavior, aggregates, schema events, delivery timing, and snapshot contents. Keep the comparison focused on business acceptance criteria, not only whether both systems produced files.

Cutover

Move one downstream workload at a time. Define ownership for the catalog, maintenance jobs, failed commits, rollback decisions, and alerts. Keep the legacy path available until the new path has demonstrated stable reads, writes, recovery, and monitoring under normal operating conditions.

Decommissioning

Remove redundant writers and storage paths only after retention, audit, lineage, and rollback requirements are documented. Archive the evidence needed to explain when the cutover occurred and which table snapshot became authoritative.

Common adoption mistakes are predictable: underestimating catalog operations, skipping partition-evolution tests, treating the format as a warehouse replacement, ignoring concurrent-writer conflicts, and neglecting rollback planning. The strongest selection process includes observability from the pilot onward, so teams can see not just whether a commit succeeded, but whether the resulting data arrived on time and remained correct for its consumers.

For broader architecture decisions, enterprise data platform guidance can help place table formats alongside governance, quality, and operational ownership.

digna helps enterprises monitor data quality, timeliness, schema changes, anomalies, and platform behavior inside their own infrastructure, which makes it a practical companion to an open table format's metadata and snapshot control plane. Visit digna to evaluate how those signals can support safer lakehouse operations.