Open Table Formats Compared: Iceberg, Delta, Hudi, Paimon

Apache Iceberg, Delta Lake, Hudi, Paimon and DuckLake compared: metadata design, row-level updates, engine support, and when to choose each table format.

Updated: at 12:00 PM
Alex Merced

Cross-posted. This article’s canonical home is Data Lakehouse Hub.

Get Data Lakehouse Books:

Join the Data Lakehouse Community Data Lakehouse Blog Roll

Reviewed October 15, 2026 against the Apache Iceberg spec, DuckLake docs and Delta Lake docs.

Modern lakehouse stacks live or die by how they manage tables on cheap, scalable object storage. That “how” is the job of open table formats, the layer that turns piles of Parquet/ORC files into reliable, ACID-compliant tables with schema evolution, time travel, and efficient query planning. If you’ve ever wrestled with brittle Hive tables, small-file explosions, or “append-only” lakes that can’t handle updates and deletes, you already know why this layer matters.

In this guide, we’ll demystify the five formats you’re most likely to encounter:

We’ll start beginner-friendly, clarifying what a table format is and why it’s essential, then progressively dive into expert-level topics: metadata internals (snapshots, logs, manifests, LSM levels), row-level change strategies (COW, MOR, delete vectors), performance trade-offs, ecosystem support (Spark, Flink, Trino/Presto, DuckDB, warehouses), and adoption trends you should factor into your roadmap.

By the end, you’ll have a practical mental model to choose the right format for your workloads, whether you’re optimizing petabyte-scale analytics, enabling near-real-time CDC, or simplifying your metadata layer for developer velocity.

Why Open Table Formats Exist

Before diving into each format, it’s worth understanding why open table formats became necessary in the first place.

Traditional data lakes, built on raw files like CSV, JSON, or Parquet, were cheap and scalable, but brittle. They had no concept of transactions, which meant if two jobs wrote data at the same time, you could easily end up with partial or corrupted results. Schema evolution was painful, renaming or reordering columns could break queries, and updating or deleting even a single row often meant rewriting entire partitions.

Meanwhile, enterprises still needed database-like features, updates, deletes, versioning, auditing, on their data lakes. That tension set the stage for open table formats. These formats layer metadata and transaction protocols on top of files to give the data lake the brains of a database while keeping its open, flexible nature.

In practice, open table formats deliver several critical capabilities:

In other words, table formats solve the “wild west of files” problem, turning data lakes into lakehouses that balance scalability with structure. The differences among Iceberg, Delta, Hudi, Paimon, and DuckLake lie in how they achieve this and what trade-offs they make to optimize for batch, streaming, or simplicity.

Next, we’ll walk through the history and evolution of each format to see how these ideas took shape.

The Evolution of Open Table Formats

The journey of open table formats reflects the challenges companies faced as data lakes scaled from terabytes to petabytes. Each format emerged to solve specific pain points:

These formats represent waves of innovation:

Next, we’ll dive into Apache Iceberg in detail, its metadata structure, features, and why it has become the default choice for many modern lakehouse deployments.

Apache Iceberg: The Batch-First Powerhouse

Background & Origins
Apache Iceberg was created at Netflix in 2017, entered the Apache Incubator in 2018, and graduated to a top-level Apache project in 2020. Its mission was clear: fix the long-standing problems of Hive tables, unreliable schema changes, expensive directory scans, and lack of true atomicity. Iceberg introduced a clean-slate design that scaled to petabytes while guaranteeing ACID transactions, schema evolution, and time-travel queries.

Metadata Structure
Iceberg’s metadata model is built on a hierarchy of files:

This design avoids reliance on directory listings, making planning queries over millions of files feasible.

Core Features

Row-Level Changes
Iceberg supports both copy-on-write and merge-on-read. Format v2 added delete files for merge-on-read, and format v3 replaced new position delete files with compact deletion vectors stored in Puffin files. Deletes can be tracked separately and applied at read time, reducing write amplification for frequent updates. Background compaction later consolidates these into optimized Parquet files.

Ecosystem & Adoption
Iceberg’s neutrality and technical strengths have driven broad adoption. It is supported in:

By late 2024, Iceberg had become the de facto industry standard for open table formats, with adoption by Netflix, Apple, LinkedIn, Adobe, and major cloud vendors. Its community-driven governance and rapid innovation ensure it continues to evolve, format v3 features like deletion vectors, row lineage and the variant type, plus the REST catalog spec, are making it even more capable.

Next, we’ll look at Delta Lake, the transaction-log-driven format that became the backbone of Databricks’ lakehouse vision.

Delta Lake: The Transaction-Log

Background & Origins
Delta Lake was introduced by Databricks around 2017-2018 to address Spark’s biggest gap: reliable transactions on cloud object storage. Open-sourced in 2019 under the Linux Foundation, Delta Lake became the backbone of Databricks’ lakehouse pitch, combining data warehouse reliability with the scalability of data lakes. Its design centered on a simple but powerful idea: use a transaction log to coordinate all changes.

Metadata Structure
At the core of every Delta table is the _delta_log directory:

This log-based design is simple and easy to reconstruct: replay JSON logs from the last checkpoint to reach the latest state.

Core Features

Row-Level Changes
Delta primarily uses copy-on-write: updates and deletes rewrite entire Parquet files while marking old ones as removed in the log. This guarantees atomicity but can be expensive at scale. To mitigate, Delta introduced deletion vectors (in newer releases), which track row deletions without rewriting whole files, closer to merge-on-read semantics. Upserts are supported via SQL MERGE INTO, commonly used for database change capture workloads.

Ecosystem & Adoption
Delta Lake is strongest in the Spark ecosystem and is the default format in Databricks. It’s also supported by:

While its openness has improved since Delta 2.0, much of its adoption remains tied to Databricks. Still, Delta Lake is one of the most widely used formats in production, powering pipelines at thousands of organizations.

Next, we’ll explore Apache Hudi, the pioneer of incremental processing and near-real-time data lake ingestion.

Apache Hudi: The Incremental Pioneer

Background & Origins
Apache Hudi (short for Hadoop Upserts Deletes and Incrementals) was created at Uber in 2016 to solve a pressing challenge: keeping Hive tables up to date with fresh, continuously changing data. Uber needed a way to ingest ride updates, user changes, and event streams into their Hadoop data lake without waiting hours for batch jobs. Open-sourced in 2017 and donated to Apache in 2019, Hudi became the first widely adopted table format to support row-level upserts and deletes directly on data lakes.

Metadata Structure
Hudi organizes tables around a commit timeline stored in a .hoodie directory:

This dual-mode design gives engineers control over the trade-off between write latency and read latency.

Core Features

Row-Level Changes
Hudi was designed for this problem. In COW mode, updates rewrite files. In MOR mode, updates are appended as log blocks, making them queryable almost immediately. Readers can choose:

Deletes are handled similarly, either as soft deletes in logs or hard deletes during compaction.

Ecosystem & Adoption
Hudi integrates tightly with:

While Iceberg and Delta now dominate conversations, Hudi remains a strong choice for near real-time ingestion and CDC use cases, particularly in AWS-centric stacks. Its flexibility (COW vs MOR) and incremental consumption features make it especially valuable for pipelines that need fast data freshness without sacrificing reliability.

Next, we’ll examine Apache Paimon, the streaming-first format that extends Hudi’s incremental vision with an LSM-tree architecture.

Apache Paimon: Streaming-First by Design

Background & Origins
Apache Paimon began life as Flink Table Store at Alibaba in 2022, targeting the need for continuous, real-time data ingestion directly into data lakes. It entered the Apache Incubator in 2023 under the name Paimon and graduated to a top-level Apache project in 2024. Unlike Iceberg or Delta, which started with batch analytics and later added streaming features, Paimon was streaming-first. Its mission: make data lakes act like a materialized view that is always up to date.

Metadata & Architecture
Paimon uses a Log-Structured Merge-tree (LSM) design inspired by database internals:

This architecture makes frequent row-level changes cheap (append-only writes) while deferring heavy merges to compaction tasks.

Core Features

Row-Level Changes
Unlike Iceberg (COW with delete files) or Delta (COW with deletion vectors), Paimon is natively merge-on-read. Updates and deletes are appended as small log segments, queryable immediately. Background compaction gradually merges them into optimized columnar files. This makes Paimon highly efficient for high-velocity workloads like IoT streams, CDC pipelines, or real-time leaderboards.

Ecosystem & Adoption
Paimon integrates tightly with Apache Flink, where it feels like a natural extension of Flink SQL. It also has growing support for Spark, Hive, Trino/Presto, and OLAP systems like StarRocks and Doris. Adoption is strongest among teams building streaming lakehouses, particularly those already invested in Flink. While younger than Iceberg or Delta, Paimon is rapidly attracting attention as organizations push for sub-minute data freshness.

Next, we’ll turn to DuckLake, the newest entrant that rethinks table metadata management by moving it entirely into SQL databases.

DuckLake: Metadata Reimagined with SQL

Background & Origins
DuckLake is the newest table format, introduced in 2025 by the DuckDB team, and it has since reached a 1.0 stable release. Unlike earlier formats that manage metadata with JSON logs or Avro manifests, DuckLake flips the script: it stores all table metadata in a relational SQL database. This approach is inspired by how cloud warehouses like Snowflake and BigQuery already manage metadata internally, but DuckLake makes it open and interoperable.

Metadata & Architecture

This design dramatically reduces the complexity of planning queries (no manifest scanning), makes commits faster, and enables features like cross-table consistency (possible in Apache Iceberg only through catalogs that implement the REST catalog’s multi-table commit endpoint).

Core Features

Row-Level Changes
DuckLake handles deletes with merge-on-read: a delete writes a small delete file that points at the removed rows in the original Parquet file, and the metadata transaction is nearly instantaneous. A maintenance function, ducklake_rewrite_data_files, rewrites heavily deleted files later. Row-level changes are coordinated by the SQL catalog, avoiding the latency and eventual consistency pitfalls of cloud storage-based logs. In effect, DuckLake behaves like Iceberg for data files but with much faster commit cycles.

Ecosystem & Adoption

Even at 1.0, DuckLake is still young but has sparked excitement by simplifying lakehouse architecture. It’s best seen as a complement to more mature formats, with particular appeal to DuckDB users and teams tired of managing complex metadata stacks.

Next, we’ll step back and compare all five formats side by side, looking at metadata design, row-level update strategies, ecosystem support, and adoption trends.

Comparing the Open Table Formats

Now that we’ve walked through each format individually, let’s compare them across the dimensions that matter most to data engineers and architects.

1. Metadata Architecture

2. Row-Level Changes

3. Ecosystem Support

4. Adoption Trends

Next, we’ll step back and examine industry trends shaping the adoption of these formats and what they signal for the future of the lakehouse ecosystem.

The “table format wars” of the past few years are starting to settle into clear patterns of adoption. While no single format dominates every use case, the industry is coalescing around certain choices based on scale, latency, and ecosystem needs.

Iceberg as the Default Standard
Iceberg has emerged as the most widely supported and vendor-neutral choice. Cloud providers like AWS, Google, and Snowflake have all added native support, and query engines like Trino, Presto, Hive, and Flink integrate with it out-of-the-box. Its Apache governance and cross-engine compatibility make it the safe long-term bet for enterprises standardizing on a single open format.

Delta Lake in the Spark/Databricks World
Delta Lake remains the default in Spark- and Databricks-heavy shops. Its simplicity (transaction logs) and seamless batch/stream integration continue to attract teams already invested in Spark. While its ecosystem is narrower than Iceberg’s, Delta Lake’s deep integration with Databricks runtime and machine learning workflows ensures strong adoption in that ecosystem.

Hudi in CDC and Incremental Ingestion
Hudi carved out a niche in change data capture (CDC) and near real-time ingestion. Telecom, fintech, and e-commerce companies still rely on Hudi for incremental pipelines, especially on AWS where Glue and EMR make it easy to deploy. While Iceberg and Delta have added incremental features, Hudi’s head start and MOR tables keep it relevant for low-latency ingestion scenarios.

Paimon and the Rise of Streaming Lakehouses
As real-time analytics demand grows, Paimon is gaining momentum in the Flink community and among companies building streaming-first pipelines. Its LSM-tree design positions it as the go-to choice for high-velocity data, IoT streams, and CDC-heavy architectures. Although young, its momentum signals a broader shift: the next wave of lakehouse innovation is about sub-minute freshness.

DuckLake and Metadata Simplification
DuckLake reflects a newer trend: rethinking metadata management. By moving metadata into SQL databases, it dramatically simplifies operations and enables cross-table transactions. Adoption is still experimental, but DuckLake has sparked interest among teams who want lakehouse features without managing complex catalogs or metastores. Its trajectory will likely influence how future formats handle metadata.

Convergence and Interoperability
One notable trend: features are converging. Iceberg supports row-level deletes via delete files and, in v3, deletion vectors; Delta added deletion vectors; Hudi and Paimon both emphasize streaming upserts. Tooling is also evolving toward interoperability, catalog services like Apache Polaris and Unity Catalog aim to serve more than one format, and BI engines increasingly connect to all.

In short:

Next, we’ll wrap up with guidance on how to choose the right format based on your workloads, ecosystem, and data engineering priorities.

Choosing the Right Open Table Format

With five strong options on the table, Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, the choice depends less on “which is best” and more on which aligns with your workloads, ecosystem, and priorities. Here’s how to think about it:

When to Choose Apache Iceberg

When to Choose Delta Lake

When to Choose Apache Hudi

When to Choose Apache Paimon

When to Choose DuckLake

Final Takeaway

No matter which you choose, adopting an open table format is the key to turning your data lake into a true lakehouse: reliable, flexible, and future-proof.

Frequently Asked Questions

What is an open table format?

An open table format is a metadata layer over data files such as Parquet that turns a folder of files into a table with ACID transactions, schema evolution, time travel and file pruning. Apache Iceberg, Delta Lake, Apache Hudi, Apache Paimon and DuckLake are the main options.

What is the difference between Iceberg, Delta Lake and Hudi?

Iceberg tracks table state with snapshots, manifest lists and manifests, Delta Lake with a sequential transaction log in the _delta_log folder, and Hudi with a commit timeline in the .hoodie folder. Iceberg has the widest engine and vendor support, Delta is closest to Spark and Databricks, and Hudi is built around key-based upserts and incremental pulls.

Which open table format has the broadest support?

Apache Iceberg. Spark, Flink, Trino, Presto, Dremio, Snowflake, BigQuery, Athena, DuckDB and most other engines can read and write it, and catalogs such as Apache Polaris, AWS Glue and Unity Catalog serve it through the Iceberg REST catalog API.

When should I use Apache Paimon instead of Iceberg?

Choose Paimon when the workload is streaming-first, such as high-volume CDC with constant upserts on primary keys, especially if you already run Apache Flink. Its LSM-tree design makes frequent row changes cheap, while Iceberg is the stronger default for batch analytics and multi-engine access.

How is DuckLake different from other table formats?

DuckLake stores all table metadata in a SQL database (DuckDB, SQLite or PostgreSQL) instead of JSON and Avro files on object storage. That makes commits and query planning fast and allows multi-table transactions, while data still lives in Parquet files.

Conclusion

Open table formats are no longer niche, they’re the foundation of the modern data stack. Whether your challenge is batch analytics, real-time ingestion, or simplifying metadata, there’s a format designed to meet your needs. The smart path forward isn’t just picking one blindly, but aligning your choice with your data velocity, tooling ecosystem, and long-term governance strategy.

In practice, many organizations run more than one format side by side. The good news: as open standards mature, interoperability and ecosystem support are expanding, making it easier to evolve over time without locking yourself into a dead end.

The lakehouse era is here, and open table formats are its backbone.

Get Data Lakehouse Books:

Join the Data Lakehouse Community Data Lakehouse Blog Roll

Work with Alex