Open Table Formats Explained
An open table format is a specification that defines how a table stored in object storage tracks its own state. It answers the questions that a raw directory of Parquet files cannot: which files are currently part of this table, what is the schema, what changed in the last write, is this write complete?
The Four Main Formats
Apache Iceberg
Iceberg was created at Netflix to handle petabyte-scale tables where Hive-style folder semantics caused reliability problems. Its defining design is theimmutable snapshot tree: every commit produces a new snapshot that references a complete manifest list. No log replay is needed to find current state. The REST Catalog specification is Iceberg's other key contribution — an open HTTP API any catalog can implement and any engine can call.
Delta Lake
Delta Lake takes a transaction-log approach. All commits are recorded as sequential JSON files in a _delta_log/ directory. Readers replay the log from the last Parquet checkpoint forward. Works very well within the Spark ecosystem. Delta's UniForm feature (Delta 3.x+) auto-generates Iceberg metadata alongside Delta metadata, allowing external Iceberg readers to access Delta tables read-only.
Apache Hudi
Hudi was created at Uber to efficiently update ride-sharing records by primary key without rewriting entire partitions. It maintains a timeline in .hoodie/and builds per-record indexes (bloom filter, bucket, HBase) that let writers find the exact files containing a given record key. Hudi's native incremental query returns exactly which record keys changed between two points in time.
Apache Paimon
Paimon emerged from the Flink ecosystem. Its distinguishing feature is anLSM-tree architecture (the same structure that powers RocksDB), making it extremely efficient for streaming writes and key-based point lookups. Best format for sub-second streaming ingestion with efficient lookup joins.
How a Table Format Sits in Your Architecture
Shared Properties All Four Formats Provide
| Property | What it means in practice |
|---|---|
| ACID transactions | Writers either fully commit or fully fail. No partial writes visible to readers. |
| Schema evolution | Add, rename, reorder, or drop columns without rewriting data files. |
| Time travel | Query any past version of the table by timestamp or version identifier. |
| Row-level deletes | DELETE and UPDATE statements that affect individual rows, not whole partitions. |
| Concurrent write safety | Multiple writers can operate without corrupting each other. |
Key Differences
| Dimension | Iceberg | Delta Lake | Hudi | Paimon |
|---|---|---|---|---|
| Metadata model | Immutable snapshot tree | Sequential transaction log | Timeline + record index | LSM-tree + manifest |
| Multi-engine writes | Excellent (REST Catalog) | Good (Spark-primary) | Good (Spark + Flink) | Good (Flink-primary) |
| Open catalog spec | Yes (REST Catalog) | No (Unity proprietary) | No | No |
| Key-based upserts | Via MoR (no native index) | Via MERGE INTO | Native (built-in index) | Native (LSM) |
| Python client | PyIceberg (mature) | delta-rs (Rust-based) | Limited | Limited |
| Cloud-managed service | S3 Tables, BigLake, Azure Fabric | Databricks (Unity Catalog) | None major | None yet |