Apache Polaris and Multi-Engine Iceberg Catalogs

Apache Iceberg solved a hard problem: it gave analytics teams a table format that multiple engines can read and write without corrupting each other’s view of the data. Snapshot isolation, schema evolution, hidden partitioning, and time travel all live in the table format itself. What Iceberg did not fully solve on its own is the question one layer up. How does an engine find a table in the first place? How does it know which snapshot is current, commit a new one safely, or organize tables into namespaces? That job belongs to a catalog, and for years the catalog is where the openness of a lakehouse quietly leaked away. You could have perfectly open Iceberg files sitting in object storage and still be locked into whichever engine controlled the catalog that pointed at them.

Apache Polaris exists to close that gap. It is an open source implementation of an Iceberg REST catalog, and its significance is not that it is one more catalog option. It is that it makes catalog interoperability an implementable pattern rather than a diagram on a slide. What follows covers why open table formats need open catalogs, what Polaris actually does at the protocol level, how role-based access control works when several engines share one catalog, why a shared catalog is the natural spine for a multi-cloud lakehouse, and what you should check before you standardize on it. The through-line: the open lakehouse needs an open metadata plane, and Polaris turns that requirement into working software.

Why Open Table Formats Need Open Catalogs

There is a common misconception that adopting Iceberg makes your data open, full stop. Open files are necessary but not sufficient. Openness has two independent dimensions, and confusing them causes teams real pain later.

The first dimension is table format openness. Iceberg data and metadata files sit in your object storage in a documented, engine-neutral format. Any engine that speaks Iceberg can read them. That is genuine openness, and it is valuable.

The second dimension is catalog openness. To use those files, an engine has to resolve a table name to a metadata location, learn the current snapshot, and commit changes atomically. That is the catalog’s job. If the catalog only speaks a proprietary dialect that one vendor controls, then your data files are open but your access to them is not. You cannot point a second engine at the same tables without either replicating catalog state or routing everything through the one engine that owns the metadata. In practice that means recreating table definitions, permissions, and lineage in every tool, which is exactly the fragmentation the lakehouse was supposed to end.

So the real question for a multi-engine environment is not “are my files open” but “can any engine I choose discover and safely modify my tables through a standard protocol.” Without a shared metadata plane, every engine becomes an island. Spark has its view, Flink has its view, Trino has its view, and keeping them consistent turns into a synchronization project that never quite finishes. A shared open catalog is what lets Spark, Flink, Trino, Dremio, Snowflake, and other engines all agree on what tables exist, what state they are in, and who is allowed to touch them.

The cost of getting this wrong is concrete, not theoretical. Teams that skip the shared catalog end up recreating table definitions in each tool, which means the same table can be registered with slightly different schemas or partition specs depending on which engine last touched it. Permissions get redefined per tool, so a column that is masked in one engine is exposed in another. Lineage fragments, because each tool only knows about the operations it performed, and no single system has the full picture of how a dataset was produced and consumed across engines. Every one of these is a maintenance burden that grows with the number of tools and the number of tables, and none of them are solved by the table format being open. They are catalog problems, and they need a catalog answer.

Polaris Project Status and Milestones

A note on status, because accuracy matters more than momentum here. Apache Polaris entered the Apache Software Foundation through the incubator after Snowflake originally open-sourced it. There has been public discussion of Polaris progressing toward top-level maturity within the ASF. Rather than assert a specific graduation date or claim it has already reached top-level status, the responsible framing is that Polaris has been on the path toward that maturity. If you need the authoritative current state, check the Apache Incubator projects list and the Polaris project site directly, since those reflect whatever the foundation has formally recorded at the time you read this.

What matters more than a graduation badge is the trajectory the project sits on. Contributor activity, release cadence, and adoption signals are the metrics worth watching, and those too are best pulled fresh from the Polaris GitHub repository rather than quoted from a stale number. The strategically important fact does not depend on any single milestone: an open catalog implementation is being developed in the open under a neutral foundation, aligned with the Iceberg REST catalog standard, with contribution from more than one vendor. That governance model is the point. A catalog controlled by a foundation with multiple contributing companies is structurally harder to turn into a lock-in mechanism than a catalog controlled by a single vendor, regardless of where it sits in the incubation process on any given day.

On ownership, one clarification. Snowflake open-sourced the original code, but the interesting story is interoperability, not lineage. An open catalog that only worked well with the engine that donated it would not be open in any meaningful sense. The value is in the standard it implements and the range of engines that can speak to it.

How Polaris Implements REST Catalog Interoperability

The mechanism that makes Polaris interoperable is the Iceberg REST catalog specification. Instead of each engine talking to a catalog through an engine-specific library with its own semantics, engines talk to the catalog over a standard HTTP API. The REST catalog spec defines the operations, and any compliant client can use any compliant server.

The core operations are straightforward to describe. A client can list namespaces and tables to discover what exists. It can load a table, which returns the current metadata location and the information needed to read it. It can create tables and namespaces. And critically, it can commit changes to a table through an atomic operation that the catalog serializes, so two engines committing concurrently do not clobber each other. The catalog is the arbiter of what “current” means for every table, and every engine defers to it.

Here is roughly what an engine’s interaction looks like, in shape if not exact syntax:

GET  /v1/namespaces
GET  /v1/namespaces/analytics/tables
POST /v1/namespaces/analytics/tables/orders/load
POST /v1/namespaces/analytics/tables/orders/commit
     body: { requirements: [assert-current-snapshot], updates: [add-snapshot, set-current-snapshot] }

Because this is a standard protocol, compatibility is a property of compliance, not of vendor partnership. If Spark’s Iceberg REST client and Dremio’s Iceberg REST client both speak the spec, and Polaris implements the spec, then all three interoperate without anyone writing bespoke connectors. That is the difference between openness as a marketing claim and openness as a testable behavior. You can point a compliant client at a compliant server and it works, or it does not, and the spec tells you which.

This also reframes what “adding a new engine” costs. In a proprietary-catalog world, adding an engine means building or buying a connector to that catalog’s specific dialect. In a REST-catalog world, adding an engine means checking that the engine ships a compliant REST catalog client, which increasingly it does. The integration burden shifts from custom work per engine to conformance to a shared standard.

The commit path deserves a closer look because it is where multi-engine correctness is won or lost. When two engines try to commit to the same table at the same time, one of them has to lose safely, and the loser has to know it lost so it can retry against the new current state rather than silently overwriting the winner’s work. The REST spec handles this with optimistic concurrency. A commit request carries requirements, assertions about the table’s current state, such as “the current snapshot is still the one I read.” The catalog checks those requirements atomically at commit time. If they hold, the commit applies and the table advances to a new snapshot. If they do not, because another engine committed in between, the catalog rejects the commit, and the client retries against the updated state. This is the same optimistic-concurrency discipline that databases use for conflict-free concurrent writes, lifted up to the catalog layer so that engines with no knowledge of each other can still write to shared tables without corruption. Getting this right in a single catalog implementation is exactly why a shared open catalog is worth more than each engine coordinating on its own. The catalog is the one place that can serialize commits authoritatively, and every engine trusting that one arbiter is what keeps a multi-engine table consistent.

Contrast this with the older Hive Metastore world, where catalog operations were not designed around this kind of atomic, spec-defined commit protocol, and where different engines layered their own conventions on top. The REST catalog is a deliberate reset: a single, versioned, documented protocol that defines these operations once so every compliant client behaves the same way. That is the substance behind the word “interoperability.” It is not a vibe. It is a specification with defined request and response shapes and defined concurrency semantics.

Delegated RBAC Across Spark, Flink, Trino, and Dremio

Interoperability at the read-and-commit level is only half the story. The harder half is access control. If five engines can all reach the same tables, then “who is allowed to do what” has to be answered consistently, or you have interoperability without governance, which is a security problem dressed up as a feature.

Consistency across engines is not automatic, and it is worth understanding why it is hard before crediting a catalog for solving it. Every engine arrives with its own assumptions. Spark authenticates one way, Trino another, and a cloud warehouse a third. If each engine also enforced its own access rules, then a permission would mean one thing in Spark and something subtly different in Trino, and the gaps between those interpretations are where data leaks. The only way to get consistency is to move the authorization decision out of the engines and into a shared place they all consult. That shared place is the catalog.

The subtlety here is that there are two distinct authorization decisions, and conflating them is a frequent mistake. The first is catalog authorization: is this identity allowed to read this table’s metadata, or commit a change to it, or create a table in this namespace? The second is storage authorization: is this identity allowed to actually read or write the underlying data files in object storage? These are different questions with different enforcement points. Catalog authorization is enforced by the catalog. Storage authorization is enforced by the cloud storage layer, which knows nothing about tables and only understands objects and credentials.

A Polaris-style catalog centralizes catalog authorization. It holds role-based access control rules about namespaces and tables, and it enforces them consistently no matter which engine asks. That consistency is exactly what a multi-engine environment needs, because the alternative is defining permissions separately in Spark, in Trino, and in every other engine, and hoping they stay in sync. They will not stay in sync. Centralizing catalog RBAC means a role defined once applies everywhere.

The delegation challenge is real and worth being honest about. Each engine may have its own identity model, its own session model, and its own way of obtaining storage credentials. A REST catalog can issue scoped, short-lived storage credentials as part of a table-load response, so an engine gets exactly the access it needs for exactly the tables it is authorized for, without every engine holding broad standing credentials to the whole bucket. That pattern, catalog-mediated credential vending, is how you get consistent authorization across engines that otherwise do not share an identity system. It is not automatic and it is not free. It requires the catalog and the storage layer to be configured to trust each other correctly, and misconfiguration there is a genuine risk. But the architecture is sound: the catalog decides who can touch what, and it hands out narrowly scoped storage access to match.

Polaris as a Multi-Cloud Metadata Plane

Zoom out from a single cluster and the argument for a shared open catalog gets stronger. A modern lakehouse rarely lives in one place. Data sits in AWS S3, in Azure Data Lake Storage, in Google Cloud Storage, in on-premises object stores, and alongside existing warehouses that are not going anywhere soon. Compute is just as scattered: a Spark cluster for heavy transformation, Flink for streaming, Trino or Dremio for interactive queries, each possibly in a different environment.

In that reality, forcing every workload into a single compute engine is not viable, and forcing every dataset into a single cloud is expensive and slow. What you actually want is a metadata plane that spans the whole estate: one place that knows what tables exist, where their files live, and who can access them, independent of which engine runs the query or which cloud holds the bytes. That is what an open REST catalog provides. Engines coordinate table state through the catalog while executing compute independently, wherever they happen to run.

This maps cleanly onto the query-in-place philosophy. If your catalog is open and your table format is open, you do not need to move data to a specific engine to make it usable. The engine comes to the data. You avoid the copy-and-sync tax that closed architectures impose, where using a second tool means duplicating datasets and then fighting to keep the copies consistent. A unified metadata plane lets teams manage table state once and query it from many engines across many clouds, which is precisely the shape a distributed lakehouse needs.

It also changes the economics of vendor choice over time. When the catalog is proprietary, the cost of leaving a vendor includes rebuilding the entire metadata plane: re-cataloging every table, re-authoring every permission, re-establishing every lineage relationship in the new tool’s model. That switching cost is itself the lock-in, and it grows with every table you add, which is why closed catalogs get stickier the longer you use them. An open catalog inverts this. Because the catalog speaks a standard protocol, the metadata plane is portable. You can change which engines you use, or even move the catalog itself between compliant implementations, without rebuilding your table definitions from scratch. The lock-in that lived in the catalog dissolves, and what you are left with is a choice of engines made on their merits rather than on the sunk cost of the metadata you already loaded into one of them. That is the strategic reason open catalogs matter, distinct from the technical reason that they enable multi-engine access.

The table below contrasts the two operating models.

What to Watch Before Standardizing

Choosing an open catalog is a good direction, and it is not a decision to make on enthusiasm alone. A few things deserve scrutiny before you build critical workloads on top of any catalog, Polaris included.

Maturity and operations come first. A catalog is on the critical path for every query and every commit. If it is down, your lakehouse is effectively down. So the operational questions are the important ones: what is the high-availability story, how does it handle catalog-level backup and recovery, how does it scale as table and commit counts grow, and how battle-tested is it under production load. A project moving through the incubator is exactly the kind of software where these answers are still firming up, which is a reason to pilot carefully rather than a reason to avoid it.

Migration is the second thing to plan for. Moving from an existing catalog, whether a Hive Metastore, a cloud-managed catalog, or a proprietary vendor catalog, is not instantaneous. Table pointers have to be reregistered, permissions have to be rebuilt in the new RBAC model, and you need a cutover plan that does not leave two catalogs disagreeing about the current snapshot of a table. Test the migration path on non-critical tables first.

Engine support gaps are the third. REST catalog compliance is a spectrum, not a binary. Most major engines have Iceberg REST clients, and coverage of specific features, such as credential vending or particular commit semantics, can vary by engine and version. Before you commit, verify that every engine you actually use interoperates with the catalog for the operations you actually need, at the versions you actually run. The spec is the contract, but implementations lag the spec at their own pace.

One more consideration sits underneath all three: the difference between a self-managed catalog and a managed one. Running Polaris yourself gives you full control and full operational responsibility, including the high-availability and backup work above. Consuming an open catalog through a managed service shifts that operational burden to the provider while preserving the open protocol, so you keep the interoperability and portability benefits without owning the uptime. Which path fits depends on your team’s appetite for operating infrastructure. The important point is that because the protocol is open, this is a genuine choice rather than a lock-in, and you can change your mind later without re-cataloging everything.

None of this argues against open catalogs. It argues for treating the catalog as the piece of critical infrastructure it is, and validating it the way you would validate any system you cannot afford to have fail.

Why This Favors an Open Agentic Lakehouse

Step back and the market signal is clear. The industry is moving away from any single engine owning table metadata and toward open, multi-engine metadata planes. That shift favors architectures built on open standards and disfavors closed warehouse catalogs, because a closed catalog is a lock-in point no matter how open the file format underneath it is.

This is the direction Dremio has built toward. Dremio is a unified lakehouse on open standards, Apache Iceberg, Apache Arrow, and Apache Polaris, with an Open Catalog built on Polaris itself. Its federated query engine queries data in place across sources without forcing data movement, which is the query-in-place philosophy that a shared metadata plane makes possible. Because the catalog and table format are open, the data stays portable: you are not trading one vendor’s warehouse lock-in for another’s. Fine-grained access control handles row and column security consistently, matching the centralized-authorization argument that makes multi-engine catalogs work in the first place. For a deeper look at how the catalog anchors this architecture, Dremio’s writeup on its Open Catalog architecture is worth reading.

The honest positioning is that Polaris and open catalogs are an industry direction Dremio participates in, not a proprietary advantage anyone owns. That is the whole point. An open metadata plane is valuable precisely because no single vendor controls it. What Dremio adds on top is a data platform designed so that governed, fast, multi-engine access is the default rather than something you assemble by hand, which is exactly what makes a lakehouse ready for AI agents that need to query broadly and safely.

If you want to work with an Iceberg lakehouse on an open catalog without stitching the pieces together yourself, try Dremio Cloud free for 30 days and connect your Iceberg tables to see open, multi-engine access in practice.