Polaris vs Unity Catalog vs Glue: Pick an Iceberg Catalog

Originally published on IcebergLakehouse.com by Alex Merced.

Use Apache Polaris for open multi-engine Iceberg, Unity Catalog for mixed Delta and Iceberg, or AWS Glue for an AWS-only stack. Features and tradeoffs compared.

Updated: at 12:00 PM
Alex Merced

Reviewed October 15, 2026 against the Apache Polaris docs, Unity Catalog releases, Snowflake Horizon Catalog for external engines, and the AWS Glue Iceberg REST endpoint.

Modern data architecture is undergoing a quiet but fundamental shift. For years, teams focused on choosing the right open table format, debating the file-level mechanics of Delta Lake versus Apache Iceberg. Today, that format debate is largely settled by metadata interoperability.

Databricks launched managed Iceberg tables in Unity Catalog in June 2025 and has since made them generally available, Google Cloud serves Iceberg tables to open engines through its Lakehouse runtime catalog (formerly BigLake metastore), and Snowflake’s Horizon Catalog now exposes an Iceberg REST endpoint that external engines can read and write. The real battleground has moved up the stack from file formats to the metadata control plane.

When multiple compute engines, such as Apache Spark for batch ETL, Apache Flink for real-time streaming, Trino for ad-hoc SQL, and Snowflake or BigQuery for enterprise BI, need to read and write to the same shared files simultaneously, they cannot rely on local file structures. They need a centralized authority to coordinate table updates, track snapshots, and enforce security policies.

This centralized coordinator is the Apache Iceberg catalog. Selecting the right catalog control plane is now the most critical design decision in lakehouse engineering, dictating your platform’s security boundaries, cloud costs, and multi-engine interoperability.

To understand why this layer has become so strategic, you must look at how the data lakehouse has evolved. The first wave of lakehouse design separated compute from storage. You stored your data as Parquet files in an open S3 bucket and spun up compute engines dynamically to run queries.

However, this model created a metadata vacuum. Because S3 has no native understanding of schemas, transaction records, or table partitions, every engine had to maintain its own list of what files made up a table. The second wave of the lakehouse is about separating metadata from compute and storage. The catalog control plane is that decoupled metadata layer.

Architecture diagram showing multiple engines like Spark, Flink, Trino, and Snowflake communicating with a unified Iceberg REST Catalog control plane which delegates to S3 storage

Decoupled Interoperability: The Iceberg REST Catalog Standard

In early data lake designs, engines interacted directly with physical storage catalogs. The query engine read metadata files directly from S3 or polled a Hive Metastore database to discover which Parquet files belonged to a table.

This tightly coupled design introduced several structural problems. Every engine had to implement its own lock management and metadata parsing logic, which frequently led to commit collisions, read-write drift, and vendor lock-in.

The Apache Iceberg REST Catalog specification decouples the compute engine from metadata management. It defines a standardized, language-agnostic OpenAPI specification for catalog operations.

Under this model, the query engine never inspects raw storage directories to determine table states. Instead, it sends standard HTTP requests to a REST Catalog endpoint:

Query Engine (Spark/Trino) ──► GET /v1/namespaces/db/tables/events ──► REST Catalog Server
Query Engine (Spark/Trino) ◄── [JSON Table Metadata & Storage Token] ◄── REST Catalog Server

The REST Catalog server handles the request, interacts with the underlying metadata database, and returns a JSON payload containing the table’s schema, partition specifications, current snapshot details, and temporary security credentials to access the data files.

This API-first design provides several platform benefits. It standardizes catalog operations across diverse compute engines, allowing a Spark write commit and a Trino read request to use the exact same catalog interface.

It centralizes transaction management, allowing the REST server to handle commit conflicts and enforce Optimistic Concurrency Control (OCC) without relying on engine-specific file locking.

Finally, it secures storage access through credential vending. The catalog server issues temporary, scoped access tokens (such as AWS STS tokens) to the compute engine for a specific table path, avoiding the security risk of sharing root-level storage credentials with query clients.

Additionally, standardizing the JSON response schema for table metadata ensures that secondary metadata details (such as column statistics, sort orders, and partition specs) are interpreted identically by all reading engines.

Without this standard interface, differences in how engines (like Trino versus Spark) parsed physical metadata files often led to query plan mismatches. The REST catalog eliminates this discrepancy by acting as the single, authoritative interpreter of table state.

A standard Iceberg REST Catalog response for a table request contains structured sections detailing the table state:

{
  "metadata-location": "s3://my-lakehouse/metadata/00001-abc.metadata.json",
  "metadata": {
    "format-version": 2,
    "table-uuid": "a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d",
    "location": "s3://my-lakehouse/data",
    "last-updated-ms": 1779629242000,
    "last-column-id": 3,
    "schemas": [
      {
        "type": "struct",
        "fields": [
          { "id": 1, "name": "id", "required": true, "type": "int" },
          { "id": 2, "name": "event_date", "required": true, "type": "date" },
          {
            "id": 3,
            "name": "metric_value",
            "required": false,
            "type": "double"
          }
        ]
      }
    ],
    "current-schema-id": 0,
    "partition-specs": [
      {
        "spec-id": 0,
        "fields": [
          {
            "source-id": 2,
            "field-id": 1000,
            "name": "event_date",
            "transform": "identity"
          }
        ]
      }
    ],
    "default-spec-id": 0,
    "last-partition-id": 1000,
    "snapshots": [
      {
        "snapshot-id": 987654321,
        "timestamp-ms": 1779629242000,
        "summary": {
          "operation": "append",
          "added-data-files": "4"
        },
        "manifest-list": "s3://my-lakehouse/metadata/snap-987654321.manifest.list.avro"
      }
    ],
    "current-snapshot-id": 987654321
  },
  "storage-credentials": [
    {
      "prefix": "s3://my-lakehouse/data",
      "config": {
        "s3.access-key-id": "<temporary key id>",
        "s3.secret-access-key": "<temporary secret>",
        "s3.session-token": "<temporary session token>"
      }
    }
  ]
}

This clean structure isolates the reading client from having to query physical directories or parse raw metadata files. The catalog server performs the lookups and serves the exact logical plan foundations in a unified JSON response.

Independent and Open: Apache Polaris Architecture

Apache Polaris is an open-source, vendor-neutral metadata control plane designed for Apache Iceberg tables. Polaris graduated to a Top-Level Project (TLP) at the Apache Software Foundation (ASF) in February 2026, and its current release is 1.8.0 (September 2026). This independent status means Polaris operates under community-driven governance rather than a single vendor’s roadmap. Managed services build on it as well: Dremio Catalog is built on Apache Polaris.

Architecturally, Polaris acts as a stateless REST Catalog server that communicates with a backend metadata store (PostgreSQL or CockroachDB through its relational JDBC backend, or MongoDB through its NoSQL backend). Polaris provides a unified namespace where you can manage Iceberg tables and register external catalog sources.

Polaris implements a zero-trust security model centered on temporary credential vending. When a client engine queries a table, Polaris does not share raw IAM keys. Instead, the Polaris server negotiates temporary tokens (such as AWS STS scoped sessions or Google Cloud Service Account impersonations) that allow the engine to access only the specific storage path linked to that table.

The v1.4 release of Apache Polaris (April 2026) introduced several updates designed for production deployments:

To configure an Apache Spark session to connect to a Polaris server using the standard Iceberg REST Catalog API, you define the catalog properties in your configuration file or code:

from pyspark.sql import SparkSession

# Initialize Spark Session with Apache Polaris REST Catalog configuration
spark = SparkSession.builder \
    .appName("PolarisCatalogConnection") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.polaris", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.polaris.type", "rest") \
    .config("spark.sql.catalog.polaris.uri", "http://polaris-server.data-platform.local:8181/api/catalog") \
    .config("spark.sql.catalog.polaris.credential", "client_id_123:client_secret_xyz") \
    .config("spark.sql.catalog.polaris.warehouse", "my_s3_warehouse") \
    .getOrCreate()

# Query an Iceberg table managed by Polaris
df = spark.read.table("polaris.db.sales_records")
df.show()

Releases since 1.4 have continued on the same track through 1.8.0, adding features such as semantic model privileges, more federation options, and safer commit error handling. This open architecture makes Polaris a strong default for organizations building multi-engine, multi-cloud platforms on open-source technologies.

Managed by Snowflake: Horizon Catalog and Open Catalog

For teams that want the interoperability of Apache Polaris without running a metadata server themselves, Snowflake offers two managed options, and the recommended one has changed.

Snowflake Open Catalog is Snowflake’s managed service for Apache Polaris. Existing Open Catalog customers can keep using it, but Snowflake’s docs state that customers without an Open Catalog account can no longer create one; new customers are directed to Snowflake Horizon Catalog.

Horizon Catalog exposes Snowflake-managed Iceberg tables to external engines such as Spark, Trino, and DuckDB through an Iceberg REST endpoint (https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog). Snowflake describes Apache Polaris as integrated into Horizon Catalog for this interoperability, and both reads and writes from external engines are supported with vended credentials. Snowflake’s docs say billing for this access is scheduled to begin in the second half of 2026.

External Engine (Trino/Spark) ──► Horizon Iceberg REST endpoint
                                            │
                                            ▼
                              Snowflake Horizon Governance
                                            │
                                            ├─► Row-Level Security
                                            └─► Column Masking

You define access policies, such as row access policies and masking policies, inside Snowflake using standard SQL, and grant roles SELECT on the tables external engines will use.

External engines then connect in one of two ways. A plain Iceberg REST connection works for tables without Snowflake data protection policies and does not enforce them. For tables protected by row access and masking policies, Snowflake documents a separate Spark connection path that enforces those policies, so check that path before you rely on masking for Spark users.

This makes Horizon Catalog a fit when Snowflake is the system of record for Iceberg tables and other engines need governed access to them. It is a weaker fit when you want the catalog to stay independent of any one warehouse vendor.

Multi-Format Security: Open-Source Unity Catalog

While Polaris and Snowflake’s catalogs center on the Apache Iceberg format, Databricks has taken a multi-format approach by open-sourcing Unity Catalog. Licensed under the Apache 2.0 license and hosted as a sandbox project under the LF AI & Data Foundation, open-source Unity Catalog catalogs Delta Lake and Apache Iceberg tables (Hudi through UniForm), plus files, functions, and AI models.

Unity Catalog provides a metadata governance layer that extends beyond basic table files. It manages volumes (unstructured files), registers functions and machine learning models, and handles access control.

Key milestones for open-source Unity Catalog in 2026 include:

To work with Iceberg tables in an open-source Unity Catalog server, point any Iceberg REST client at its Iceberg endpoint. For example, in PySpark:

from pyspark.sql import SparkSession

# Connect Spark to open-source Unity Catalog through its Iceberg REST endpoint
# (Iceberg 1.11.0 is the version shown in the Unity Catalog 0.7.0 release notes)
spark = SparkSession.builder \
    .appName("UnityCatalogIcebergREST") \
    .config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.unity", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.unity.type", "rest") \
    .config("spark.sql.catalog.unity.uri", "http://unity-server.data-platform.local:8080/api/2.1/unity-catalog/iceberg") \
    .config("spark.sql.catalog.unity.warehouse", "unity") \
    .config("spark.sql.catalog.unity.token", "<uc_token>") \
    .getOrCreate()

spark.sql("SELECT * FROM unity.db.inventory").show()

For Delta tables, DuckDB’s unity_catalog extension attaches a Unity Catalog server directly (CREATE SECRET (TYPE unity_catalog, TOKEN ..., ENDPOINT ..., AWS_REGION ...) followed by ATTACH 'main' AS main (TYPE unity_catalog)); it reads and inserts into Delta tables but does not cover Iceberg tables.

Unity Catalog’s multi-format support makes it a strong option for teams managing mixed data environments containing both Delta Lake and Apache Iceberg tables.

Catalog-managed commits matter most for multi-engine architectures. When each engine writes its own commit files straight to object storage, a misbehaving or outdated writer can corrupt table state. With catalog-managed commits, the catalog validates each commit before it lands, which keeps tables consistent across Spark, DuckDB, and other writers.

Stack diagram showing the security and catalog commit layers of modern Iceberg control planes including RBAC, credential vending, and multi-engine commits

Cloud Native Scoping: AWS Glue & Google Cloud Lakehouse

In addition to open-source and Snowflake options, the major cloud providers offer native managed REST endpoints for Iceberg catalogs.

AWS Glue Iceberg REST Catalog

AWS Glue serves the Iceberg REST API at https://glue.<region>.amazonaws.com/iceberg, authenticated with AWS SigV4. This catalog integrates directly with the AWS security and metadata stack, including AWS IAM for authentication, AWS Lake Formation for fine-grained access control, and Amazon S3.

Google Cloud Lakehouse Runtime Catalog

On April 20, 2026, Google renamed BigLake to Lakehouse and BigLake metastore to the Lakehouse runtime catalog. Its Iceberg REST catalog tables are the option Google recommends for most open lakehouse workloads.

Feature Matrix: Comparing Access Control and Credential Vending

Selecting a catalog requires comparing their support for key platform features:

CapabilityApache Polaris (1.8)Open-Source Unity Catalog (0.7)Snowflake Horizon CatalogAWS Glue REST Catalog
Governance BodyApache Software FoundationLinux Foundation (LF AI & Data)Snowflake (integrates Apache Polaris)Proprietary (AWS)
Primary FormatsApache IcebergDelta Lake, IcebergApache IcebergApache Iceberg
Credential VendingNative (AWS STS, GCP, Azure)Temporary credential vendingVended credentialsIAM and Lake Formation
Access ControlRole-Based Access (RBAC)Grants on catalog objectsSnowflake roles; masking and row policiesAWS Lake Formation
Metadata CommitsIceberg REST commit protocolIceberg REST and UC Delta APIIceberg REST commit protocolIceberg REST or Glue API
Billing ModelFree (Open Source)Free (Open Source)Billing scheduled for second half of 2026Serverless (Glue request pricing)

Decision Framework: Choosing the Right Catalog Control Plane

No single catalog control plane fits every data architecture. Your choice depends on your existing technology stack, cloud provider alignment, and format requirements.

Use this engineering decision framework to guide your selection:

  1. Are you a Databricks-centric shop running Delta Lake tables?
    • Recommendation: Use Unity Catalog. It provides catalog-managed Delta commits, metric views, and governance for files and models. The open-source version and its Iceberg REST endpoint let non-Databricks engines work in the same catalog namespace.
  2. Are you building an open-source, multi-engine lakehouse utilizing Apache Iceberg?
    • Recommendation: Use Apache Polaris, self-hosted or as a managed Polaris-based service such as Dremio Catalog. Its vendor-neutral ASF governance, credential vending across AWS, Azure, and GCS, and production features (STS session tags, KMS settings, metrics persistence) make it a strong default for open data platforms.
  3. Do you use Snowflake as your primary query engine but want to query tables with external engines (like Spark or Flink)?
    • Recommendation: Use Snowflake Horizon Catalog and its Iceberg REST endpoint. Existing Snowflake Open Catalog customers can stay on Open Catalog, but new customers cannot create Open Catalog accounts.
  4. Are you hosted entirely inside a single cloud provider (AWS or Google Cloud) and want a managed, serverless catalog?
    • Recommendation: Use AWS Glue REST Catalog (with S3 Tables if you want managed maintenance) or the Google Cloud Lakehouse runtime catalog. These options integrate with cloud-native security and IAM policies, reducing operational setup overhead.

For example, if your pipeline is built primarily on Spark and Trino running on AWS EMR, using self-hosted Apache Polaris or AWS Glue REST Catalog is the logical choice. However, if your business units query data using both Databricks and Snowflake, catalog federation or the Iceberg REST endpoints both platforms now expose provide the bridge to avoid data duplication.

Flowchart decision tree helping engineers evaluate catalog requirements and select the correct Iceberg control plane

Conclusion

The metadata control plane is the strategic layer of the modern data lakehouse. By decoupling query execution from metadata management, the Apache Iceberg REST Catalog specification enables true multi-engine interoperability and secure data access.

Whether you deploy the open-source Apache Polaris server, open-source Unity Catalog, Snowflake Horizon Catalog, or cloud-native managed REST endpoints, establishing a centralized catalog is the critical step to scale your data platform and secure your metadata.

Frequently Asked Questions

Should I choose Apache Polaris or Unity Catalog for Iceberg?

Choose Apache Polaris when Iceberg is your main format and you want a vendor-neutral Apache catalog with RBAC and credential vending for many engines. Choose Unity Catalog when you also manage Delta Lake tables, files, and models, especially alongside Databricks; open source Unity Catalog 0.7.0 added full Iceberg REST table support.

What is an Iceberg control plane?

It is the catalog service that every engine talks to before touching data: it resolves table names to current metadata, makes commits atomic, enforces access rules, and vends short-lived storage credentials. In practice it is a server that implements the Iceberg REST catalog API.

Is Snowflake Open Catalog still available?

Existing Open Catalog customers can keep using it and add accounts, but Snowflake no longer lets new customers create a first Open Catalog account. Snowflake directs new customers to Horizon Catalog, whose Iceberg REST endpoint integrates Apache Polaris and supports reads and writes from external engines.

Does AWS Glue support the Iceberg REST catalog API?

Yes. Glue serves the Iceberg REST API at https://glue.{region}.amazonaws.com/iceberg with SigV4 authentication, and it can reach tables in both general purpose S3 buckets and S3 Tables. It suits stacks that run entirely on AWS.

Is Apache Polaris a top-level Apache project?

Yes. Polaris graduated from the Apache Incubator to a top-level project in February 2026, and its current release is 1.8.0 from September 2026. Dremio Catalog is built on Apache Polaris.

Master the Modern Lakehouse

To build your expertise in modern data architectures, open table formats, and semantic metadata design, consider the following next steps:

Work with Alex