AWS Glue Catalog for Apache Iceberg

AWS Glue Catalog for Apache Iceberg

The AWS Glue Data Catalog can serve as your Iceberg catalog: it stores each table’s current metadata file pointer, Athena, EMR, Glue ETL, and Redshift read it natively, and other engines can connect through its Iceberg REST endpoint at https://glue.<region>.amazonaws.com/iceberg.

AWS Glue Data Catalog is Amazon Web Services’ managed metadata catalog service, deeply integrated with the AWS analytics ecosystem. It supports Apache Iceberg tables natively and also exposes an Iceberg REST Catalog API, making it the natural catalog choice for teams running Iceberg workloads entirely within AWS.

Overview

The AWS Glue Data Catalog serves as a centralized metadata repository for data assets in AWS:

• Table definitions: Column names, types, partition info, and (for Iceberg tables) the metadata file location.
• Database/namespace management: Logical grouping of tables.
• Lake Formation integration: Fine-grained access control via AWS Lake Formation on top of Glue catalog entries.
• Service integration: Native discovery by Athena, EMR, Glue ETL jobs, Redshift Spectrum, and more.

Glue and Apache Iceberg

AWS Glue supports Iceberg through two modes:

Native Iceberg Table Support

Glue can register and manage Iceberg tables directly. When using Glue as an Iceberg catalog:

• The Glue catalog stores the current metadata file pointer for each Iceberg table.
• AWS services (Athena, EMR, Glue ETL) can read and write Iceberg tables via Glue.
• Glue table optimizers can run managed compaction, snapshot retention, and orphan file deletion on Iceberg tables.

Iceberg REST Catalog Endpoint

Glue exposes an Iceberg REST Catalog endpoint (https://glue.<region>.amazonaws.com/iceberg), allowing engines and clients that support the REST Catalog API (PyIceberg, Spark with Iceberg REST config, Flink) to use Glue as their Iceberg catalog via the standard protocol. Requests are signed with AWS SigV4 (signing name glue), the warehouse setting is your AWS account ID (the Glue catalog ID), and the endpoint supports Iceberg format v1 and v2 tables, including tables in Amazon S3 Tables.

Using Glue with Apache Spark (EMR/Glue ETL)

spark = SparkSession.builder \
    .config("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.glue_catalog.warehouse", "s3://my-bucket/warehouse/") \
    .config("spark.sql.catalog.glue_catalog.catalog-impl",
            "org.apache.iceberg.aws.glue.GlueCatalog") \
    .config("spark.sql.catalog.glue_catalog.io-impl",
            "org.apache.iceberg.aws.s3.S3FileIO") \
    .getOrCreate()

# Create an Iceberg table in Glue
spark.sql("""
    CREATE TABLE glue_catalog.db.orders (
        order_id BIGINT,
        order_date TIMESTAMP,
        total DOUBLE
    ) USING iceberg PARTITIONED BY (days(order_date))
""")

Using Glue with AWS Athena

Athena has native Iceberg support using the Glue catalog:

-- Athena: create Iceberg table in Glue
CREATE TABLE orders (
    order_id bigint,
    order_date timestamp,
    total double
)
PARTITIONED BY (day(order_date))
LOCATION 's3://my-bucket/warehouse/db/orders/'
TBLPROPERTIES ('table_type'='ICEBERG');

-- Time travel query
SELECT * FROM orders FOR TIMESTAMP AS OF TIMESTAMP '2026-01-01 00:00:00 UTC';

AWS Lake Formation Integration

AWS Lake Formation provides fine-grained access control for Glue-cataloged Iceberg tables:

• Table-level permissions: Grant SELECT, INSERT, DROP to specific IAM roles/users.
• Column-level security: Restrict access to specific columns.
• Row-level security: Filter rows based on the identity of the querying principal.

Lake Formation integrates with Glue to enforce these permissions across all AWS services that query the catalog (Athena, EMR, Redshift Spectrum).

Glue vs. Apache Polaris for AWS Workloads

For teams building on AWS exclusively, Glue is the most frictionless choice. For multi-cloud or cross-engine portability requirements, Apache Polaris (available via Dremio Cloud or self-hosted) offers broader interoperability.

Pricing

AWS Glue Data Catalog stores the first million objects and serves the first million requests each month for free; beyond that, AWS lists $1.00 per 100,000 objects per month and $1.00 per million requests. Iceberg metadata operations (table loads, commits) count against Glue API request quotas. For very high-throughput Iceberg workloads, catalog request costs should be factored into architecture decisions.