# Alex Merced's Lakehouse Blog

> Alex Merced's Lakehouse Blog (iceberglakehouse.com) is Alex Merced's independent blog on the open lakehouse. It has 474 articles on Apache Iceberg internals, lakehouse catalogs such as Apache Polaris and the Iceberg REST Catalog, table maintenance and performance, query engines, and the agentic lakehouse, where AI agents query governed data through semantic layers and open protocols such as MCP. Alex is Head of Developer Relations at Dremio and a co-author of Apache Iceberg: The Definitive Guide. The pillar pages below are the best starting points; the posts are listed newest first within each topic.

## Pillar Pages

- [Agentic Analytics and the Semantic Layer: Natural Language to SQL on Iceberg](https://iceberglakehouse.com/agentic-analytics/): Agentic analytics lets AI agents answer business questions by generating and executing SQL against governed lakehouse data. This guide explains how NL2SQL works, what role the semantic layer plays in accuracy, and how to build a production natural language analytics system on Apache Iceberg.
- [What Is an Agentic Lakehouse | Architecture, Governance, and AI Agents](https://iceberglakehouse.com/agentic-lakehouse/): An agentic lakehouse is an architecture where AI agents can safely query, reason over, and act on governed lakehouse data. Learn what the four required layers are, how governance and trust work, and why open standards matter for AI-ready data infrastructure.
- [Apache Iceberg Architecture: Metadata Tree, Snapshots, and Catalogs Explained](https://iceberglakehouse.com/apache-iceberg-architecture/): A technical deep dive into how Apache Iceberg works internally — the metadata JSON, manifest list, manifest files, data files, commit flow, query planning, and concurrency model.
- [Iceberg REST Catalog Spec: Endpoints, Auth, and Polaris](https://iceberglakehouse.com/apache-iceberg-rest-catalog/): The Iceberg REST Catalog spec is the open HTTP API engines use to reach any catalog: endpoints, auth, scan planning, and Polaris setup, current to Iceberg 1.12.
- [Apache Iceberg Schema Evolution: Add, Rename, Drop Safely](https://iceberglakehouse.com/apache-iceberg-schema-evolution/): Apache Iceberg schema evolution lets you add, rename, drop, reorder, and widen columns as metadata-only changes. Spark SQL examples plus partition evolution.
- [Apache Iceberg Snapshots and Time Travel With SQL Examples](https://iceberglakehouse.com/apache-iceberg-snapshots-and-time-travel/): Apache Iceberg time travel queries any past snapshot by ID or timestamp. SQL for Spark, Trino, and Dremio, plus branches, tags, rollback, and snapshot expiry.
- [Apache Iceberg vs Delta Lake vs Apache Hudi: Complete Comparison](https://iceberglakehouse.com/apache-iceberg-vs-delta-lake-vs-hudi/): A neutral, technical comparison of Apache Iceberg, Delta Lake, and Apache Hudi covering architecture, multi-engine support, streaming, governance, and a decision framework by workload type.
- [Apache Iceberg Explained: Architecture, Snapshots, and Catalogs](https://iceberglakehouse.com/apache-iceberg/): Apache Iceberg is an open table format for huge analytic tables. Learn how it works internally, from the metadata tree and snapshot model to hidden partitioning, schema evolution, and the catalog API that makes multi-engine access possible.
- [Open Table Formats Benchmark: Iceberg vs Delta Lake vs Hudi](https://iceberglakehouse.com/benchmarks/open-table-formats/): An exhaustive, production-grade performance benchmark of open table formats. Analyze query scan speed, write throughput, maintenance overhead, and engine selection caveats across Apache Iceberg, Delta Lake, and Apache Hudi.
- [Books by Alex Merced](https://iceberglakehouse.com/books/): Books on Apache Iceberg, lakehouse architecture, catalogs, and agentic analytics, written by Alex Merced.
- [Data Lakehouse vs Data Lake vs Data Warehouse: Key Differences](https://iceberglakehouse.com/data-lakehouse-vs-data-lake-vs-data-warehouse/): A practical comparison of the three main data storage architectures — data lake, data warehouse, and data lakehouse — covering storage model, query performance, governance, cost, AI/ML support, and when each is the right choice.
- [What Is a Data Lakehouse | Apache Iceberg & Lakehouse Guide](https://iceberglakehouse.com/data-lakehouse/): A data lakehouse combines the low-cost storage of a data lake with the reliability and query performance of a data warehouse. Learn how the architecture works, what layers it includes, and when it makes sense to build one.
- [Lakehouse for AI Agents: Architecture and Implementation Guide](https://iceberglakehouse.com/lakehouse-for-ai-agents/): AI agents need governed, documented, queryable data to work reliably. This guide explains how to build a data architecture that serves AI agents through structured interfaces, covering the data layer, catalog, semantic layer, and agent connection patterns.
- [Open Table Formats Explained: Iceberg, Delta Lake, Hudi, and Paimon](https://iceberglakehouse.com/open-table-formats/): Open table formats add ACID transactions, schema evolution, and query optimization to raw object storage. This guide explains what they are, how they work, and how Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon each approach the problem.

## Articles: Apache Iceberg and Table Formats

- [Keeping Audit Snapshots Alive While Iceberg Snapshot Expiration Runs Every Night](https://iceberglakehouse.com/posts/iceberg-tags-regulatory-snapshot-retention/) (2026-09-28): How Iceberg snapshot tags keep audit snapshots alive through nightly expiration: retention calendars, RETAIN semantics, compaction cost, and erasure conflicts.
- [Why the Iceberg DataFusion Integration Is Moving to Apache DataFusion](https://iceberglakehouse.com/posts/datafusion-iceberg-moves-to-apache-datafusion/) (2026-09-21): Why the Iceberg DataFusion integration moved to the DataFusion project, and what the split means for users, Comet, and iceberg-rust contributors.
- [What Iceberg v4's Proposed FILE Type Means for Multimodal Tables](https://iceberglakehouse.com/posts/iceberg-v4-file-type-multimodal-tables/) (2026-09-21): Iceberg v4's proposed FILE type brings first-class media references to tables, via Parquet's FILE logical type, ranges, checksums, and pre-signed URLs.
- [CVE-2026-73334 and the Trust Boundary Inside an Encrypted Parquet File](https://iceberglakehouse.com/posts/parquet-kms-url-cve-2026-73334-iceberg/) (2026-09-21): CVE-2026-73334 lets a tampered Parquet footer route a reader's KMS token to an attacker. Here's the fix, Iceberg's safe path, and how to audit your lakehouse.
- [Parquet Page Indexes and the Last Mile of Pruning in Apache Iceberg](https://iceberglakehouse.com/posts/parquet-page-index-pruning-iceberg/) (2026-09-21): Parquet page indexes can cut selective Iceberg scans by an order of magnitude on sorted data. How they work, what they cost, and how to lay out tables.
- [How Apache Polaris Plans to Share Iceberg Tables Across Organizations](https://iceberglakehouse.com/posts/polaris-open-sharing-iceberg-tables/) (2026-09-21): Apache Polaris's Open Sharing proposal adds first-class shares, external consumers, and listings so any Iceberg REST engine can read shared tables.
- [Keeping Lakehouse Traffic Off the Public Internet With Apache Polaris](https://iceberglakehouse.com/posts/polaris-private-networking-lakehouse/) (2026-09-21): A private Polaris lakehouse still leaks data if the storage hop goes public. How to bind vended credentials to private networks on AWS, Azure, and GCP.
- [Vector Search Directly Over Iceberg Tables, and When You Still Need a Vector Database](https://iceberglakehouse.com/posts/vector-search-over-iceberg-tables/) (2026-09-21): Embeddings are just columns. When exact vector search over Iceberg scans beats a vector database, when it does not, and how to lay out tables.
- [Running Apache Polaris in Production](https://iceberglakehouse.com/posts/apache-polaris-in-production/) (2026-09-10): Apache Polaris past the quickstart: persistence backends, realm bootstrap, replica token signing, upgrades, backups, and which failures take the lakehouse offline.
- [Migrating Into Iceberg Without Moving Data](https://iceberglakehouse.com/posts/iceberg-in-place-migration/) (2026-09-10): add_files, snapshot, and migrate compared: the three in-place paths into Iceberg, the reconciliation each requires, the layout traps, and the rollback story.
- [What Iceberg Table Maintenance Actually Costs](https://iceberglakehouse.com/posts/iceberg-maintenance-budget/) (2026-09-10): A cost model for compaction, snapshot expiry, orphan cleanup, and manifest rewriting: what each operation spends, on which meter, and how to set a schedule.
- [Partition Statistics Files in Apache Iceberg](https://iceberglakehouse.com/posts/iceberg-partition-statistics-files/) (2026-09-10): The underused Iceberg metadata for planning: what the partition statistics file holds, what the spec guarantees, how to write one, and when it earns its slot.
- [The 2026 Iceberg REST Catalog Compatibility Report](https://iceberglakehouse.com/posts/iceberg-rest-catalog-compatibility-report-2026/) (2026-09-10): A repeatable test of what Iceberg REST catalogs actually serve, a scoring scheme that separates design choices from breakage, and 2026 results for 7 catalogs.
- [Serving Iceberg Tables From Two Regions](https://iceberglakehouse.com/posts/iceberg-two-region-serving/) (2026-09-10): Three multi-region topologies that work and one that mostly does not, what an Iceberg commit costs across regions, and where the catalog has to live.
- [Disaster Recovery for Iceberg Tables: Replication, Backup, and Restore](https://iceberglakehouse.com/posts/disaster-recovery-for-iceberg-tables/) (2026-09-02): Disaster recovery for Iceberg across four tiers: snapshots, object versioning, catalog backup, and cross-region replication.
- [Geospatial Data in Apache Iceberg: Geometry, Geography, and GeoParquet](https://iceberglakehouse.com/posts/geospatial-data-in-apache-iceberg/) (2026-09-02): How Iceberg v3 geometry and geography types, bounding boxes, and native Parquet types give spatial data first-class standing.
- [Default Column Values and Field IDs: How Iceberg Schema Evolution Works at the Spec Level](https://iceberglakehouse.com/posts/iceberg-default-values-and-field-ids/) (2026-09-02): How field IDs and initial and write defaults let Iceberg change schemas on large tables without rewriting data, at the spec level.
- [The Iceberg Table Properties That Actually Matter](https://iceberglakehouse.com/posts/iceberg-table-properties-that-matter/) (2026-09-02): The Iceberg table properties that decide file count, pruning, write amplification, retention, and metadata growth, by workload.
- [Inside the Puffin File Format](https://iceberglakehouse.com/posts/inside-the-puffin-file-format/) (2026-09-02): A byte-by-byte guide to the Iceberg Puffin file format, its Theta sketch and deletion vector blobs, how metadata points to them, and how to operate them safely.
- [Local Iceberg Development Environments: Docker, MinIO, and In-Memory Catalogs for CI](https://iceberglakehouse.com/posts/local-iceberg-development-environments/) (2026-09-02): Local Iceberg development environments: in-process catalogs, a Docker Compose stack with MinIO, and CI configurations that run either.
- [Moving Iceberg Tables Between Catalogs Without Rewriting Data](https://iceberglakehouse.com/posts/moving-iceberg-tables-between-catalogs/) (2026-09-02): Why moving Iceberg tables between catalogs is a pointer copy, and the protocol that makes a cutover safe for one table or thousands.
- [Storage-Partitioned Joins and the Bucket Transform](https://iceberglakehouse.com/posts/storage-partitioned-joins-and-the-bucket-transform/) (2026-09-02): How the spec-defined bucket transform lets engines skip the shuffle in joins, and how to set it up and keep it engaged in Spark.
- [Agent-Driven Storage Tiering for Apache Iceberg: Moving Cold Data Without Breaking Queries](https://iceberglakehouse.com/posts/agent-driven-iceberg-storage-tiering/) (2026-08-25): A background agent can move cold Iceberg partitions to cheaper tiers without breaking live queries. Heatmaps, path-safe moves, and restore paths.
- [DataFusion Comet 1.0 and What Native Rust Scans Change for Spark on Iceberg](https://iceberglakehouse.com/posts/datafusion-comet-1-spark-iceberg/) (2026-08-25): DataFusion Comet 1.0 replaces Spark Iceberg scans with native Rust. What speeds up, what still falls back to the JVM, and how to deploy it.
- [FSST and ALP: The Two Encodings Fixing Parquet's Weakest Compression Cases](https://iceberglakehouse.com/posts/fsst-alp-parquet-encodings/) (2026-08-25): ALP and FSST target Parquet's worst cases: floats and high-cardinality strings. How they work and what they change for Iceberg tables.
- [High-Throughput Branch Merging: Automating Concurrency and Conflict Resolution in Multi-Branch Iceberg Pipelines](https://iceberglakehouse.com/posts/multi-branch-iceberg-merge-automation/) (2026-08-25): High-throughput Iceberg branch merges need conflict detection and automation. How to reconcile concurrent writes without stalling pipelines.
- [Multi-Cloud REST Catalog Topologies: Running Apache Polaris Across AWS, Azure, and GCP](https://iceberglakehouse.com/posts/multi-cloud-polaris-rest-catalog-topologies/) (2026-08-25): Polaris can catalog Iceberg tables across AWS, Azure, and GCP. Four topologies, credential vending, and the tradeoffs of each design.
- [Parquet-Only Manifests in Iceberg v4: Why the Metadata Layer Is Going Columnar](https://iceberglakehouse.com/posts/parquet-manifests-iceberg-v4/) (2026-08-25): Iceberg v4 is moving manifests from Avro to Parquet so planners can read only the stats they need. Why the metadata layer is going columnar.
- [Serverless Iceberg Ingestion with PyIceberg and DuckDB: Micro-Batches Without a Spark Cluster](https://iceberglakehouse.com/posts/serverless-iceberg-microbatch-pyiceberg-duckdb/) (2026-08-25): Land small Iceberg micro-batches with PyIceberg and DuckDB in a serverless function. Commits, concurrency, and why Spark is the wrong default.
- [Zero-Copy Warehouse Modernization: Migrating Legacy Databases to Apache Iceberg Without Downtime](https://iceberglakehouse.com/posts/zero-copy-legacy-db-to-iceberg/) (2026-08-25): Move a legacy warehouse to Iceberg without downtime by virtualizing first. Consumer cutover, parity checks, and background copy without double-ETL.
- [The Hidden Cost of Tiny Iceberg Commits](https://iceberglakehouse.com/posts/hidden-cost-of-tiny-iceberg-commits/) (2026-08-24): Trace what one tiny Iceberg commit writes, then model hourly, per-minute, and per-second cadences so streaming costs become arithmetic, not adjectives.
- [Deletion Vectors vs Position Deletes vs Equality Deletes: The Iceberg Delete Story in 2026](https://iceberglakehouse.com/posts/iceberg-delete-story-2026/) (2026-08-24): Position deletes, equality deletes, and deletion vectors compared from the Iceberg spec: what each writes, how readers apply it, and when to use which.
- [Iceberg Is Becoming a Library, Not Just a Table Format](https://iceberglakehouse.com/posts/iceberg-is-becoming-a-library/) (2026-08-24): Iceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself.
- [Iceberg Is Escaping the JVM: Why Rust, Go, Python and C++ Implementations Matter](https://iceberglakehouse.com/posts/iceberg-is-escaping-the-jvm/) (2026-08-24): Rust, Go, Python, and C++ Iceberg implementations change who can write the format. Why multi-language clients matter more than another JVM engine.
- [The Iceberg REST Catalog Compatibility Test: One Suite of Operations Every Platform Should Pass](https://iceberglakehouse.com/posts/iceberg-rest-catalog-compatibility-test/) (2026-08-24): One suite of REST catalog operations every Iceberg platform should pass. What sameness means, where implementations diverge, and how to test it.
- [Iceberg REST Remote Scan Planning Changes More Than Query Performance](https://iceberglakehouse.com/posts/iceberg-rest-remote-scan-planning/) (2026-08-24): Remote scan planning moves Iceberg file selection into the catalog. What that changes for engines, governance, and operational cost beyond query speed.
- [Iceberg Row Lineage: The Feature AI and CDC Workloads Will Eventually Depend On](https://iceberglakehouse.com/posts/iceberg-row-lineage/) (2026-08-24): Iceberg row lineage gives rows a durable identity across rewrites. Why CDC pipelines and AI workloads will eventually depend on it.
- [Iceberg v4's Adaptive Metadata Tree, Explained From First Principles](https://iceberglakehouse.com/posts/iceberg-v4-adaptive-metadata-tree/) (2026-08-24): Iceberg v4's adaptive metadata tree, explained from first principles: why commits rewrite too much today and how the tree makes change cheaper.
- [Why Iceberg v4 Is Really About Making the Cost of Change Proportional to the Change](https://iceberglakehouse.com/posts/iceberg-v4-cost-of-change-principle/) (2026-08-24): Iceberg v4 is really about making the cost of a change proportional to the change. The principle, the current tax, and what the redesign pays down.
- [The Catalog Can Now Plan Your Iceberg Query: Inside REST Scan Planning](https://iceberglakehouse.com/posts/inside-iceberg-rest-scan-planning/) (2026-08-24): A mechanics walkthrough of Iceberg REST scan planning: client-side planning, remote endpoints, pagination, and where engine support stands in 2026.
- [Can Seven Different Iceberg REST Catalogs Really Run the Same DuckDB Code?](https://iceberglakehouse.com/posts/seven-rest-catalogs-one-duckdb-script/) (2026-08-24): Can seven Iceberg REST catalogs run the same DuckDB script? What the protocol makes portable, what still differs, and a test matrix you can rerun.
- [Stop Flattening Your JSON: How Iceberg Variant Changes Semi-Structured Analytics](https://iceberglakehouse.com/posts/stop-flattening-your-json-iceberg-variant/) (2026-08-24): Iceberg Variant stores JSON as navigable binary with shredding for columnar filters. Why flattening wide tables is no longer the only performance path.
- [What Actually Happens When Two Engines Write the Same Iceberg Table at Once?](https://iceberglakehouse.com/posts/two-engines-one-iceberg-table/) (2026-08-24): What happens when two engines write the same Iceberg table at once: snapshot isolation, optimistic commits, conflict detection, and when retries fail.
- [Variant Shredding Explained: How Iceberg Gets Columnar Performance From Messy JSON](https://iceberglakehouse.com/posts/variant-shredding-explained/) (2026-08-24): Variant shredding turns messy JSON into Parquet columns with statistics. How the layout works, how readers reassemble values, and why some queries prune.
- [Who Actually Owns an Iceberg Table? Managed, External, and the New Vocabulary of Lakehouse Control](https://iceberglakehouse.com/posts/who-owns-an-iceberg-table/) (2026-08-24): Managed and external Iceberg tables mean different things on every platform. Five ownership dimensions and a translation method for vendor vocabulary.
- [Mastering Apache Iceberg v3 Deletion Vectors for High-Throughput Streaming Ingest](https://iceberglakehouse.com/posts/apache-iceberg-v3-deletion-vectors-streaming-ingest/) (2026-08-19): Apache Iceberg v3 deletion vectors for high-throughput streaming ingest: how bitmaps and Puffin files fix CDC write amplification and read decay.
- [The Decoupled Data Lakehouse: Multi-Engine Freedom with Open REST Catalogs](https://iceberglakehouse.com/posts/decoupled-data-lakehouse-multi-engine-rest-catalogs/) (2026-08-19): The decoupled data lakehouse: multi-engine freedom with open REST catalogs, credential vending, and an estate that outlives its tools.
- [Building Lightweight Serverless Ingestion to Apache Iceberg with PyIceberg and DuckDB](https://iceberglakehouse.com/posts/serverless-iceberg-ingestion-pyiceberg-duckdb/) (2026-08-19): Build lightweight serverless ingestion to Apache Iceberg with PyIceberg and DuckDB, running small feeds in functions that bill for seconds.
- [Write-Audit-Publish with Apache Iceberg Branches: CI/CD for Your Data](https://iceberglakehouse.com/posts/write-audit-publish-iceberg-branches/) (2026-08-19): Write-Audit-Publish with Apache Iceberg branches brings CI/CD to your data: stage, audit, then fast-forward to main so consumers never see unvalidated bytes.
- [Zero-Copy Warehouse Modernization: Moving to Apache Iceberg Without Downtime](https://iceberglakehouse.com/posts/zero-copy-warehouse-modernization-iceberg/) (2026-08-19): A four-phase plan for moving a warehouse to Apache Iceberg without downtime: federate first, redirect new data, materialize selectively, and cut over gradually.
- [Autonomous Table Optimization When Your Query Workload Stops Being Predictable](https://iceberglakehouse.com/posts/autonomous-table-optimization/) (2026-08-04): Autonomous table optimization when query workloads stop being predictable: observing file layout and query patterns, scoring compaction work, adaptive.
- [Surviving Optimistic Commit Collisions When Hundreds of Agents Write to Iceberg](https://iceberglakehouse.com/posts/high-concurrency-agent-writes-iceberg/) (2026-08-04): Which Iceberg commit conflicts are real when hundreds of agents write, plus commit buffers, partitioning, catalog support, benchmarks, and a checklist.
- [How Iceberg V3 Deletion Vectors Fixed Merge-on-Read for Streaming Tables](https://iceberglakehouse.com/posts/iceberg-v3-deletion-vectors-merge-on-read/) (2026-08-04): How Iceberg V3 deletion vectors replaced accumulating positional delete files and made merge-on-read viable for streaming and CDC tables.
- [How Iceberg V3 Variant Shredding Changed Semi-Structured Data on S3 Tables](https://iceberglakehouse.com/posts/iceberg-v3-variant-shredding-s3-tables/) (2026-08-04): How Iceberg V3's Variant type and Parquet shredding turn JSON columns into prunable typed columns, with real benchmark tradeoffs and a migration path.
- [Reading the Apache Iceberg V4 Proposals Before They Land](https://iceberglakehouse.com/posts/iceberg-v4-roadmap/) (2026-08-04): A field guide to the Apache Iceberg V4 proposals: adaptive metadata trees, single-file commits, typed statistics, column families, and what is safe.
- [Cross-Cloud Credential Vending in Apache Polaris and the End of Permanent Storage Keys](https://iceberglakehouse.com/posts/polaris-cross-cloud-credential-vending/) (2026-08-04): How Apache Polaris vends short-lived, prefix-scoped storage credentials across AWS, Azure, and GCP, and how to retire permanent storage keys for good.
- [How the Iceberg REST Catalog Turned Into the Lakehouse Control Plane](https://iceberglakehouse.com/posts/rest-catalog-v2-multi-table-commits/) (2026-08-04): How the Iceberg REST catalog became the lakehouse control plane: multi-table atomic commits, credential vending, capability negotiation, and what still breaks.
- [What Zero-Copy Data Sharing Actually Does Between Salesforce, Snowflake, and Databricks](https://iceberglakehouse.com/posts/zero-copy-data-sharing-iceberg/) (2026-08-04): How zero-copy sharing works between Salesforce, Snowflake, and Databricks: query, file, and catalog federation compared, with failure modes and governance.
- [Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy](https://iceberglakehouse.com/posts/apache-polaris-1-7-0/) (2026-08-02): Apache Polaris 1.7.0 deep dive: idempotent writes, semantic models, stricter credential vending, orphan cleanup, and what the upgrade asks of you.
- [Designing Batch Pipelines That Write Well Into Apache Iceberg](https://iceberglakehouse.com/posts/batch-pipelines-into-apache-iceberg/) (2026-08-02): How to design batch pipelines that write well into Apache Iceberg: commit strategy, partitioning, sort order, write-audit-publish, and maintenance done right.
- [Apache Iceberg Support Across the Major Hyperscalers](https://iceberglakehouse.com/posts/iceberg-across-the-hyperscalers/) (2026-08-02): How AWS, Google Cloud, and Microsoft Azure actually support Apache Iceberg: storage, catalogs, maintenance, governance, and interoperability, layer by layer.
- [Running an Apache Iceberg Lakehouse With No Internet Connection](https://iceberglakehouse.com/posts/air-gapped-iceberg-lakehouse/) (2026-07-28): A practical guide to deploying an Iceberg lakehouse in air-gapped environments: component choices, artifact pipelines, identity without a cloud.
- [When the Query Optimizer Starts Managing Its Own Materializations](https://iceberglakehouse.com/posts/autonomous-reflections-agentic-lakehouse/) (2026-07-28): Autonomous materialized view management replaces quarterly review meetings with workload-driven scoring, and it's essential when AI agents generate.
- [Why Iceberg V4 Wants to Retire Equality Deletes, and What Streaming Teams Should Do About It](https://iceberglakehouse.com/posts/equality-deletes-iceberg-v4/) (2026-07-28): Why equality deletes slow Iceberg reads for streaming upserts, what the V4 proposal changes, and how to convert them to deletion vectors with Flink and verify.
- [Apache Fluss and Kafka Solve Different Problems in an Iceberg Pipeline](https://iceberglakehouse.com/posts/fluss-kafka-iceberg-streaming/) (2026-07-28): Fluss puts a columnar, indexed hot tier between Kafka and Iceberg. Here's what it changes structurally, what Kafka still does better, and how to benchmark.
- [Serving Sub-Second Queries Over an Iceberg Lakehouse With a Hot Tier](https://iceberglakehouse.com/posts/hot-tier-iceberg-serving/) (2026-07-28): A lakehouse cannot serve sub-second queries over seconds-old data. A hot tier in front solves it, with consequences for consistency, governance.
- [Surviving Commit Conflicts When Dozens of Writers Hit the Same Iceberg Table](https://iceberglakehouse.com/posts/iceberg-concurrent-commits-agents/) (2026-07-28): Why Iceberg commits fail on stale metadata as writer counts grow, how isolation levels and retry settings change the outcome, and how to budget for contention.
- [The Jackson 3 Problem in Apache Iceberg, and What It Means for Your Code](https://iceberglakehouse.com/posts/iceberg-jackson-3-migration/) (2026-07-28): Jackson 3 changes everything: package names, unchecked exceptions, flipped defaults. Here's what breaks, why the engines are fine and your service isn't.
- [Wiring an AI Agent to Apache Polaris with the Model Context Protocol](https://iceberglakehouse.com/posts/mcp-apache-polaris/) (2026-07-28): How to connect an AI agent to Apache Polaris with its official MCP server: authentication, adding a read path, tool descriptions, and credential vending.
- [Governing Iceberg Tables Across Regions Without Three Sets of Permissions](https://iceberglakehouse.com/posts/multi-region-catalog-federation/) (2026-07-28): How catalog federation gives Iceberg tables in several regions one authorization model and audit point, what it cannot fix, and how to build the topology.
- [Federating Oracle With an Open Lakehouse Instead of Migrating It](https://iceberglakehouse.com/posts/oracle-lakehouse-federation/) (2026-07-28): Federate first so analytics work now, migrate what benefits from migrating, and leave the rest where it is indefinitely.
- [The Parquet Versioning Problem, and Why Iceberg Cares About It](https://iceberglakehouse.com/posts/parquet-versioning-iceberg-v4/) (2026-07-28): Parquet files have a version field that doesn't reliably signal feature requirements. A new versioning discipline is coming, borrowing from Iceberg's.
- [Building Iceberg Pipelines in Python Without Standing Up Spark](https://iceberglakehouse.com/posts/python-native-iceberg-pipelines/) (2026-07-28): A large share of production transformations fit comfortably on one machine. PyIceberg, DuckDB, and branch isolation give you a production path that debugs.
- [Apache Polaris and Multi-Engine Iceberg Catalogs](https://iceberglakehouse.com/posts/apache-polaris-asf-top-level-multi-engine-catalog-interoperability/) (2026-07-13): Why open Iceberg tables need an open catalog, and how Apache Polaris implements the REST catalog with delegated RBAC across Spark, Flink, Trino, and Dremio.
- [Iceberg v3 Deletion Vectors and Faster DML](https://iceberglakehouse.com/posts/iceberg-v3-deletion-vectors-merge-on-read-dml-performance/) (2026-07-13): Copy-on-write versus merge-on-read in Iceberg, how v3 deletion vectors track row-level deletes, what readers pay, and compaction patterns for high-churn tables.
- [Iceberg Variant Type for AI JSON Data](https://iceberglakehouse.com/posts/iceberg-variant-type-semi-structured-ai-datasets/) (2026-07-13): Why JSON strings and map columns fall short for LLM outputs and agent logs, and how a native Iceberg variant type stores and queries nested AI payloads instead.
- [Iceberg Concurrency for AI Agent Writes](https://iceberglakehouse.com/posts/llm-agency-concurrency-isolation-high-frequency-agent-writes-iceberg/) (2026-07-13): How Iceberg optimistic concurrency behaves when fleets of agents commit, and how partition isolation, commit queues, and ingestion gateways prevent conflicts.
- [REST Catalog Credential Vending for Iceberg](https://iceberglakehouse.com/posts/rest-catalog-credential-vending-zero-copy-security/) (2026-07-13): How Iceberg REST catalog credential vending replaces static cloud keys with short-lived scoped credentials, what the catalog controls, and how to limit agents.
- [REST Catalog V2: Fixing Iceberg Protocol Debt](https://iceberglakehouse.com/posts/rest-catalog-v2-standard-iceberg-scaling-protocol-debt/) (2026-07-13): Where the Iceberg REST catalog v1 protocol strains under BI and agent planning load, and how batch table loading, caching, and stateless servers address it.
- [The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter](https://iceberglakehouse.com/posts/iceberg-v4-state-july-2026/) (2026-07-06): What the Iceberg v4 dev list tells us about adaptive metadata trees, single-file commits, column updates, and the format's next chapter in mid-2026.
- [The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere](https://iceberglakehouse.com/posts/state-of-apache-arrow-2026/) (2026-07-06): Apache Arrow at 10, ADBC, Flight SQL, nanoarrow, the AI reinterpretation, and how an in-memory standard eliminated the copy tax across the data stack.
- [The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade](https://iceberglakehouse.com/posts/state-of-apache-parquet-2026/) (2026-07-06): Apache Parquet in 2026, variant types, geospatial, ALP encoding, footer redesign, the versioning debate, and how the decade-old format is renovating.
- [The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse](https://iceberglakehouse.com/posts/state-of-apache-polaris-july-2026/) (2026-07-06): Apache Polaris as a TLP, federation, credential vending, semantic layers, lineage, and how the open catalog became the governance plane.
- [The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough](https://iceberglakehouse.com/posts/streaming-to-iceberg-july-2026/) (2026-07-06): Every path for streaming data into Iceberg in 2026, Flink, Spark, Kafka Connect, broker-native, managed pipelines, with honest latency numbers.
- [High-Performance Columnar Transfers: Combining Apache Arrow Flight and Iceberg REST Catalogs](https://iceberglakehouse.com/posts/arrow-flight-iceberg-rest-catalog-columnar-transfers/) (2026-07-06): How Iceberg REST catalogs run the control plane and Arrow Flight runs the data plane, plus the security, performance, and lock-in tradeoffs of pairing them.
- [S3 Tables MCP Server: Load and Query Iceberg with AI](https://iceberglakehouse.com/posts/conversational-ai-managed-iceberg-s3-tables-mcp-server/) (2026-07-06): AWS Labs' S3 Tables MCP server lets an AI agent create Iceberg tables, import CSV or Parquet files from S3, and append rows once --allow-write is enabled.
- [Decoupled Catalogs vs. Managed Tables: Architectural Freedom in the Age of Table Format Convergence](https://iceberglakehouse.com/posts/decoupled-catalogs-vs-managed-tables-table-format-convergence/) (2026-07-06): A decision matrix for choosing between managed tables and decoupled catalogs for Iceberg, covering credentials, optimization, engine choice, and hybrid setups.
- [Implementing Positional Deletes in Iceberg v3: Streamlining Merge-on-Read for Fast-Inbound Event Lakes](https://iceberglakehouse.com/posts/iceberg-v3-positional-deletes-deletion-vectors-event-lakes/) (2026-07-06): How positional deletes and deletion vectors support merge-on-read in Iceberg for late, duplicate, and corrected event data, plus compaction and engine checks.
- [Mapping the Variant Type in Iceberg v3: Standardizing Semi-Structured AI JSON Payloads](https://iceberglakehouse.com/posts/iceberg-v3-variant-type-ai-json-payloads/) (2026-07-06): How Iceberg v3 variant-style typing stores evolving AI payloads like prompts and tool calls, and when to promote JSON fields into governed, typed columns.
- [Multi-Engine Catalog Federation with Apache Polaris: Syncing Google Cloud, AWS, and Azure Metadata](https://iceberglakehouse.com/posts/multi-engine-catalog-federation-apache-polaris-multicloud/) (2026-07-06): How Apache Polaris federates Iceberg catalogs across AWS, Azure, and Google Cloud, covering identity mapping, scoped credentials, namespaces, and pitfalls.
- [Designing Private, Air-Gapped Data Lakehouses: Scaling Iceberg in Highly Secure, On-Premises Clouds](https://iceberglakehouse.com/posts/private-air-gapped-data-lakehouses-iceberg-secure-clouds/) (2026-07-06): How to run Iceberg lakehouses in air-gapped and on-prem clouds: storage, catalogs, engine placement, agent access, supply chain upgrades, audit, and recovery.
- [Lakehouse Table Formats in 2026: Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, How They Work, Where They Stand, and Where They're Going](https://iceberglakehouse.com/posts/table-formats-2026-breakdown/) (2026-07-06): How Iceberg, Delta Lake, Hudi, Paimon, and DuckLake work, how each one handles row-level change, and a practical framework for choosing a table format in 2026.
- [Mastering Apache Iceberg v3: What's New and How to Plan Your Upgrade](https://iceberglakehouse.com/posts/apache-iceberg-v3-upgrade/) (2026-05-28): Apache Iceberg v3 adds deletion vectors, VARIANT type, row lineage, and table encryption. Here's what changed and how to plan your upgrade.
- [How Apache Iceberg Resolves the Hybrid-Cloud Challenge in Heavily Regulated Markets](https://iceberglakehouse.com/posts/iceberg-hybrid-cloud-regulated-markets/) (2026-05-28): Apache Iceberg gives regulated enterprises data sovereignty with hybrid-cloud deployments. Learn how open catalogs and Iceberg handle compliance requirements.
- [Securing Apache Iceberg Tables with Fine-Grained Row and Column Level Access Control](https://iceberglakehouse.com/posts/iceberg-row-column-access-control/) (2026-05-28): Apache Iceberg doesn't enforce row and column security on its own. Learn how catalog-level governance with Apache Polaris and Dremio delivers enterprise FGAC.
- [Designing an Immutable Data Lakehouse: Best Practices for Iceberg Snapshot Expiration](https://iceberglakehouse.com/posts/iceberg-snapshot-expiration/) (2026-05-28): Iceberg snapshot bloat silently kills query performance. Learn the best practices for snapshot expiration, manifest cleanup, and automated table maintenance.
- [Decoupling Storage and Compute in Apache Iceberg: A Deep Dive into Cost Optimization](https://iceberglakehouse.com/posts/iceberg-storage-compute-decoupling/) (2026-05-28): Learn how Apache Iceberg decouples storage from compute to cut data platform costs, enable multi-engine routing, and eliminate proprietary lock-in.
- [The Endgame – Building an Autonomous Optimization Pipeline for Apache Iceberg](https://iceberglakehouse.com/posts/iceberg-autonomous-optimization-pipeline/) (2025-09-16): Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools.
- [Managing Large-Scale Optimizations – Parallelism, Checkpointing, and Fail Recovery](https://iceberglakehouse.com/posts/iceberg-large-scale-optimization/) (2025-09-09): Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability.
- [Hidden Pitfalls – Compaction and Partition Evolution in Apache Iceberg](https://iceberglakehouse.com/posts/iceberg-partition-evolution-compaction/) (2025-09-02): Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction.
- [Using Iceberg Metadata Tables to Determine When Compaction Is Needed](https://iceberglakehouse.com/posts/iceberg-metadata-triggered-compaction/) (2025-08-26): Discover how to use Apache Iceberg's metadata tables to proactively detect small files, bloated manifests, and table fragmentation - so you can trigger.
- [Designing the Ideal Cadence for Compaction and Snapshot Expiration](https://iceberglakehouse.com/posts/iceberg-optimization-cadence/) (2025-08-19): Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness.
- [Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests](https://iceberglakehouse.com/posts/iceberg-metadata-bloat-cleanup/) (2025-08-12): Learn how to prevent and clean up metadata bloat in Apache Iceberg by expiring snapshots and rewriting manifests for better performance and manageability.
- [Smarter Data Layout – Sorting and Clustering Iceberg Tables](https://iceberglakehouse.com/posts/iceberg-clustering-sorting-zorder/) (2025-08-05): How sorting and Z-order clustering lay out data in Apache Iceberg tables to improve file pruning and query speed, when to use each, and the rewrite costs.
- [Optimizing Compaction for Streaming Workloads in Apache Iceberg](https://iceberglakehouse.com/posts/iceberg-streaming-compaction/) (2025-07-29): Learn how to design fast, incremental compaction strategies in Apache Iceberg to support high-throughput streaming pipelines without disrupting freshness.
- [The Basics of Compaction – Bin Packing Your Data for Efficiency](https://iceberglakehouse.com/posts/iceberg-optimization-compaction-basics/) (2025-07-22): Learn how standard compaction works in Apache Iceberg and why bin packing your data files is essential for maintaining query performance and cost efficiency.
- [The Cost of Neglect – How Apache Iceberg Tables Degrade Without Optimization](https://iceberglakehouse.com/posts/iceberg-optimization-degradation/) (2025-07-15): Learn how Apache Iceberg tables can degrade over time without optimization and what issues this causes for performance, cost, and governance.

## Articles: Data Lakehouse Architecture

- [The Open Lakehouse Explained, Then Built on Your Laptop with Dremio and RustFS or Ceph](https://iceberglakehouse.com/posts/open-lakehouse-on-your-laptop/) (2026-09-10): The five layers of the open lakehouse explained, then a lab: Parquet, Iceberg, Polaris, Arrow, and Ossie running in two containers on your own machine.
- [Postgres Meets the Lakehouse: pg_lake, pg_duckdb, and When Postgres Is Enough](https://iceberglakehouse.com/posts/postgres-meets-the-lakehouse/) (2026-09-02): What pg_lake, pg_duckdb, and pg_mooncake do with Iceberg tables, how they compare on catalogs and performance, and honest thresholds for when Postgres suffices.
- [Building Apache Iceberg Lakehouses That Run Without an Internet Connection](https://iceberglakehouse.com/posts/building-air-gapped-iceberg-lakehouse/) (2026-08-04): How to build an Apache Iceberg lakehouse that runs fully offline: storage, catalog, compute, cross-zone transfer, compliance, and the failure modes that bite.
- [Building an Honest TCO Model for Open Lakehouses and Proprietary Warehouses](https://iceberglakehouse.com/posts/lakehouse-vs-warehouse-tco/) (2026-08-04): An honest TCO framework for open lakehouses versus proprietary warehouses: five cost categories, measured numbers, sensitivity analysis, and where each.
- [A Migration Playbook for Moving Legacy Warehouses onto Apache Iceberg](https://iceberglakehouse.com/posts/warehouse-to-iceberg-migration-playbook/) (2026-08-04): A dependency-first playbook for migrating legacy warehouses onto Apache Iceberg: snapshot vs migrate vs add_files, four-level parity validation.
- [Automated Materialized Views in Dremio](https://iceberglakehouse.com/posts/automated-materialized-views-autonomous-table-optimization-dremio-agentic-lakehouse/) (2026-07-13): How Dremio Reflections, Autonomous Reflections, and automatic Iceberg table optimization replace cron-based maintenance, and how to measure savings honestly.
- [Data Lakehouse Open Standards for AI Agents](https://iceberglakehouse.com/posts/data-lakehouse-open-standards-agentic-era/) (2026-07-13): Why AI agents expose data silos, and how Iceberg tables, REST catalogs, and federated query across object stores give agents one governed path to company data.
- [Dremio Lakehouse AI Report: Agentic Lessons](https://iceberglakehouse.com/posts/dremio-state-data-lakehouse-ai-report-agentic-lakehouse/) (2026-07-13): Directional signals from Dremio's State of the Data Lakehouse report on AI: semantic consistency, autonomous performance, governance, and an agent-ready plan.
- [Five-Layer Agentic Lakehouse Architecture](https://iceberglakehouse.com/posts/five-layer-agentic-lakehouse-architecture-enterprise-pattern/) (2026-07-13): A five-layer pattern for trusted agentic analytics (data, knowledge, agents, tools, policy), how one request moves through it, and a checklist for your stack.
- [Hybrid Lakehouse Design for Regulated Markets](https://iceberglakehouse.com/posts/hybrid-lakehouse-regulated-markets-on-prem-public-metadata/) (2026-07-13): How regulated firms keep data on-prem in Iceberg tables on private object storage while sharing approved metadata via Nessie or Polaris and read-only gateways.
- [Zero-Copy Mirroring to Open Iceberg Tables](https://iceberglakehouse.com/posts/zero-copy-mirroring-migration-proprietary-warehouses-open-iceberg/) (2026-07-13): When zero-copy mirroring to Iceberg works and when it does not: assessing schemas, translating files, rebuilding logic, staying fresh, and validating cutover.
- [Federation and the Lakehouse: Two Roads to Unified Data Access, and How to Know Which One to Take](https://iceberglakehouse.com/posts/federation-and-lakehouse/) (2026-07-06): When to federate queries across sources and when to consolidate into an Iceberg lakehouse, with pushdown mechanics, cost math, and a federate-first pattern.
- [The Who, What, and Why of Semantic Layers: The Layer That Decides Whether Your Numbers Can Be Trusted](https://iceberglakehouse.com/posts/who-what-why-semantic-layers/) (2026-07-06): What a semantic layer is, how it rewrites queries, who built the category, today's major tools, and how teams roll one out so AI agents and BI share one metric.
- [AI-Ready Metadata Prevents Query Failures](https://iceberglakehouse.com/posts/ai-ready-metadata-lineage-quality-llm-query-failures/) (2026-06-22): AI-ready metadata reduces query failures by making ownership, freshness, lineage, quality, and policy visible at execution time.
- [Autonomous Materialization for Agentic Analytics](https://iceberglakehouse.com/posts/autonomous-materialization-ai-agents-table-performance/) (2026-06-22): How to run autonomous materialization for agent query workloads: tie it to workload evidence, add governance checks, measure results, and avoid failure modes.
- [Composable Semantic Layers for Analytical Agents](https://iceberglakehouse.com/posts/beyond-metrics-lists-composable-semantic-layers-agents/) (2026-06-22): Why a metric catalog is not enough for AI agents, and how composable semantic layers keep business logic intact across multi-step analysis, with a rollout plan.
- [Built for Agents and Managed by Agents](https://iceberglakehouse.com/posts/built-for-agents-managed-by-agents-dremio-agentic-lakehouse/) (2026-06-22): Dremio Agentic Lakehouse is easiest to understand as two ideas: data built for agent access and platform work managed by agents.
- [ClickHouse in the Loop for Active Agents](https://iceberglakehouse.com/posts/clickhouse-loop-real-time-event-streams-active-agents/) (2026-06-22): Low-latency analytical systems can help active agents, but only when event loops include validation, context, and safety boundaries.
- [The Context Layer for AI Agents](https://iceberglakehouse.com/posts/context-layer-semantic-lineage-quality-ai-agents/) (2026-06-22): AI agents need more than a semantic layer. Learn how to add lineage, quality, freshness, compliance, and ownership context, with guardrails and a rollout plan.
- [Lakehouse as the Operating Layer for Agentic AI](https://iceberglakehouse.com/posts/databricks-summit-2026-lakehouse-agentic-ai-operating-layer/) (2026-06-22): Agentic AI announcements are useful when they validate the need for governed data, semantic context, and cost-aware execution.
- [Event-Driven Table Compaction with Agents](https://iceberglakehouse.com/posts/event-driven-table-compaction-agentic-coordination/) (2026-06-22): Event-driven compaction is valuable when agents coordinate maintenance with workload signals, table health, and commit safety.
- [Fabric Agentic Analytics and Lakehouse Schema Design](https://iceberglakehouse.com/posts/fabric-build-2026-lakehouse-schema-agentic-analytics/) (2026-06-22): Microsoft Fabric agentic analytics is a reminder that schemas, semantic models, and governed lakehouse design now shape AI behavior.
- [Iceberg v4 Performance: Root Manifests and Calls](https://iceberglakehouse.com/posts/iceberg-v4-performance-root-manifests-combined-calls/) (2026-06-22): Apache Iceberg v4 discussion should focus on planning cost, metadata layout, and object storage round trips, not vague claims about faster tables.
- [What Is LTAP in the Lakehouse?](https://iceberglakehouse.com/posts/ltap-lakehouse-transactional-analytical-processing-2026/) (2026-06-22): Lakehouse transactional analytical processing is useful only when teams define freshness, isolation, and workload boundaries clearly.
- [The Model Is Not the Moat](https://iceberglakehouse.com/posts/model-not-moat-semantic-lakehouse-layer-enterprise-ai/) (2026-06-22): Enterprise AI advantage increasingly comes from governed context, semantic models, and operational data contracts, not only from model choice.
- [PyIceberg at Scale Without Apache Spark](https://iceberglakehouse.com/posts/pyiceberg-scale-high-concurrency-no-spark/) (2026-06-22): Where PyIceberg works without Spark, such as serverless functions appending small batches, where it does not, and what to measure before scaling it up.
- [The Real-Time Lakehouse with Streaming and Iceberg](https://iceberglakehouse.com/posts/real-time-lakehouse-streaming-sql-cold-iceberg-storage/) (2026-06-22): The real-time lakehouse is not one engine. It is a contract between streams, table commits, query paths, and freshness expectations.
- [REST Catalog V2 LoadTable and Client Capability](https://iceberglakehouse.com/posts/rest-catalog-v2-loadtable-protocol-design/) (2026-06-22): REST Catalog V2 LoadTable work matters because clients and catalogs need explicit contracts, not optimistic assumptions.
- [Rust vs C++ in Native Iceberg Scan Operators](https://iceberglakehouse.com/posts/rust-vs-cpp-table-layer-native-scan-operators/) (2026-06-22): The Rust versus C++ discussion is really about table-layer execution safety, interoperability, and performance envelopes.
- [Server-Side Commit Deconflicting in REST Catalogs](https://iceberglakehouse.com/posts/server-side-commit-deconflicting-rest-catalogs/) (2026-06-22): Why per-client retry loops break down with many writers, and how server-side commit deconflicting in Iceberg REST catalogs moves concurrency control there.
- [Snowflake Interoperable Lakehouse Lessons](https://iceberglakehouse.com/posts/snowflake-interoperable-lakehouse-production-lessons/) (2026-06-22): Interoperable lakehouse announcements matter when they change production contracts, not just import and export narratives.
- [Unity AI Gateway vs Apache Polaris Control Planes](https://iceberglakehouse.com/posts/unity-ai-gateway-vs-apache-polaris-governance-control-plane/) (2026-06-22): Unity AI Gateway and Apache Polaris govern different parts of the analytics path. Compare a closed AI gateway with an open catalog control plane, and their fit.
- [Evaluating the TCO of an Open Lakehouse vs. Proprietary Data Warehouses](https://iceberglakehouse.com/posts/open-lakehouse-vs-proprietary-warehouse-tco/) (2026-05-28): Open lakehouse vs proprietary warehouse: a comprehensive TCO breakdown covering storage, compute, engineering, and hidden costs to help you make the right.
- [The Data Lakehouse - The Benefits and Enhancing Implementation](https://iceberglakehouse.com/posts/2025-01-the-data-lakehouse-benefits-and-enhancing/) (2025-01-31): Problems a lakehouse solves, from consistency and cost to governance, the gaps that remain, and how Dremio closes them with a semantic layer and Reflections.
- [When to use Apache Xtable or Delta Lake Uniform for Data Lakehouse Interoperability](https://iceberglakehouse.com/posts/2025-01-xtable-or-uniform/) (2025-01-07): How Apache XTable and Delta Lake UniForm differ for table format interoperability, which fits joins, streaming, and migration use cases, and their limits.
- [Deep Dive into Dremio's File-based Auto Ingestion into Apache Iceberg Tables](https://iceberglakehouse.com/posts/2024-11-deep-dive-auto-ingest-dremio-iceberg/) (2024-11-15): How to load JSON, CSV, and Parquet files into Iceberg tables with Dremio Auto-Ingest pipes: syntax, file format options, deduplication, and error handling.
- [Dremio, Apache Iceberg and their role in AI-Ready Data](https://iceberglakehouse.com/posts/2024-11-dremio-and-ai-ready-data/) (2024-11-05): What makes data AI-ready, how Iceberg features like schema evolution and time travel support it, and how Dremio speeds data prep for AI workloads.
- [Hands-on with Apache Iceberg & Dremio on Your Laptop within 10 Minutes](https://iceberglakehouse.com/posts/2024-10-hands-on-with-iceberg-dremio-laptop/) (2024-10-31): Run Dremio, Nessie, and MinIO with Docker Compose in about 10 minutes, create Iceberg tables, run transactions, and inspect the files and metadata.
- [All About Parquet Part 01 - An Introduction](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-01/) (2024-10-21): An introduction to Apache Parquet: how it differs from row formats like CSV and JSON, key features such as compression and schema evolution, and when to use it.
- [All About Parquet Part 02 - Parquet's Columnar Storage Model](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-02/) (2024-10-21): How Parquet's columnar storage model works and why it speeds analytics through faster scans, better compression, and quick aggregation, plus when to avoid it.
- [All About Parquet Part 03 - Parquet File Structure | Pages, Row Groups, and Columns](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-03/) (2024-10-21): Inside a Parquet file: how row groups, column chunks, and pages are laid out, what footer metadata stores, and how their sizes affect query performance.
- [All About Parquet Part 04 - Schema Evolution in Parquet](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-04/) (2024-10-21): How Parquet handles schema evolution when you add, remove, retype, rename, or reorder columns, how it compares to other formats, and best practices.
- [All About Parquet Part 05 - Compression Techniques in Parquet](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-05/) (2024-10-21): The compression codecs Parquet supports (Snappy, Gzip, Brotli, Zstandard, LZO), the speed and size tradeoffs of each, and how they pair with encoding.
- [All About Parquet Part 06 - Encoding in Parquet | Optimizing for Storage](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-06/) (2024-10-21): How Parquet encodings work and when to use them: dictionary, run-length, bit-packing, delta, and plain, and how encoding pairs with compression.
- [Parquet Metadata Explained: Footer, Row Groups, Statistics](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-07/) (2024-10-21): Parquet metadata lives in the file footer: the schema, row group layout and per-column min/max statistics that let query engines skip data they do not need.
- [Read and Write Parquet Files in Python (PyArrow, Pandas)](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-08/) (2024-10-21): How to read and write Parquet files in Python with PyArrow, FastParquet, and Pandas, including column selection, compression, and partitioned datasets.
- [All About Parquet Part 09 - Parquet in Data Lake Architectures](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-09/) (2024-10-21): Why Parquet is the default file format for data lakes, how it underpins Iceberg, Delta Lake, and Hudi, and best practices for partitioning and pushdown.
- [Parquet Performance Tuning: 8 Best Practices](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-10/) (2024-10-21): Tune Apache Parquet for speed and cost: row group and file sizes, partitioning, Snappy vs ZSTD compression, encodings, sorting, and predicate pushdown.
- [A Guide to dbt Macros - Purpose, Benefits, and Usage](https://iceberglakehouse.com/posts/2024-10-a-guide-to-dbt-macros/) (2024-10-18): What dbt macros are and how they cut repeated SQL: defining them, calling them in models, generating SQL with variables, and keeping them maintainable.
- [Getting Started with Data Analytics Using PyArrow in Python](https://iceberglakehouse.com/posts/2024-10-getting-started-with-pyarrow-in-python/) (2024-10-15): A practical PyArrow guide: core objects like Tables and RecordBatches, reading and writing Parquet, running computations, and using Arrow Flight.
- [What is Three-Tier Data (Bronze, Silver, Gold) and How Dremio Simplifies It](https://iceberglakehouse.com/posts/2024-10-bronze-silver-gold-data/) (2024-10-09): What the bronze, silver, and gold data tiers are, the copy and sync costs of moving data between them, and how Dremio views and Reflections cut that work.
- [A Brief Guide to the Governance of Apache Iceberg Tables](https://iceberglakehouse.com/posts/2024-10-governing-apache-iceberg-tables/) (2024-10-07): How access control for Apache Iceberg tables works at the file, engine, and catalog level, and how Project Nessie and Apache Polaris governance compare.
- [Exploring Data Operations with PySpark, Pandas, DuckDB, Polars, and DataFusion in a Python Notebook](https://iceberglakehouse.com/posts/2024-10-exploring-data-operations-python/) (2024-10-07): Hands-on examples of loading and querying data with PySpark, Pandas, DuckDB, Polars, and DataFusion in one Docker notebook, plus querying Dremio.
- [Ultimate Directory of Apache Iceberg Resources](https://iceberglakehouse.com/posts/2024-10-ultimate-directory-of-apache-iceberg-resources/) (2024-10-05): A directory of Apache Iceberg learning material by topic: architecture, ingestion, migration, streaming, partitioning, maintenance, and catalogs.
- [Change Data Capture (CDC) when there is no CDC](https://iceberglakehouse.com/posts/2024-10-cdc-when-there-is-no-cdc/) (2024-10-04): How to sync changing data when a source has no native CDC: timestamp and version columns, SQL to find changes, MERGE, and scheduled incremental batches.
- [Virtualization + Lakehouse + Mesh = Data At Scale](https://iceberglakehouse.com/posts/2024-9-virtualization-lakehouse-mesh-data-at-scale/) (2024-09-25): How virtualization, a lakehouse, and data mesh ideas combine into one point of access with data curated by many domain teams, and where Dremio fits.
- [Why Data Analysts, Engineers, Architects and Scientists Should Care about Dremio and Apache Iceberg](https://iceberglakehouse.com/posts/2024-9-why-dremio-iceberg-matters/) (2024-09-10): How Dremio and Apache Iceberg address the daily problems of data engineers, architects, analysts, and data scientists, from data silos to governance.
- [Hands-on with Apache Iceberg on Your Laptop - Deep Dive with Apache Spark, Nessie, Minio, Dremio, Polars and Seaborn](https://iceberglakehouse.com/posts/2024-9-hands-on-iceberg-dremio-minio-nessie/) (2024-09-10): Build an Iceberg lakehouse on your laptop: ingest with Spark into MinIO, track tables in Nessie, query in Dremio, then analyze with Polars and Seaborn.
- [5 Trends in the Data Lakehouse Space](https://iceberglakehouse.com/posts/2024-9-five-trends-in-data-lakehouse/) (2024-09-01): Five lakehouse trends: storage vendors adding warehouse features, table format convergence, managed Iceberg upkeep, catalog wars, and unified platforms.
- [Using the alexmerced/datanotebook Docker Image](https://iceberglakehouse.com/posts/2024-8-using-the-alexmerced-datanotebook-image/) (2024-08-30): How to use the alexmerced/datanotebook Docker image for quick Python data work: bundled libraries, saving your work, and connecting to Spark and MinIO.
- [Understanding Apache Iceberg Delete Files](https://iceberglakehouse.com/posts/2024-8-understanding-apache-iceberg-delete-files/) (2024-08-29): Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.
- [Understanding the Apache Iceberg Manifest](https://iceberglakehouse.com/posts/2024-8-understanding-apache-iceberg-manifest/) (2024-08-27): Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture.
- [Understanding the Apache Iceberg Manifest List (Snapshot)](https://iceberglakehouse.com/posts/2024-8-understanding-apache-iceberg-manifest-list/) (2024-08-25): Continuing the Understand Apache Iceberg series, this article delves into the Manifest List, a critical component of Apache Iceberg's architecture.
- [Understanding Apache Iceberg's Metadata.json](https://iceberglakehouse.com/posts/2024-8-apache-iceberg-metadata-json/) (2024-08-21): A field-by-field walkthrough of Iceberg's metadata.json, covering format version, schemas, partition specs, and snapshots, and how it enables time travel.
- [What Apache Iceberg REST Catalog is and isn't](https://iceberglakehouse.com/posts/2024-8-what-apache-iceberg-rest-catalog-is-and-isnt/) (2024-08-18): What the Iceberg REST catalog spec covers, a uniform API for table operations, and what it does not: it is not a catalog and does not standardize security.
- [ACID Guarantees and Apache Iceberg - Turning Any Storage into a Data Warehouse](https://iceberglakehouse.com/posts/2024-8-acid-guarantees-and-apache-iceberg/) (2024-08-15): What atomicity, consistency, isolation, and durability mean, how warehouses provide them, and how Apache Iceberg brings ACID transactions to data lakes.
- [Data Lakehouse 101 - The Who, What and Why of Data Lakehouses](https://iceberglakehouse.com/posts/2024-8-data-lakehouses-101/) (2024-08-05): What a data lakehouse is, why it costs less than copying data into warehouses, a three-step migration path with Dremio, and seven signs it is time to switch.
- [Apache Iceberg Reliability](https://iceberglakehouse.com/posts/2024-7-apache-iceberg-reliability/) (2024-07-26): How Apache Iceberg avoids the consistency problems of Hive tables on S3 with a persistent metadata tree, atomic commits, and serializable isolation.
- [Understanding the Polaris Iceberg Catalog and Its Architecture](https://iceberglakehouse.com/posts/2024-7-understanding-polaris-apache-iceberg-catalog/) (2024-07-20): An overview of the Polaris Iceberg catalog: its entities, internal versus external catalogs, and how principals and roles control access to your tables.
- [Databases Deconstructed - The Value of Data Lakehouses and Table Formats](https://iceberglakehouse.com/posts/2024-7-databases-decontstructed-value-of-data-lakehouses-and-table-formats/) (2024-07-12): How a lakehouse splits a database into layers (Parquet storage, an Iceberg table format, a catalog, and query engines) and why that modularity matters.
- [Partitioning with Apache Iceberg - A Deep Dive](https://iceberglakehouse.com/posts/2024-5-partitioning-with-apache-iceberg-deep-dive/) (2024-05-29): How Apache Iceberg partitioning works under the hood: hidden partitioning, time, bucket, and truncate transforms, and partition evolution versus Hive.
- [3 Reasons Data Engineers Should Embrace Apache Iceberg](https://iceberglakehouse.com/posts/2024-5-3-reasons-data-engineers-should-embrace-apache-iceberg/) (2024-05-15): Three reasons data engineers adopt Apache Iceberg: hidden partitioning, partition evolution without rewrites, and wide engine support, plus labs to try.
- [Understanding the Future of Apache Iceberg Catalogs](https://iceberglakehouse.com/posts/2024-4-understanding-the-future-of-apache-iceberg-catalogs/) (2024-04-04): What an Apache Iceberg catalog does, why Java-only catalog implementations caused friction across engines, and how the REST catalog spec fixes that.
- [End-to-End Basic Data Engineering Tutorial (Spark, Dremio, Superset)](https://iceberglakehouse.com/posts/2024-4-end-to-end-data-engineering-tutorial-spark-dremio-superset/) (2024-04-01): A beginner tutorial: move Postgres data into an Iceberg data lake with Spark, curate and serve it through Dremio, then build a BI dashboard in Superset.
- [5 Reasons Dremio is the Ideal Apache Iceberg Lakehouse Platform](https://iceberglakehouse.com/posts/2024-3-5-reasons-dremio-is-the-ideal-iceberg-lakehouse-platform/) (2024-03-09): Five Dremio features that matter for Iceberg lakehouses: dataset promotion, Reflections, first-class Iceberg support, catalog versioning, and table upkeep.
- [The Apache Iceberg Lakehouse - The Great Data Equalizer](https://iceberglakehouse.com/posts/2024-3-06-apache-iceberg-the-great-data-equalizer/) (2024-03-06): How Apache Iceberg undercut the Snowflake and Databricks lock-in, how each vendor answered with partial Iceberg support, and what a fully open lakehouse offers.
- [10 Reasons to Make Apache Iceberg and Dremio Part of Your Data Lakehouse Strategy](https://iceberglakehouse.com/posts/2024-3-10-reasons-to-make-dremio-part-of-your-data-lakehouse-strategy/) (2024-03-01): Five reasons to pick Apache Iceberg, such as partition evolution and versioning, and five for Dremio, from Arrow and C3 caching to Reflections and hybrid use.
- [A deep dive into the concept and world of Apache Iceberg Catalogs](https://iceberglakehouse.com/posts/2024-3-deep-dive-into-apache-iceberg-catalogs/) (2024-03-01): How Apache Iceberg catalogs work, service versus file-system catalogs, and a tour of Nessie, Hive, REST, Glue, Snowflake, and LakeFS to help you pick one.
- [What is the Data Lakehouse and the Role of Apache Iceberg, Nessie and Dremio?](https://iceberglakehouse.com/posts/2024-2-what-a-data-lakehouse-and-role-of-dremio-iceberg-nessie/) (2024-02-21): What a data lakehouse is, why table formats matter, and the role of each piece: Apache Iceberg for tables, Nessie for versioned catalogs, Dremio for queries.
- [Partitioning Practices in Apache Hive and Apache Iceberg](https://iceberglakehouse.com/posts/2024-2-partitioning-in-apache-hive-and-apache-iceberg/) (2024-02-12): How Hive-style partitioning works and where it breaks down, and how Iceberg's hidden partitioning, partition transforms, and partition evolution fix it.
- [Embracing the Future of Data Management - Why Choose Lakehouse, Iceberg, and Dremio?](https://iceberglakehouse.com/posts/2024-1-why-choose-lakehouse-iceberg-dremio/) (2024-01-25): Why the lakehouse is replacing separate lakes and warehouses, what shifting left means, and how Apache Iceberg and Dremio fit together in an open architecture.
- [Open Lakehouse Engineering/Apache Iceberg Lakehouse Engineering - A Directory of Resources](https://iceberglakehouse.com/posts/2024-1-open-lakehouse-engineering-resources/) (2024-01-19): A curated list of getting-started guides, hands-on tutorials, and conceptual articles for building an open lakehouse with Apache Iceberg, Nessie, and Arrow.
- [Nessie - An Alternative to Hive & JDBC for Self-Managed Apache Iceberg Catalogs](https://iceberglakehouse.com/posts/2024-1-nessie-an-alternative-to-hive/) (2024-01-08): Why Project Nessie is a self-managed Iceberg catalog alternative to Hive Metastore and JDBC, what it adds with Git-like versioning, and how to try it hands-on.
- [Apache Iceberg, Git-Like Catalog Versioning and Data Lakehouse Management - Pillars of a Robust Data Lakehouse Platform](https://iceberglakehouse.com/posts/2024-1-apache-iceberg-git-life-catalog-versioning/) (2024-01-03): This is where the combined power of Dremio’s Lakehouse Management features and Project Nessie's catalog-level versioning comes into play.

## Articles: Agentic Analytics and Semantic Layers

- [How Apache Ossie Is Deciding What Agents and BI Tools Can Ask a Semantic Layer](https://iceberglakehouse.com/posts/ossie-semantic-query-interface-agents/) (2026-09-21): How Apache Ossie's layered query design gives AI agents both a constrained dimensional interface and a grain-safe SQL interface for semantic layers.
- [Apache Ossie and Apache Polaris: Putting Semantic Models in the Open Catalog](https://iceberglakehouse.com/posts/apache-ossie-polaris-semantic-models/) (2026-08-25): Apache Ossie and Polaris put metric definitions in the open catalog. What the spec covers, what Polaris stores, and what is still unfinished.
- [Semantic Layer Federation: One Logical Model Over Data on Three Clouds](https://iceberglakehouse.com/posts/semantic-layer-federation-three-clouds/) (2026-08-25): One logical model over Iceberg and databases on three clouds. Pushdown, egress, Reflections, and where semantic federation still breaks.
- [GSA MCP Servers for Open Data AI Agents](https://iceberglakehouse.com/posts/gsa-model-context-protocol-server-hackathon-open-data-ai-agents/) (2026-07-13): Why federal open data needs agent-ready interfaces, what MCP adds over legacy APIs, the GSA hackathon as a signal, and a blueprint for a read-only SQL MCP tool.
- [MCP Stateless Gateways for Data Agents](https://iceberglakehouse.com/posts/mcp-stateless-specifications-scalable-data-gateways-ai-agents/) (2026-07-13): Why stateful MCP servers fail to scale under agent load, and how stateless gateways handle credential delegation, context isolation, and database guardrails.
- [The Buyer's Scorecard for Agentic Analytics: Evaluating Tooling in the Enterprise AI Era](https://iceberglakehouse.com/posts/buyers-scorecard-agentic-analytics-enterprise-ai-era/) (2026-07-06): An eight-category scorecard for judging agentic analytics tools: semantic grounding, governance, validation, action controls, cost, and pilot questions.
- [Designing Idempotent Pipelines in the Agentic Lakehouse: Eliminating Double-Write Anomalies](https://iceberglakehouse.com/posts/idempotent-pipelines-agentic-lakehouse-double-write-anomalies/) (2026-07-06): How to stop agent retries and reruns from writing duplicates in an Iceberg lakehouse, using operation IDs, intent logs, deterministic writes, and validation.
- [The Metric Contract Mandate: Standardizing Semantic Layers Before AI Agent Access](https://iceberglakehouse.com/posts/metric-contract-mandate-semantic-layers-ai-agent-access/) (2026-07-06): What a metric contract contains (definition, grain, owner, freshness, tests, access) and why a semantic layer needs one before AI agents are allowed to query.
- [Trustworthy Concurrency in the Agentic Lakehouse: Reconciling Academic Proofs with High-Frequency Production Writes](https://iceberglakehouse.com/posts/trustworthy-concurrency-agentic-lakehouse-production-writes/) (2026-07-06): How Iceberg optimistic concurrency behaves under many agent writers, and the retry discipline, isolation contracts, write leases, and queues that keep it sane.

## Articles: AI and Agents

- [Turning an Analytics Question Into a Verified Agentic Graph](https://iceberglakehouse.com/posts/ags-multi-agent-analytics-workflow/) (2026-09-28): A complete AGS 1.0 graph for governed metric questions, with verification gates, deterministic check scripts, and reconciliation against a semantic layer.
- [Twelve Queries That Pass Every Schema Check and Still Return the Wrong Number](https://iceberglakehouse.com/posts/semantic-failure-catalog-ai-agent-queries/) (2026-09-28): Twelve query patterns that pass schema validation and still return wrong numbers, with detectors, required metadata, and tested code for each.
- [Fast Classification Models, LLMs, and the Apache Iceberg Lakehouse](https://iceberglakehouse.com/posts/jev-classification-models-iceberg-lakehouse/) (2026-09-21): How fast classification models like Jev and GLiClass compare with LLMs on fixed-label tasks, and how to run both together over Apache Iceberg tables at scale.
- [Agentic Data Architecture](https://iceberglakehouse.com/posts/agentic-data-architecture/) (2026-09-10): A six-layer reference architecture for agents on company data: planners, tool boundaries, identity, the semantic layer, and what breaks when a layer is missing.
- [Context Engineering for Data Agents](https://iceberglakehouse.com/posts/context-engineering-for-data-agents/) (2026-09-10): Why text-to-SQL accuracy collapses on enterprise schemas, the five kinds of context an agent needs, where each one hides, and how to make the semantics legible.
- [The Data Team of the Agentic Era: Generalists Owning End-to-End Workflows](https://iceberglakehouse.com/posts/data-team-of-the-agentic-era/) (2026-09-02): The case for generalists owning end-to-end data workflows with agents, the counterargument, and how to make the transition work.
- [Will AI Replace Data Engineers?](https://iceberglakehouse.com/posts/will-ai-replace-data-engineers/) (2026-09-02): What the evidence shows about whether AI replaces data engineers, which parts of the job compress, and which parts do not.
- [Securing the Agentic Lakehouse Gateway: Preventing Prompt Injection and Data Exfiltration](https://iceberglakehouse.com/posts/agentic-lakehouse-gateway-prompt-injection/) (2026-08-25): Agentic lakehouse gateways face prompt injection and exfiltration through query results. A threat model and defenses for the layer in front of data.
- [Metric Contracts in Code: Testing, Versioning, and Serving Business Logic to Multi-Agent Systems](https://iceberglakehouse.com/posts/metric-contracts-code-multi-agent/) (2026-08-25): Metric contracts in code let teams test, version, and serve business logic to multi-agent systems without each agent inventing its own SQL.
- [The Agent Is Now a Named Coworker, and It Needs a File Format](https://iceberglakehouse.com/posts/agents-with-personalities/) (2026-08-24): Named, persistent agents need a file format. Open Agent Profile, Buzz, Grok Bot, and Hermes Bot Mode show why a portable agent identity matters.
- [Your Agent Should Answer the Phone: A Field Guide to AI Gateways on Slack, Discord, Telegram, Signal, and Teams](https://iceberglakehouse.com/posts/ai-gateways-field-guide/) (2026-08-24): A field guide to AI gateways on Slack, Discord, Telegram, Signal, and Teams: architecture, auth, cost, and the failure modes that matter.
- [Graphs in AI Engineering Have Solved Three Problems. The Fourth Is the Plan.](https://iceberglakehouse.com/posts/graphs-in-ai-engineering/) (2026-08-24): Knowledge graphs, GraphRAG, and LangGraph solved three problems. The fourth is the work itself: a reviewable graph of bounded agentic loops.
- [The Five Layers of an Agentic Lakehouse](https://iceberglakehouse.com/posts/five-layers-agentic-lakehouse/) (2026-08-19): The five layers of an agentic lakehouse: Storage, Catalog, Semantic, Gateway, and Agent Surface, and how one question travels through all of them.
- [Goal-Directed Data Quality Agents: Anomaly Quarantine on Apache Iceberg](https://iceberglakehouse.com/posts/goal-directed-data-quality-agents-iceberg/) (2026-08-19): Goal-directed data quality agents that watch Apache Iceberg tables, detect anomalies, and quarantine suspect data safely with snapshot isolation and branches.
- [Managing the TCO of Agentic Analytics: Token Budgets, Query Throttles, and the Economics of Autonomy](https://iceberglakehouse.com/posts/managing-tco-agentic-analytics/) (2026-08-19): Managing the total cost of agentic analytics: token budgets, query throttles, unit economics, and the FinOps discipline that keeps AI spend under control.
- [Metric Contracts in 2026: Standardizing Business Logic Across Multi-Agent Frameworks](https://iceberglakehouse.com/posts/metric-contracts-2026/) (2026-08-19): Metric contracts in 2026: versioned, testable definitions of business logic that let multi-agent frameworks compute revenue identically, with OSI interchange.
- [Query Routing at Machine Scale: Multi-Engine Workload Distribution for the Agentic Lakehouse](https://iceberglakehouse.com/posts/query-routing-machine-scale/) (2026-08-19): Query routing at machine scale for the agentic lakehouse: engine selection, acceleration substitution, admission control, and placement.
- [Building Stateless AI Tool Gateways with FastMCP, the 2026 MCP Spec, and Kubernetes](https://iceberglakehouse.com/posts/stateless-mcp-tool-gateways-fastmcp/) (2026-08-19): How to build stateless AI tool gateways with FastMCP, the 2026 MCP specification, and Kubernetes, and why statelessness finally makes MCP scale.
- [The Plan and the Worker: Two Open Specifications for Agent Harnesses](https://iceberglakehouse.com/posts/agentic-graph-open-agent-profile-two-open-specs-agent-harnesses/) (2026-08-10): Two open specifications, the Agentic Graph Specification and the Open Agent Profile, turn agent plans and agent identity into portable, reviewable files.
- [Budgeting for Agentic Analytics When Every Question Costs Something Different](https://iceberglakehouse.com/posts/agentic-analytics-tco-token-budgets/) (2026-08-04): Budgeting for agentic analytics when every question costs something different: token economics, query economics, instrumentation, and the cost controls.
- [The Five Layers of an Agentic Lakehouse and Where the MCP Server Sits](https://iceberglakehouse.com/posts/agentic-lakehouse-mcp-architecture/) (2026-08-04): The five layers of an agentic lakehouse and where the MCP server sits: storage, catalog, semantic layer, MCP gateway, and agent surface, plus identity.
- [Wiring Analytical Queries to Transactional APIs in Closed-Loop Decision Agents](https://iceberglakehouse.com/posts/closed-loop-decision-agents/) (2026-08-04): How decision agents can act on analytical reads through transactional APIs safely: conditional writes, sagas with compensations, decision records, blast limits.
- [Why Agentic AI Needs a Governed Semantic Layer Behind the Model Context Protocol](https://iceberglakehouse.com/posts/mcp-governed-semantic-layer/) (2026-08-04): Why agentic AI needs a governed semantic layer behind the Model Context Protocol: metric consistency, access control, Apache Ossie for portable.
- [Moving From Supply Chain Dashboards to Decision Loops With the Model Context Protocol](https://iceberglakehouse.com/posts/mcp-supply-chain-decision-loops/) (2026-08-04): Moving from supply chain dashboards to decision loops with MCP: sense, decide, act, and verify, with typed action tools, idempotency keys, and graduated.
- [Metric Contracts as the Interface AI Agents Actually Need](https://iceberglakehouse.com/posts/metric-contracts-for-ai-agents/) (2026-08-04): Metric contracts as the interface AI agents need: calculation, inclusion rules, grain, temporal semantics, ownership, semantic versioning, and testing.
- [Designing Policy-Aware Telemetry Tables for AI Systems in Apache Iceberg](https://iceberglakehouse.com/posts/policy-aware-ai-telemetry-iceberg/) (2026-08-04): Designing policy-aware AI telemetry tables in Apache Iceberg: what to log, tamper evidence, retention against conflicting deletion requirements.
- [Defending the Lakehouse Gateway Against Prompt Injection and Data Exfiltration](https://iceberglakehouse.com/posts/preventing-prompt-injection-lakehouse-gateway/) (2026-08-04): Defending the lakehouse gateway against prompt injection and data exfiltration: per-user identity, no-SQL tool surfaces, volume bounds, and detection.
- [Guardrails for Analytics Agents That Do More Than Answer Questions](https://iceberglakehouse.com/posts/agent-guardrails-human-in-the-loop/) (2026-07-28): How to put guardrails on analytics agents that take actions: classify by blast radius, gate capabilities, build approval steps, and stop runaway loops.
- [Building Agent Telemetry Tables in Iceberg That Survive an Audit](https://iceberglakehouse.com/posts/agent-telemetry-iceberg-audit/) (2026-07-28): A practical guide to building agent decision traces in Apache Iceberg that support audit reconstruction, governance review, and cost attribution.
- [What Agentic Analytics Actually Costs, and How to Keep It Bounded](https://iceberglakehouse.com/posts/agentic-analytics-tco/) (2026-07-28): Why agentic analytics produces two bills, engine compute and model tokens, how to attribute and forecast each one, and which levers actually lower total cost.
- [The Five Layers Between Your Lakehouse and a Trustworthy Agent](https://iceberglakehouse.com/posts/five-layer-agentic-lakehouse/) (2026-07-28): Agent reliability is a property of the stack the model sits on. Five layers with distinct owners and failure modes turn the agent is unreliable.
- [Governing What Agents Cost You](https://iceberglakehouse.com/posts/agent-cost-governance/) (2026-07-25): Agents break the four assumptions analytics platforms were built on. A practical guide to identity, budgets, semantic layers, caching, and instrumentation.
- [The Model Class Enterprise Data Teams Are Not Using Yet](https://iceberglakehouse.com/posts/tabular-foundation-models/) (2026-07-25): How tabular foundation models predict on structured rows in one forward pass with no training run, where they beat gradient boosting and LLMs, and licensing.
- [The Own Goal: What the Hugging Face Breach Reveals About Why Open Models Are a Security Necessity](https://iceberglakehouse.com/posts/value-of-open-models-security/) (2026-07-24): How the 2026 Hugging Face breach proved that open models are a security necessity, closed models failed to defend, an open Chinese model succeeded.
- [Active Analytics Loops for AI Action Agents](https://iceberglakehouse.com/posts/active-analytics-loop-goal-directed-action-agents/) (2026-07-13): The seven steps of an active analytics loop where agents watch signals, validate causes, and act with approval, plus tool design, guardrails, and case studies.
- [The Chase-Catch Gap in Enterprise AI Agents](https://iceberglakehouse.com/posts/chase-catch-gap-agentic-ai-enterprise-scale-2026/) (2026-07-13): Why enterprise AI agent pilots stall between demo and production, and how a knowledge layer, readiness metrics, and lakehouse architecture help close that gap.
- [Semantic View Autopilot for AI Governance](https://iceberglakehouse.com/posts/semantic-view-autopilot-dynamic-metric-definition-governance/) (2026-07-13): Where AI-drafted metric definitions and semantic views help, where they go wrong, and how lineage, programmatic review, and accuracy checks keep them honest.
- [Agentic AI Standards in 2026: MCP vs A2A and the Full Stack](https://iceberglakehouse.com/posts/state-of-agentic-ai-standards-2026/) (2026-07-06): MCP connects agents to tools; A2A connects agents to each other. A 2026 map of agentic AI standards: MCP, A2A, WebMCP, Apache Ossie (OSI), and agent payments.
- [Building Closed-Loop Decision Agents: Moving from Passive BI Dashboards to Active Goal-Directed Workflows](https://iceberglakehouse.com/posts/closed-loop-decision-agents-passive-bi-active-workflows/) (2026-07-06): How to build decision agents that observe signals, validate evidence, and act within approved limits, using metric contracts, risk tiers, and human approval.
- [Designing Your Own AI Harness: A Deep Dive Into the Architecture of Agent Loops, Tools, Context, and Control](https://iceberglakehouse.com/posts/designing-your-own-ai-harness/) (2026-07-06): A deep dive into custom AI harness architecture: model layers, tool design, context management, permissions, control budgets, persistence, orchestration.
- [Enforcing Fine-Grained Security at Machine Speed: Dynamic Access Control for High-Frequency AI Agents](https://iceberglakehouse.com/posts/fine-grained-security-machine-speed-ai-agents/) (2026-07-06): How to secure high-frequency AI agents with dynamic access control: row, column, and tool policies, short-lived credentials, token exchange, and audit trails.
- [MCP 2026-07-28 Spec: Stateless MCP for Data Platforms](https://iceberglakehouse.com/posts/mcp-2026-07-28-stateless-spec-data-platforms/) (2026-07-06): MCP 2026-07-28 shipped July 28, 2026: no initialize handshake, no session IDs, a new server/discover call. What stateless MCP means for data platform tools.
- [Personal Context vs. Shared Context: A Deep Dive Into How Humans and Organizations Should Feed Their AI Agents](https://iceberglakehouse.com/posts/personal-vs-shared-context/) (2026-07-06): How personal context for one user and governed shared context for an organization feed AI agents, with optimization techniques, failure cases, and frameworks.
- [When Gatekeepers Panic: The Encyclopédie, Open AI Models, and the Politics of Accessible Knowledge](https://iceberglakehouse.com/posts/when-gatekeepers-panic/) (2026-07-06): The fight over open AI models mirrors the 18th-century suppression of Diderot's Encyclopédie, revealing the same pattern of institutional fear.
- [Hermes Agent Setup: DeepSeek V4 and Slack Socket Mode](https://iceberglakehouse.com/posts/2026-05-hermes-agent-free-deepseek-setup/) (2026-05-25): Set up Hermes Agent with DeepSeek V4 and Slack: install it, pick Nous Portal or the DeepSeek API, then connect Slack via Socket Mode with xoxb and xapp tokens.
- [Zed AI Context: Agent Panel, AGENTS.md, Rules, and MCP](https://iceberglakehouse.com/posts/2026-03-context-zed/) (2026-03-07): Zed's Agent Panel loads one project instruction file (.rules first, then AGENTS.md or CLAUDE.md), takes @-mentions as context, and adds MCP via context_servers.
- [Windsurf Flows vs MCP Servers: Context and MCP Config](https://iceberglakehouse.com/posts/2026-03-context-windsurf/) (2026-03-07): Windsurf Flows track your edits and terminal; MCP servers add outside tools. In Devin Desktop (ex-Windsurf), MCP lives in ~/.config/devin/mcp_config.json.
- [Perplexity Spaces (Now Projects), Memory, and File Limits](https://iceberglakehouse.com/posts/2026-03-context-perplexity/) (2026-03-07): Perplexity Spaces are now Projects: up to 10,000 files and 5 GB each, 100 MB per file, 8,000-character instructions. Thread uploads are capped at 40 MB a file.
- [OpenWork Guide: Open-Source Claude Cowork Alternative](https://iceberglakehouse.com/posts/2026-03-context-openwork/) (2026-03-07): OpenWork is a free, open-source desktop agent built on OpenCode. Context comes from your workspace folder, SKILL.md skills, MCP connectors, and saved sessions.
- [Google Antigravity Rules, Skills, and Workflows Explained](https://iceberglakehouse.com/posts/2026-03-context-google-antigravity/) (2026-03-07): Antigravity reads AGENTS.md, GEMINI.md, and .agents/rules/*.md as rules and .agents/skills/ as skills. Legacy workflows stop working on October 19, 2026.
- [Gemini CLI Context Window: GEMINI.md, /memory, /compress](https://iceberglakehouse.com/posts/2026-03-context-gemini-cli/) (2026-03-07): Gemini CLI uses Gemini models with a 1M-token context window. GEMINI.md files load as memory, /memory show displays them, and /compress summarizes long chats.
- [NotebookLM Source Limits and the Gemini Context Window](https://iceberglakehouse.com/posts/2026-03-context-gemini-web-notebooklm/) (2026-03-07): NotebookLM, now Gemini Notebook, allows 50 sources per notebook free and up to 600 on Ultra, each up to 500,000 words or 200MB. Gemini chats: 32K to 1M tokens.
- [Claude Code Context Window: CLAUDE.md, /compact, /context](https://iceberglakehouse.com/posts/2026-03-context-claude-code/) (2026-03-07): Claude Code's context window is 1M tokens on current models and auto-compacts near 967K. Check usage with /context, summarize with /compact, reset with /clear.
- [Claude Cowork Guide: Instructions, Projects, Scheduled Tasks](https://iceberglakehouse.com/posts/2026-03-context-claude-cowork/) (2026-03-07): Claude Cowork runs multi-step tasks for you on paid plans. Set global, folder, and project instructions, connect tools, and schedule work with /schedule.
- [Claude Desktop MCP Setup: Config File, Extensions, Context](https://iceberglakehouse.com/posts/2026-03-context-claude-desktop/) (2026-03-07): Add MCP servers to Claude Desktop with one-click .mcpb extensions or by editing claude_desktop_config.json (Settings > Developer > Edit Config), then restart.
- [Claude Context Window on Claude.ai: Projects and Memory](https://iceberglakehouse.com/posts/2026-03-context-claude-web/) (2026-03-07): Claude.ai chat on paid plans gets a 1M token context window with Opus 5.5, Sonnet 5.5, Haiku 5.5, and Fable 5.1. Use Projects and memory to keep it focused.
- [Codex Context Window: Limits, Compaction, and AGENTS.md](https://iceberglakehouse.com/posts/2026-03-context-openai-codex/) (2026-03-07): Codex context window: GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna list 1.05M tokens in the API. Check your Codex limit with /status and free space with /compact.
- [ChatGPT Memory vs Custom Instructions vs Projects (2026)](https://iceberglakehouse.com/posts/2026-03-context-chatgpt/) (2026-03-07): Custom instructions set rules for every chat, memory keeps facts about you, and Projects bundle chats, files, and instructions for one body of work.
- [T3 Chat Guide: Models, $8 Plan, and Usage Limits](https://iceberglakehouse.com/posts/2026-03-context-t3-chat/) (2026-03-07): T3 Chat puts GPT-6, Claude, Gemini, and Kimi behind one picker for $8 a month. Long threads, files, and search cost more, so here is how to keep context lean.
- [VS Code AI Context: Copilot, AGENTS.md, MCP, Cline, Continue](https://iceberglakehouse.com/posts/2026-03-context-vscode-llm-plugins/) (2026-03-07): VS Code Copilot reads copilot-instructions.md and AGENTS.md, loads MCP from .vscode/mcp.json, and compacts long chats. Cline and Continue add their own rules.
- [What Is MCP? Model Context Protocol and How Tools Connect](https://iceberglakehouse.com/posts/2026-03-context-mcp-deep-dive/) (2026-03-07): MCP is an open protocol that lets AI apps call tools and read data on servers over stdio or Streamable HTTP. How it works and how 17 AI tools connect to it.
- [Recursive Language Models (RLMs): How They Work](https://iceberglakehouse.com/posts/2026-01-recursive-langauge-models/) (2026-01-10): Recursive Language Models keep a long prompt in a Python REPL and let the model call itself on pieces of it, handling inputs 100x beyond its context window.
- [Building a Basic MCP Server with Python](https://iceberglakehouse.com/posts/2025-04-basics-of-making-mcp-server/) (2025-04-04): Build a basic Model Context Protocol server in Python with uv, add tools that read local CSV and Parquet files, and connect it to Claude for Desktop.
- [Crash Course on Developing AI Applications with LangChain](https://iceberglakehouse.com/posts/2025-02-crash-course-on-langchain/) (2025-02-01): A guide on building AI applications with LangChain, a framework for developing AI applications powered by Large Language Models (LLMs).

## Articles: Dremio

- [How to Use Dremio with Zed: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-zed/) (2026-03-05): Connect the Zed editor's AI assistant to the Dremio Cloud MCP server with OAuth, then add an AGENTS.md covering Dremio SQL conventions, credentials, and terms.
- [How to Use Dremio with Windsurf: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-windsurf/) (2026-03-05): Connect Windsurf's Cascade agent to the Dremio Cloud MCP server, then add Windsurf rules and community Dremio skills and docs so it writes correct Dremio SQL.
- [How to Use Dremio with OpenWork: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-openwork/) (2026-03-05): Connect OpenWork, the open-source desktop agent built on OpenCode, to Dremio through the MCP server, and give it Dremio SQL context with an AGENTS.md file.
- [How to Use Dremio with OpenCode: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-opencode/) (2026-03-05): Connect the open-source OpenCode terminal agent to the Dremio Cloud MCP server, then place an AGENTS.md with Dremio SQL conventions so it queries correctly.
- [How to Use Dremio with OpenAI Codex CLI: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-openai-codex/) (2026-03-05): Connect OpenAI Codex CLI to Dremio through the Dremio Cloud MCP server with OAuth, then write an AGENTS.md with Dremio SQL conventions and credential guidance.
- [How to Use Dremio with JetBrains AI Assistant: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-jetbrains-ai/) (2026-03-05): Connect JetBrains AI Assistant to the Dremio Cloud MCP server with OAuth, then add project rules and custom prompts that teach it Dremio SQL conventions.
- [How to Use Dremio with Google Antigravity: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-google-antigravity/) (2026-03-05): Connect Google Antigravity to the Dremio Cloud MCP server, then add a SKILL.md and agent rules so its autonomous agents write correct Dremio SQL.
- [How to Use Dremio with GitHub Copilot: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-github-copilot/) (2026-03-05): Connect GitHub Copilot agent mode to the Dremio Cloud MCP server, set enterprise policies, and add a copilot-instructions.md file with Dremio SQL rules.
- [How to Use Dremio with Gemini CLI: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-gemini-cli/) (2026-03-05): Connect Gemini CLI to Dremio through the Dremio Cloud MCP server with OAuth, then write a GEMINI.md with Dremio SQL conventions so it writes correct queries.
- [Connect Dremio to Cursor: MCP Server, Rules, and Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-cursor/) (2026-03-05): Connect Dremio to Cursor with the hosted Dremio MCP server or the open-source dremio-mcp, then add .cursor/rules so the agent writes correct Dremio SQL.
- [How to Use Dremio with Claude CoWork: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-claude-cowork/) (2026-03-05): Connect Claude CoWork to Dremio via the MCP connector, then add global and folder instructions and community Dremio docs so it can query and build data apps.
- [How to Use Dremio with Claude Code: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-claude-code/) (2026-03-05): Connect Claude Code to Dremio with the Dremio Cloud MCP server or the official plugin, then write a CLAUDE.md with Dremio SQL rules to query and build apps.
- [How to Use Dremio with Amazon Kiro: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-amazon-kiro/) (2026-03-05): Connect Amazon Kiro to Dremio through the Dremio Cloud MCP server with OAuth, then use Kiro specs, steering files, and Powers to give it Dremio SQL context.
- [Connect Apache Druid to Dremio Cloud: Add SQL Joins, AI, and Governance to Your Real-Time Analytics](https://iceberglakehouse.com/posts/2026-03-connector-apache-druid/) (2026-03-01): Connect Apache Druid to Dremio Cloud to join real-time event data with business context in SQL, run historical analysis, and govern it. Setup steps included.
- [Connect MongoDB to Dremio Cloud: SQL Analytics on Document Data](https://iceberglakehouse.com/posts/2026-03-connector-mongodb/) (2026-03-01): Connect MongoDB to Dremio Cloud and query document data with SQL: flatten nested arrays, join with relational sources, add reflections, and use AI analytics.
- [Connect Vertica to Dremio Cloud: Federation for Analytics-Optimized Data](https://iceberglakehouse.com/posts/2026-03-connector-vertica/) (2026-03-01): Connect Vertica to Dremio Cloud to cut license costs, federate with cloud sources, and modernize without a big-bang migration. Step-by-step setup included.
- [Connect Azure Synapse Analytics to Dremio Cloud: Multi-Cloud Data Warehouse Federation](https://iceberglakehouse.com/posts/2026-03-connector-azure-synapse/) (2026-03-01): Connect Azure Synapse Analytics to Dremio Cloud for multi-cloud federation, lower costs through reflections, and shared governance. Step-by-step setup included.
- [Connect Snowflake to Dremio Cloud: Federate, Govern, and Accelerate Beyond Snowflake](https://iceberglakehouse.com/posts/2026-03-connector-snowflake/) (2026-03-01): Connect Snowflake to Dremio Cloud to reduce credit use with reflections, federate beyond Snowflake, and apply one governance model. Step-by-step setup included.
- [Connect Google BigQuery to Dremio Cloud: Cross-Cloud Analytics Without Data Movement](https://iceberglakehouse.com/posts/2026-03-connector-google-bigquery/) (2026-03-01): Connect Google BigQuery to Dremio Cloud to control query costs with reflections, run cross-cloud analytics, and govern data in one place. Setup steps included.
- [Connect Amazon Redshift to Dremio Cloud: Extend Your Warehouse with Federation and AI Analytics](https://iceberglakehouse.com/posts/2026-03-connector-amazon-redshift/) (2026-03-01): Connect Amazon Redshift to Dremio Cloud to extend the warehouse without Spectrum, trim cluster costs, and federate across warehouses. Includes setup steps.
- [Connect Azure Storage to Dremio Cloud: Query Your Microsoft Data Lake with SQL and AI](https://iceberglakehouse.com/posts/2026-03-connector-azure-storage/) (2026-03-01): Connect Azure Blob Storage and ADLS Gen2 to Dremio Cloud to query your lake with SQL, manage Iceberg tables, and federate beyond Azure. Setup steps included.
- [Connect Amazon S3 to Dremio Cloud: Query Your Data Lake with SQL, Federation, and AI](https://iceberglakehouse.com/posts/2026-03-connector-amazon-s3/) (2026-03-01): Connect Amazon S3 to Dremio Cloud to query Parquet, CSV, and JSON with SQL, manage Iceberg tables, and federate your lake with databases. Setup steps included.
- [Connect SAP HANA to Dremio Cloud: Unlock Analytics Beyond the SAP Ecosystem](https://iceberglakehouse.com/posts/2026-03-connector-sap-hana/) (2026-03-01): Connect SAP HANA to Dremio Cloud to analyze SAP data alongside other platforms, ease HANA memory pressure, and add AI analytics. Includes step-by-step setup.
- [Connect IBM Db2 to Dremio Cloud: Modernize Mainframe Analytics with Federation and AI](https://iceberglakehouse.com/posts/2026-03-connector-ibm-db2/) (2026-03-01): Connect IBM Db2 to Dremio Cloud to query mainframe data without IBM middleware, federate it with cloud sources, and modernize step by step. Setup included.
- [Connect Microsoft SQL Server to Dremio Cloud: Federate Enterprise Data Without ETL](https://iceberglakehouse.com/posts/2026-03-connector-microsoft-sql-server/) (2026-03-01): Connect SQL Server to Dremio Cloud to move past linked server limits, cut license costs, and federate enterprise data across clouds. Includes setup steps.
- [Extract Structured Data from Text with Dremio's AI_GENERATE Function](https://iceberglakehouse.com/posts/2026-03-ai-ai-generate/) (2026-03-01): Extract structured fields from emails and contract notes with Dremio's AI_GENERATE and ROW type output, using a hands-on bronze, silver, and gold view build.
- [Connect Oracle Database to Dremio Cloud: Enterprise Analytics Without Data Movement](https://iceberglakehouse.com/posts/2026-03-connector-oracle/) (2026-03-01): Connect Oracle Database to Dremio Cloud step by step, including user impersonation, then query and federate Oracle data with predicate pushdown and no copies.
- [Generate Summaries and Insights with Dremio's AI_COMPLETE Function](https://iceberglakehouse.com/posts/2026-03-ai-ai-complete/) (2026-03-01): Use Dremio's AI_COMPLETE to generate text in SQL. This hands-on guide builds views that write executive product summaries and marketing copy from your tables.
- [Connect MySQL to Dremio Cloud: Federated Analytics Without ETL](https://iceberglakehouse.com/posts/2026-03-connector-mysql/) (2026-03-01): Connect MySQL to Dremio Cloud step by step, then run analytics, federate it with other sources, and build views for the AI agent without ETL pipelines.
- [Classify Your Data with SQL: A Hands-On Guide to Dremio's AI_CLASSIFY Function](https://iceberglakehouse.com/posts/2026-03-ai-ai-classify/) (2026-03-01): Classify data in SQL with Dremio's AI_CLASSIFY. Build bronze, silver, and gold views that tag review sentiment and route support tickets without leaving SQL.
- [Connect PostgreSQL to Dremio Cloud: Query, Federate, and Accelerate Your Data](https://iceberglakehouse.com/posts/2026-03-connector-postgresql/) (2026-03-01): Step-by-step setup for connecting PostgreSQL to Dremio Cloud, then querying it with SQL, joining it with other sources, and speeding it up with reflections.
- [Connect AWS Glue Data Catalog to Dremio Cloud: Query and Manage Your AWS Iceberg Tables](https://iceberglakehouse.com/posts/2026-03-connector-aws-glue/) (2026-03-01): Connect AWS Glue Data Catalog to Dremio Cloud for read and write on Iceberg tables, automated maintenance, and federation, without Athena per-TB pricing.
- [Connect Snowflake Open Catalog to Dremio Cloud: Multi-Engine Iceberg Analytics](https://iceberglakehouse.com/posts/2026-03-connector-snowflake-open-catalog/) (2026-03-01): Connect Snowflake Open Catalog, built on Apache Polaris, to Dremio Cloud for multi-engine Iceberg analytics with credential vending and write support.
- [Connect Databricks Unity Catalog to Dremio Cloud: Query Delta Lake Tables with Federation and AI](https://iceberglakehouse.com/posts/2026-03-connector-unity-catalog/) (2026-03-01): Connect Databricks Unity Catalog to Dremio Cloud to query Delta Lake tables through UniForm, federate them with other sources, and add AI analytics.
- [Connect Any Iceberg REST Catalog to Dremio Cloud: Universal Lakehouse Access](https://iceberglakehouse.com/posts/2026-03-connector-iceberg-rest-catalog/) (2026-03-01): Connect any Apache Iceberg REST catalog to Dremio Cloud for read and write access, multi-catalog federation, automated maintenance, and credential vending.
- [Dremio's Built-in Open Catalog: Your Zero-Configuration Apache Iceberg Lakehouse](https://iceberglakehouse.com/posts/2026-03-connector-dremio-open-catalog/) (2026-03-01): Dremio Cloud's built-in Open Catalog gives you a managed Iceberg lakehouse with no setup. Create tables, use branching, time travel, and autonomous reflections.
- [Connect Dremio Software to Dremio Cloud: Hybrid Federation Across Deployments](https://iceberglakehouse.com/posts/2026-03-connector-dremio-to-dremio/) (2026-03-01): Connect self-managed Dremio Software to Dremio Cloud for hybrid federation, incremental cloud migration, and consolidated governance. Setup steps included.

## Articles: Data Engineering

- [Running an Iceberg Lakehouse on Kubernetes](https://iceberglakehouse.com/posts/iceberg-on-kubernetes/) (2026-09-10): Catalog, maintenance, and compaction as Kubernetes workloads: scheduling classes, job structure, credential flow, and the failures that come from the interaction.
- [What to Assert When You Test an Iceberg Pipeline](https://iceberglakehouse.com/posts/iceberg-pipeline-assertions/) (2026-09-10): Fixtures, in-memory catalogs, and golden metadata: the assertions that catch wrong rows, unsafe reruns, schema drift, and concurrent-write corruption in CI.
- [Kafka Connect to Iceberg: How the Commit Actually Works](https://iceberglakehouse.com/posts/kafka-connect-iceberg-commits/) (2026-09-10): How the Iceberg sink connector delivers exactly-once from Kafka: the coordinator, control topic, offsets stored in snapshots, and where duplicates still get in.
- [What a Query Costs](https://iceberglakehouse.com/posts/lakehouse-unit-economics/) (2026-09-10): Four meters, their real proportions, and how to attribute compute to a query, a table, and a team so a platform can answer what a dashboard costs to run.
- [Great Expectations vs Soda vs dbt Tests: Data Quality Tools](https://iceberglakehouse.com/posts/data-quality-tooling-compared/) (2026-09-02): Great Expectations, Soda, dbt tests and anomaly detection tools compared on check language, where they run, cost, Iceberg metadata use and 2026 vendor changes.
- [dbt on Iceberg: Incremental Models on Open Tables](https://iceberglakehouse.com/posts/dbt-on-iceberg-incremental-models/) (2026-09-02): How dbt incremental materializations map to Iceberg operations, and the configuration, predicates, and maintenance that keep them healthy.
- [The Lakehouse Ingestion Tool Landscape: Fivetran, Airbyte, dlt, and CDC vs Batch](https://iceberglakehouse.com/posts/lakehouse-ingestion-tools/) (2026-09-02): How Fivetran, Airbyte, dlt, and CDC and streaming tools land well-behaved Apache Iceberg tables, and how to choose and maintain them.
- [Logs, Traces, and Metrics as Tables: Building an OpenTelemetry Data Lake on Iceberg](https://iceberglakehouse.com/posts/opentelemetry-data-lake-on-iceberg/) (2026-09-02): Building an OpenTelemetry data lake on Iceberg: schemas for spans, logs, and metrics, ingestion, query patterns, and retention.
- [Orchestration in 2026: Airflow 3 vs Dagster vs Prefect vs Event-Driven](https://iceberglakehouse.com/posts/orchestration-in-2026/) (2026-09-02): Where Airflow 3, Dagster, Prefect, and event-driven triggering stand for lakehouse pipelines in 2026, after the Prefect acquisition of Dagster.
- [Schema Registries and Event Schemas: Avro, Protobuf, and JSON Schema on the Way Into the Lakehouse](https://iceberglakehouse.com/posts/schema-registries-and-event-schemas/) (2026-09-02): How Avro, Protobuf, and JSON Schema evolve through a registry, and how that maps to the schema evolution rules of Iceberg.
- [Synthetic Data in the Lakehouse: Generation, Governance, and Testing](https://iceberglakehouse.com/posts/synthetic-data-in-the-lakehouse/) (2026-09-02): What synthetic data in a lakehouse is for, the generation methods, how to preserve fidelity, and where synthetic tables belong.
- [Arrow Flight SQL and ADBC: Why the Database Driver Is the Slowest Part of Your Query](https://iceberglakehouse.com/posts/arrow-flight-sql-adbc-connectivity/) (2026-08-25): JDBC and ODBC often dominate large-result time. Flight SQL and ADBC keep data columnar from server to client, with Python, Go, and Rust examples.
- [Query Routing at Machine Scale: Dynamic Workload Distribution Across Lakehouse Engines](https://iceberglakehouse.com/posts/query-routing-lakehouse-engines/) (2026-08-25): Route each lakehouse query by shape, not by sender. Signals, rules, and how to keep dashboards, batch jobs, and agents from sharing one engine.
- [Semantic Layer Federation: One Meaning for Data That Lives Everywhere](https://iceberglakehouse.com/posts/semantic-layer-federation-multi-cloud/) (2026-08-19): Build a federated semantic layer across multi-cloud data so one set of governed metric definitions serves BI tools, dashboards, and AI agents identically.
- [Apache Arrow Flight and ADBC, and Why Database Connectivity Finally Went Columnar](https://iceberglakehouse.com/posts/arrow-flight-adbc-explained/) (2026-08-06): How Arrow Flight, Flight SQL, and ADBC move query results as columnar data instead of rows, removing the driver bottleneck between databases and applications.
- [Why AI Agents Fail on Raw Data, and What to Give Them Instead](https://iceberglakehouse.com/posts/data-products-for-ai-agents/) (2026-07-28): Agents fail on raw lake data because business rules live in people's heads. Data products with semantic contracts fix this at the source.
- [Freshness Is a Contract, Not a Note on a Dashboard](https://iceberglakehouse.com/posts/freshness-as-a-contract/) (2026-07-25): How to turn data freshness into a contract with a measurable budget, an owner, and consequences, so AI agents know how old the numbers are before they act.
- [The Whole Lakehouse Fits on Your Laptop Now](https://iceberglakehouse.com/posts/laptop-scale-lakehouse/) (2026-07-25): Consumer hardware, columnar formats, single-node engines, and the Iceberg REST catalog crossed a threshold: a large share of cluster work now runs locally.
- [Iceberg's Next Version Depends on Decisions Being Made in Parquet](https://iceberglakehouse.com/posts/parquet-constrains-iceberg/) (2026-07-25): The Iceberg and Parquet specifications are co-evolving. Several of the most consequential Iceberg v4 proposals are waiting on Parquet work.
- [Building a Lakehouse That Stays Inside the Border](https://iceberglakehouse.com/posts/sovereign-lakehouse/) (2026-07-25): Residency is a storage location. Sovereignty is who can compel access, who operates the systems, and whether you can leave.
- [Table Maintenance Stopped Being a Product](https://iceberglakehouse.com/posts/table-maintenance-economics/) (2026-07-25): Iceberg table maintenance commoditized when every platform started shipping it. What the six operations are, what they cost, and the observability you.
- [Three Vendors Are Rebuilding the Path From Transaction to Agent](https://iceberglakehouse.com/posts/transaction-to-agent-race/) (2026-07-25): Databricks, Snowflake, and SAP are closing the gap between operational databases and analytical platforms through acquisition, betting on different layers.
- [What Zero-Copy Actually Costs](https://iceberglakehouse.com/posts/zero-copy-actually-costs/) (2026-07-25): Six architectures that call themselves zero-copy, where the bytes actually move, the costs left out of business cases, and when copying data is the better deal.
- [The Breakdown of the Open Lakehouse in 2026: Iceberg, Arrow, Polaris, Parquet, and Ossie, and How to Actually Build One](https://iceberglakehouse.com/posts/open-lakehouse-breakdown-2026/) (2026-07-24): A detailed breakdown of the five Apache projects powering the open lakehouse in 2026, Parquet, Iceberg, Polaris, Arrow, and Ossie, and how to build one.
- [RAMpocalypse Explained: Why RAM Prices Spiked in 2026](https://iceberglakehouse.com/posts/rampocalypse-economics/) (2026-07-24): Why RAM prices spiked in 2026: AI demand pulled DRAM wafers into HBM, contract prices jumped about 90% in one quarter, and new fabs arrive no sooner than 2027.
- [Block vs. Object Storage: A Deep Dive Into the Foundation of Modern Data, and How the Lakehouse Made the Slow Option Fast](https://iceberglakehouse.com/posts/block-vs-object-storage-deep-dive/) (2026-07-06): Why analytics moved to slower object storage and got faster: how Parquet, Iceberg, and modern engines hide S3 latency, plus a guide to placing each workload.
- [Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust](https://iceberglakehouse.com/posts/deterministic-data-engineering-with-ai-harnesses/) (2026-07-06): How to use AI agent harnesses for data engineering without losing determinism, reproducibility, and trust in your data pipelines and analytics.
- [Preparing Your Data Lakehouse for the EU AI Act: Auditable Lineage and Data Provenance](https://iceberglakehouse.com/posts/eu-ai-act-data-lakehouse-lineage-provenance-compliance/) (2026-07-06): How lakehouse lineage, provenance, audit logs, and agent identity help you prepare for the EU AI Act, with a readiness checklist. Technical, not legal, advice.
- [A Deep Dive Into File Compression: How Data Gets Smaller, Why Codecs Differ, and What to Actually Use in the Lakehouse](https://iceberglakehouse.com/posts/file-compression-deep-dive/) (2026-07-06): How compression codecs work, why Zstandard became the default, how encodings differ from codecs in Parquet, and which codec to pick for each lakehouse workload.
- [The File Format Renaissance: Parquet, Lance, Vortex, Nimble, BtrBlocks, and the New Physics of Columnar Storage](https://iceberglakehouse.com/posts/file-formats-renaissance/) (2026-07-06): How Parquet, Lance, Vortex, Nimble, and BtrBlocks differ in encodings and point-lookup speed, and which columnar file format fits which workload in 2026.
- [File Encryption for the Lakehouse: The Terminology, the Machinery, and the Hard Problem of Interoperable Encrypted Tables](https://iceberglakehouse.com/posts/lakehouse-encryption-deep-dive/) (2026-07-06): Encryption terms, Parquet modular encryption, and Iceberg 1.11 table encryption explained, plus the problems of sharing encrypted tables across engines.
- [ASF vs Linux Foundation vs CNCF: Which Fits a New Project?](https://iceberglakehouse.com/posts/open-source-foundations-explained/) (2026-07-06): Pick the ASF for vendor-neutral, individual-led governance, the Linux Foundation for a funded company consortium, or the CNCF for cloud native infrastructure.
- [Operational vs. Analytical Systems: Why the Oldest Divide in Data Exists, What Physics Enforces It, and the Honest Truth About Hybrid Systems](https://iceberglakehouse.com/posts/operational-vs-analytical-systems/) (2026-07-06): Why OLTP and OLAP systems split, the storage physics that keep them apart, how HTAP has fared, and a 2026 framework for composing both on an open substrate.
- [Migrating Proprietary Warehouses to Open Lakehouses: The 2026 Playbook for Zero-Copy Metadata Translation](https://iceberglakehouse.com/posts/proprietary-warehouse-open-lakehouse-zero-copy-metadata-translation/) (2026-07-06): A staged playbook for moving proprietary warehouses to Iceberg: what zero-copy really means, federation during transition, validation, and a 90-day runbook.
- [Data Platform AI Agent Tooling in 2026: 11 Vendors Compared](https://iceberglakehouse.com/posts/data-platform-ai-agent-tooling/) (2026-05-31): How Dremio, Snowflake, Databricks, Fabric, AWS, Google Cloud, ClickHouse, Doris, Spice, Bauplan, and Qlik expose data to AI agents: MCP servers, agents, CLIs.
- [Automating Table Maintenance Before Small Files Accumulate](https://iceberglakehouse.com/posts/2026-05-24-automating-table-maintenance/) (2026-05-24): Learn how Databricks Predictive Optimization, AWS S3 Tables, and Iceberg native actions automate compaction and snapshot management before small files.
- [Polaris vs Unity Catalog vs Glue: Pick an Iceberg Catalog](https://iceberglakehouse.com/posts/2026-05-24-choosing-iceberg-control-plane/) (2026-05-24): Use Apache Polaris for open multi-engine Iceberg, Unity Catalog for mixed Delta and Iceberg, or AWS Glue for an AWS-only stack. Features and tradeoffs compared.
- [Clean Rooms for Privacy-Preserving Analytics](https://iceberglakehouse.com/posts/2026-05-24-clean-rooms-privacy/) (2026-05-24): Data clean rooms enable secure multi-party analytics without sharing raw data. Learn how Databricks Clean Rooms, AWS Clean Rooms, and BigQuery.
- [Building Composable Query Engines with Rust Runtimes](https://iceberglakehouse.com/posts/2026-05-24-composable-query-engines/) (2026-05-24): How Apache DataFusion, Velox, and Substrait form a composable query engine stack, how DataFusion and Velox compare, and a framework for choosing an engine.
- [Data Mesh After the Hype: What Actually Works](https://iceberglakehouse.com/posts/2026-05-24-data-mesh-after-hype/) (2026-05-24): Three years after Zhamak Dehghani's original papers, data mesh has proven valuable in specific organizational contexts and impractical in others.
- [How dbt Fusion Reshapes Analytics Engineering](https://iceberglakehouse.com/posts/2026-05-24-dbt-fusion-analytics-engineering/) (2026-05-24): dbt Fusion entered public beta in May 2025 with a Rust-powered runtime that changes how analytics engineers develop, validate, and deploy SQL models.
- [Using DuckDB and Polars to Query Iceberg Tables](https://iceberglakehouse.com/posts/2026-05-24-duckdb-polars-iceberg/) (2026-05-24): DuckDB 1.4 LTS and Polars streaming engine now both support reading and writing Apache Iceberg tables. Learn how to use them for lakehouse analytics in 2025.
- [FinOps for Data Warehouses with Open Billing Data](https://iceberglakehouse.com/posts/2026-05-24-finops-warehouse-cost/) (2026-05-24): Build a FinOps pipeline for data warehouses with FOCUS 1.3 billing data: cost attribution and tagging, chargeback vs. showback, commitments, and budget alerts.
- [Designing Governed RAG on Data Products](https://iceberglakehouse.com/posts/2026-05-24-governed-rag-data-products/) (2026-05-24): Design enterprise RAG on governed data products: enforce access policies at retrieval, route numeric questions through a semantic layer, and keep audit trails.
- [What Iceberg V3 Advances Mean for CDC Pipelines](https://iceberglakehouse.com/posts/2026-05-24-iceberg-cdc-pipelines/) (2026-05-24): Apache Iceberg V3 brings deletion vectors and row lineage that reshape CDC pipeline design. Learn what these features mean for your streaming data architecture.
- [Lance and Iceberg for Multimodal AI Data](https://iceberglakehouse.com/posts/2026-05-24-lance-iceberg-multimodal/) (2026-05-24): LanceDB and Apache Iceberg serve complementary roles in a multimodal AI lakehouse. Learn when to use Lance for embeddings and random access, and Iceberg.
- [Bringing MLflow and Data Pipelines Closer Together](https://iceberglakehouse.com/posts/2026-05-24-mlflow-data-pipelines/) (2026-05-24): How MLflow 3 connects ML experiments to data pipelines: training data lineage, input quality checks, model vs. data drift, CI/CD, and scheduling with Airflow.
- [OpenLineage as the Spine of Data Observability](https://iceberglakehouse.com/posts/2026-05-24-openlineage-observability/) (2026-05-24): How OpenLineage collects lineage from Airflow, Spark, Flink, and dbt, and how to use it for blast radius analysis, SLA tracking, and Marquez or DataHub.
- [When Paimon Beats Iceberg for Mutable Streams](https://iceberglakehouse.com/posts/2026-05-24-paimon-vs-iceberg-mutable-streams/) (2026-05-24): Apache Paimon uses LSM-Tree storage for native CDC upserts without restart. Learn when Paimon outperforms Iceberg for high-churn mutable streaming workloads.
- [Policy as Code for Lakehouse Governance](https://iceberglakehouse.com/posts/2026-05-24-policy-as-code-governance/) (2026-05-24): Policy as code for the lakehouse: OPA, ABAC, row filters, and column masks across Databricks, Snowflake Horizon, and BigQuery, plus CI/CD for your policies.
- [Semantic Layers for Text-to-SQL: Dremio vs Cortex vs dbt](https://iceberglakehouse.com/posts/2026-05-24-semantic-layers-text-to-sql/) (2026-05-24): In dbt Labs' 2026 benchmark, a semantic layer lifted text-to-SQL accuracy to 98 to 100%. Compare Dremio, Snowflake Cortex Analyst, and the dbt Semantic Layer.
- [Choosing Vector Stores for Retrieval Workloads](https://iceberglakehouse.com/posts/2026-05-24-vector-stores-retrieval/) (2026-05-24): pgvector, Milvus, Weaviate, and LanceDB each make different tradeoffs on index type, hybrid search, scale, and operational complexity.
- [Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail](https://iceberglakehouse.com/posts/2026-05-23-single-node-data-engineering-duckdb-datafusion-polars-lakesail/) (2026-05-23): Optimize single-node data engineering with DuckDB, DataFusion, Polars, and LakeSail. Compare architectures and learn when to transition to Dremio MPP.
- [Setting Up an AWS-Native Open Lakehouse: Querying Apache Iceberg with AWS Athena and AWS Glue Catalog](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-aws-athena-glue/) (2026-05-22): A comprehensive guide to building an open, high-performance lakehouse on AWS using Apache Iceberg, AWS Glue Catalog, Amazon S3, and S3 Tables, with query.
- [Maintaining Apache Iceberg Tables: Compaction, Snapshot Expiration, and Orphan File Cleanup](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-maintenance-compaction/) (2026-05-22): An in-depth guide to orchestrating maintenance operations on Apache Iceberg tables, covering bin-packing, sort-based, Z-Order compaction, snapshot.
- [Migrating to Apache Iceberg: Strategies for Every Source System](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-15/) (2026-04-29): Migrate to Iceberg from Hive, data warehouses, or raw files using in-place migration, full rewrite, or the zero-downtime view swap pattern.
- [Hands-On with Apache Iceberg Using Dremio Cloud](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-14/) (2026-04-29): A practical walkthrough of creating, querying, and optimizing Iceberg tables on Dremio Cloud, from account setup to AI-powered analytics.
- [Approaches to Streaming Data into Apache Iceberg Tables](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-13/) (2026-04-29): Stream data into Iceberg with Spark Structured Streaming, Flink, or Kafka Connect. Here is how each works and the trade-offs between latency and maintenance.
- [Using Apache Iceberg with Python and MPP Query Engines](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-12/) (2026-04-29): Access Iceberg tables from Python with PyIceberg, DuckDB, and Polars, or through MPP engines like Dremio, Spark, and Trino. Here is how each approach works.
- [Apache Iceberg Metadata Tables: Querying the Internals](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-11/) (2026-04-29): Iceberg metadata tables let you query snapshots, files, manifests, and partitions using SQL. Here is every metadata table and how to use them.
- [Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-10/) (2026-04-29): Keep Iceberg tables fast with compaction, snapshot expiry, orphan cleanup, and manifest rewriting. Here is when and how to run each operation.
- [Concurrency, Isolation, and MVCC: How Engines Handle Contention](https://iceberglakehouse.com/posts/2026-04-29-query-engine-10/) (2026-04-29): Databases handle concurrent access using locks, MVCC, or optimistic concurrency control. Here is how each approach works and what tradeoffs each creates.
- [How Data Lake Table Storage Degrades Over Time](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-09/) (2026-04-29): Iceberg tables degrade through small files, orphan files, metadata bloat, sort order decay, and partition skew. Here is how to diagnose each problem.
- [Hash, Sort-Merge, Broadcast: How Distributed Joins Work](https://iceberglakehouse.com/posts/2026-04-29-query-engine-09/) (2026-04-29): Distributed joins move data across the network using shuffle, broadcast, or co-location strategies. Here is how each works and when engines choose which.
- [When Catalogs Are Embedded in Storage](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-08/) (2026-04-29): S3 Tables and MinIO AI Stor embed the Iceberg catalog directly in the storage layer. Here is when embedded catalogs make sense and when they do not.
- [Partitioning, Sharding, and Data Distribution Strategies](https://iceberglakehouse.com/posts/2026-04-29-query-engine-08/) (2026-04-29): How hash, range, and list partitioning distribute data, how partition pruning speeds queries, and how bucketing and data skew affect real query engines.
- [What Are Lakehouse Catalogs? The Role of Catalogs in Apache Iceberg](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-07/) (2026-04-29): Lakehouse catalogs store metadata pointers, manage namespaces, and enforce access control. Here is the complete catalog landscape from Polaris to Glue.
- [Buffer Pools, Caches, and the Memory Hierarchy](https://iceberglakehouse.com/posts/2026-04-29-query-engine-07/) (2026-04-29): Databases use buffer pools, column caches, and result caches to keep hot data in RAM. Here is how each caching strategy works and what happens when data.
- [Writing to an Apache Iceberg Table: How Commits and ACID Actually Work](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-06/) (2026-04-29): Here is exactly how an engine writes to an Iceberg table, step by step, from data files through the atomic commit that makes ACID guarantees possible.
- [Volcano, Vectorized, Compiled: How Engines Execute Your Query](https://iceberglakehouse.com/posts/2026-04-29-query-engine-06/) (2026-04-29): The Volcano model processes one row at a time. Vectorized execution processes batches with SIMD. Code generation fuses operators into compiled code.
- [Hidden Partitioning: How Iceberg Eliminates Accidental Full Table Scans](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-05/) (2026-04-29): How Iceberg's hidden partitioning uses six transform functions to prune files without partition columns in queries, and how to choose and combine transforms.
- [Inside the Query Optimizer: How Engines Pick a Plan](https://iceberglakehouse.com/posts/2026-04-29-query-engine-05/) (2026-04-29): Query optimizers transform SQL into execution plans using rule-based rewrites, cost-based search, and adaptive runtime adjustments.
- [Partition Evolution: Change Your Partitioning Without Rewriting Data](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-04/) (2026-04-29): Iceberg lets you change partition schemes without rewriting data. Here is how partition evolution works internally and why Hive-style partitioning could.
- [B-Tree vs LSM Tree: Database Indexing Tradeoffs Explained](https://iceberglakehouse.com/posts/2026-04-29-query-engine-04/) (2026-04-29): B-trees balance reads and writes for OLTP, LSM trees trade read cost for write throughput, and bitmaps and zone maps speed analytics. Learn when to use each.
- [Performance and Apache Iceberg's Metadata](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-03/) (2026-04-29): Iceberg's three-layer metadata tree eliminates directory listing and enables multi-level data skipping. Here is how scan planning actually works.
- [How Databases Organize Data on Disk: Pages, Blocks, and File Formats](https://iceberglakehouse.com/posts/2026-04-29-query-engine-03/) (2026-04-29): Databases structure data on disk as heap files, sorted files, or LSM trees, then wrap it in formats like Parquet with metadata that lets engines skip.
- [The Metadata Structure of Modern Table Formats](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-02/) (2026-04-29): Iceberg uses a metadata tree, Delta Lake uses a transaction log, Hudi uses a timeline. Here is exactly how each format organizes metadata and why it matters.
- [Row vs. Column: How Storage Layout Shapes Everything](https://iceberglakehouse.com/posts/2026-04-29-query-engine-02/) (2026-04-29): Row stores suit transactions, column stores suit analytics. See the I/O math, why columnar compression wins, late materialization, and when to pick each.
- [What Are Table Formats and Why Were They Needed?](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-01/) (2026-04-29): Table formats like Apache Iceberg solved the ACID, schema, and performance problems that turned data lakes into data swamps. Here is how each one works.
- [How Query Engines Think: The Tradeoffs Behind Every Data System](https://iceberglakehouse.com/posts/2026-04-29-query-engine-01/) (2026-04-29): Every database is a collection of engineering tradeoffs. Learn the 9 design decisions that shape how query engines store, index, and process your data.
- [Agentic Analytics on the Apache Lakehouse](https://iceberglakehouse.com/posts/2026-03-07-agentic-analytics/) (2026-03-07): Why LLMs fail on raw Parquet in S3, and how a semantic layer, autonomous reflections, text-to-SQL, and AI SQL functions enable agentic analytics on a lakehouse.
- [What is Apache Arrow? Erasing the Serialization Tax](https://iceberglakehouse.com/posts/2026-03-07-apache-arrow/) (2026-03-07): Apache Arrow removes the serialization tax with a standard in-memory columnar format, zero-copy sharing, and Arrow Flight for fast transfer across the network.
- [What is Apache Iceberg? The Table Format Revolution](https://iceberglakehouse.com/posts/2026-03-07-apache-iceberg/) (2026-03-07): How Apache Iceberg turns piles of Parquet files into reliable tables: its metadata tree, schema and partition evolution, time travel, and atomic snapshots.
- [What is Apache Parquet? Columns, Encoding, and Performance](https://iceberglakehouse.com/posts/2026-03-07-apache-parquet/) (2026-03-07): How Apache Parquet speeds up analytics: columnar layout, dictionary encoding and compression, predicate pushdown with row group skipping, and its role today.
- [What is Apache Polaris? Unifying the Iceberg Ecosystem](https://iceberglakehouse.com/posts/2026-03-07-apache-polaris/) (2026-03-07): Treating thousands of Parquet files as a unified database table requires a brain. Apache Iceberg provides the metadata structure to do this, but.
- [Apache Software Foundation: History, Purpose, and Process](https://iceberglakehouse.com/posts/2026-03-07-apache-software-foundation/) (2026-03-07): How the Apache Software Foundation works: its history, the Apache Way, the Incubator, how it differs from the Linux Foundation, and why lakehouses depend on it.
- [Assembling the Apache Lakehouse: The Modular Architecture](https://iceberglakehouse.com/posts/2026-03-07-assembling-apache-lakehouse/) (2026-03-07): How the Apache lakehouse replaces the monolithic warehouse with four modular pillars, why DIY assembly gets hard, and what an open platform approach adds.
- [Data Engineering Best Practices: The Complete Checklist](https://iceberglakehouse.com/posts/2026-02-debp-de-best-practices-checklist/) (2026-02-18): A prioritized data engineering checklist covering pipeline design, data quality, idempotency, schema management, testing, and observability, with reasons.
- [Data Modeling Best Practices: 7 Mistakes to Avoid](https://iceberglakehouse.com/posts/2026-02-dm-data-modeling-best-practices/) (2026-02-18): Seven data modeling mistakes and fixes: undefined grain, cryptic names, skipped conceptual models, over-normalizing, thin docs, and ignored governance.
- [Semantic Layer Best Practices: 7 Mistakes to Avoid](https://iceberglakehouse.com/posts/2026-02-sl-semantic-layer-best-practices/) (2026-02-18): Seven early design mistakes that sink semantic layers, from defining metrics in several places to securing data only in the BI tool, and how to avoid each one.
- [Pipeline Observability: Know When Things Break](https://iceberglakehouse.com/posts/2026-02-debp-observability-monitoring/) (2026-02-18): An analyst messages you on Slack - "The revenue numbers look wrong. Is the pipeline broken?" You check the orchestrator - all green.
- [Data Vault Modeling: Hubs, Links, and Satellites](https://iceberglakehouse.com/posts/2026-02-dm-data-vault-modeling/) (2026-02-18): Data Vault modeling explained: Hubs hold business keys, Links hold relationships, and Satellites hold history, plus how to expose Data Vault data for analytics.
- [How a Self-Documenting Semantic Layer Reduces Data Team Toil](https://iceberglakehouse.com/posts/2026-02-sl-self-documenting-semantic-layer/) (2026-02-18): How a self-documenting semantic layer cuts data team toil with AI-generated descriptions, suggested labels, and metadata that propagates through views.
- [Testing Data Pipelines: What to Validate and When](https://iceberglakehouse.com/posts/2026-02-debp-testing-data-pipelines/) (2026-02-18): What to test in a data pipeline and when: a testing pyramid for data, schema and contract tests, runtime validation, and regression and anomaly detection.
- [Denormalization: When and Why to Denormalize Your Data](https://iceberglakehouse.com/posts/2026-02-dm-denormalization-when-why/) (2026-02-18): Denormalization adds deliberate redundancy to cut joins and speed up reads. Learn when it helps analytics, when to avoid it, and how views do it virtually.
- [Headless BI: How a Universal Semantic Layer Replaces Tool-Specific Models](https://iceberglakehouse.com/posts/2026-02-sl-headless-bi-semantic-layer/) (2026-02-18): Headless BI explained: why tool-specific models in Tableau or Power BI drift apart, and how one universal semantic layer serves every BI tool and notebook.
- [Partition and Organize Data for Performance](https://iceberglakehouse.com/posts/2026-02-debp-partition-and-organize/) (2026-02-18): A table with 500 million rows takes 45 seconds to query. After partitioning it by date, the same query : filtering on a single day, returns in 2 seconds.
- [Data Modeling for Analytics: Optimize for Queries](https://iceberglakehouse.com/posts/2026-02-dm-data-modeling-for-analytics/) (2026-02-18): Why analytics needs a different data model than your application: denormalization, wide tables, and pre-aggregation patterns that make analytical queries fast.
- [Data Virtualization and the Semantic Layer: Query Without Copying](https://iceberglakehouse.com/posts/2026-02-sl-data-virtualization-semantic-layer/) (2026-02-18): How data virtualization and a semantic layer let you query across sources without copying data, plus guidance on when to virtualize and when to materialize.
- [Batch vs. Streaming: Choose the Right Processing Model](https://iceberglakehouse.com/posts/2026-02-debp-batch-vs-streaming/) (2026-02-18): "We need real-time data. " This is one of the most expensive sentences in data engineering - because it''s rarely true, and implementing it when it''s not.
- [Slowly Changing Dimensions: Types 1-3 with Examples](https://iceberglakehouse.com/posts/2026-02-dm-slowly-changing-dimensions/) (2026-02-18): Slowly Changing Dimensions Types 1, 2, and 3 explained with examples: overwrite, add a history row, or add a column, plus how to choose the right SCD type.
- [The Role of the Semantic Layer in Data Governance](https://iceberglakehouse.com/posts/2026-02-sl-semantic-layer-data-governance/) (2026-02-18): How a semantic layer turns governance policy into practice: shared metric definitions, access rules enforced at query time, lineage, and certification.
- [Schema Evolution Without Breaking Consumers](https://iceberglakehouse.com/posts/2026-02-debp-schema-evolution/) (2026-02-18): Treat your schema like an API. Learn which changes are safe or breaking, the additive-only pattern, schema versioning, and contract checks between pipelines.
- [Dimensional Modeling Explained: Facts, Dimensions, Grain](https://iceberglakehouse.com/posts/2026-02-dm-dimensional-modeling/) (2026-02-18): Dimensional modeling organizes analytics data into fact tables and dimension tables. Learn how to declare the grain, design both table types, and conform them.
- [Why Your AI Initiatives Fail Without a Semantic Layer](https://iceberglakehouse.com/posts/2026-02-sl-why-ai-fails-without-semantic-layer/) (2026-02-18): Five ways AI agents give wrong answers without a semantic layer (metric hallucination, join confusion, column misreads, security bypass) and how to fix them.
- [Idempotent Data Pipelines: Patterns, SQL, and Pitfalls](https://iceberglakehouse.com/posts/2026-02-debp-idempotent-pipelines/) (2026-02-18): An idempotent pipeline gives the same result no matter how often it runs. Use partition overwrite, MERGE upserts, and dedup to make retries and backfills safe.
- [Data Modeling for the Lakehouse: What Changes](https://iceberglakehouse.com/posts/2026-02-dm-data-modeling-lakehouse/) (2026-02-18): How data modeling changes in a lakehouse: schema-on-read, Iceberg partition evolution, and a virtual Medallion Architecture built as SQL views over open tables.
- [Semantic Layer vs. Data Catalog: Complementary, Not Competing](https://iceberglakehouse.com/posts/2026-02-sl-semantic-layer-vs-data-catalog/) (2026-02-18): "We already have a data catalog, so we don''t need a semantic layer. " This is one of the most common misconceptions in modern data architecture.
- [Data Quality Is a Pipeline Problem, Not a Dashboard Problem](https://iceberglakehouse.com/posts/2026-02-debp-data-quality-first/) (2026-02-18): When an analyst finds null values in a revenue column, the typical response is to add a calculated field in the BI tool - `IF revenue IS NULL THEN 0`.
- [Star Schema vs Snowflake Schema: Differences and When to Use](https://iceberglakehouse.com/posts/2026-02-dm-star-schema-vs-snowflake/) (2026-02-18): A star schema keeps each dimension in one flat table; a snowflake schema splits dimensions into normalized sub-tables. Compare joins, speed, and use cases.
- [Semantic Layer vs. Metrics Layer: What's the Difference?](https://iceberglakehouse.com/posts/2026-02-sl-semantic-layer-vs-metrics-layer/) (2026-02-18): A metrics layer defines calculations; a semantic layer adds business context, joins, and governance. See how they relate and why the distinction matters.
- [How to Design Reliable Data Pipelines](https://iceberglakehouse.com/posts/2026-02-debp-design-data-pipelines/) (2026-02-18): Reliable pipelines come from architecture, not bug fixes. Covers the four pipeline layers, building DAGs instead of scripts, dependencies, and failure handling.
- [Conceptual vs Logical vs Physical Data Models (Examples)](https://iceberglakehouse.com/posts/2026-02-dm-types-of-data-models/) (2026-02-18): Conceptual, logical, and physical data models explained with one e-commerce example: what each level contains, who owns it, and how one feeds the next.
- [How to Build a Semantic Layer: A Step-by-Step Guide](https://iceberglakehouse.com/posts/2026-02-sl-how-to-build-semantic-layer/) (2026-02-18): A step-by-step plan for building a semantic layer: start with metrics, map sources, use bronze, silver, and gold views, document, and enforce access policies.
- [How to Think Like a Data Engineer](https://iceberglakehouse.com/posts/2026-02-debp-think-like-data-engineer/) (2026-02-18): Tools change every few years, principles do not. Learn a five-question framework, systems thinking, failure-first design, and anti-patterns to avoid.
- [What Is Data Modeling? Definition, Types, and Examples](https://iceberglakehouse.com/posts/2026-02-dm-what-is-data-modeling/) (2026-02-18): Data modeling defines the entities, attributes, and relationships in your data. Learn the conceptual, logical, and physical levels and the main techniques.
- [What Is a Semantic Layer? Definition, Parts, and Examples](https://iceberglakehouse.com/posts/2026-02-sl-what-is-a-semantic-layer/) (2026-02-18): A semantic layer maps raw tables and columns to business terms and metrics so every BI tool and AI agent gets the same answer. Learn what it contains.
- [A 2026 Introduction to Apache Iceberg](https://iceberglakehouse.com/posts/2026-02-intro-to-apache-iceberg/) (2026-02-13): An updated guide to Apache Iceberg: where it came from, how its metadata works, spec versions V1 through V4, the REST catalog spec, and how to start on Dremio.
- [RAG Isn’t a Modeling Problem. It’s a Data Engineering Problem.](https://iceberglakehouse.com/posts/2026-01-rag-isnt-the-problem/) (2026-01-20): Enterprise RAG breaks at the data layer, not the model. Covers vector database overkill, hybrid search, latency tradeoffs, and the missing semantic context.
- [Building Pangolin - My Holiday Break, an AI IDE, and a Lakehouse Catalog for the Curious](https://iceberglakehouse.com/posts/2026-01-the-story-of-pangolin-catalog/) (2026-01-15): How I built Pangolin, an open lakehouse catalog in Rust, Python, and Svelte, over a holiday break with an AI IDE, and what the project is and is not meant for.
- [dremioframe & iceberg - Pythonic interfaces for Dremio and Apache Iceberg](https://iceberglakehouse.com/posts/2025-12-dremioframe-and-iceframe/) (2025-12-05): Meet DremioFrame and IceFrame, two Python libraries for Dremio and Apache Iceberg, with side-by-side examples for connecting, querying, and creating tables.
- [Introducing dremioframe - A Pythonic DataFrame Interface for Dremio](https://iceberglakehouse.com/posts/2025-11-introducing-dremioframe-dataframe-python-library/) (2025-11-29): Discover dremioframe, a new Python library that offers a DataFrame-like experience for interacting with Dremio's data lakehouse platform.
- [Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)](https://iceberglakehouse.com/posts/2025-11-dremio-next-gen-cloud-tutorial/) (2025-11-12): A hands-on tour of Dremio Cloud's free trial: sign up, curate a lakehouse, run SQL, try AI visualizations and AI functions, review jobs, and grant access.
- [An Exploration of the Commercial Iceberg Catalog Ecosystem](https://iceberglakehouse.com/posts/2025-10-exploring-commerical-apache-iceberg-catalogs/) (2025-10-21): Dive into the world of commercial Iceberg catalogs and discover how they enhance data lakehouse architectures for modern data engineering.
- [Building a Universal Lakehouse Catalog - Beyond Iceberg Tables](https://iceberglakehouse.com/posts/2025-10-building-universal-lakehouse-catalog/) (2025-10-17): Exploring paths to a universal lakehouse catalog that supports multiple data formats and engines, building on Apache Iceberg's success.
- [Intro to Apache Iceberg with Apache Polaris and Apache Spark](https://iceberglakehouse.com/posts/2025-10-intro-to-apache-iceberg-with-apache-polaris-and-apache-spark/) (2025-10-16): Learn how to leverage Apache Iceberg with Apache Polaris and Apache Spark to build scalable and efficient data lakehouses.
- [Composable Analytics with Agents - Leveraging Virtual Datasets and the Semantic Layer](https://iceberglakehouse.com/posts/2025-09-composable-analytics-with-agents/) (2025-09-17): How virtual datasets in Dremio's semantic layer let AI agents extend analytics models on demand through an MCP server instead of relying on static models.
- [Unlocking the Power of Agentic AI with Apache Iceberg and Dremio](https://iceberglakehouse.com/posts/2025-09-agentic-ai-dremio-apache-iceberg/) (2025-09-05): The three bottlenecks that stall agentic AI (data access, query speed, and semantic meaning) and how Apache Iceberg and Dremio address each one.
- [What is an API? And Why Data Architecture Depends on Them](https://iceberglakehouse.com/posts/2025-06-what-is-an-api/) (2025-06-23): What an API is, how SOAP, REST, RPC, and GraphQL differ, and why data architecture runs on APIs like S3, JDBC, ADBC, Hive Metastore, Glue, and Iceberg REST.
- [Decoding AWS EC2 Instance Type Names](https://iceberglakehouse.com/posts/2025-06-aws-instance-types/) (2025-06-18): How to read AWS EC2 instance names like c6g.large and r5d.4xlarge: what the family letter, generation number, suffixes like g and d, and size each mean.
- [Introduction to Data Engineering Concepts | What is Data Engineering?](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-01/) (2025-05-02): What data engineers do, how raw data moves through a pipeline to become usable, how the role differs from data science, and how the data stack evolved.
- [Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-02/) (2025-05-02): Common data sources, batch versus streaming ingestion, structured, semi-structured, and unstructured data, and how to design reliable ingestion pipelines.
- [Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-03/) (2025-05-02): The difference between ETL and ELT pipelines, why ELT took off with cloud warehouses, and when transforming data before loading it is still the better choice.
- [Introduction to Data Engineering Concepts | Batch Processing Fundamentals](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-04/) (2025-05-02): What batch processing is, how batch jobs run under the hood, the tools that build and schedule them, their limits, and where batch still beats streaming.
- [Introduction to Data Engineering Concepts | Streaming Data Fundamentals](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-05/) (2025-05-02): What streaming data is, how producers, message brokers, and stream processors work together, real-time use cases, and the challenges of streaming pipelines.
- [Introduction to Data Engineering Concepts | Data Modeling Basics](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-06/) (2025-05-02): Data modeling basics: OLTP versus OLAP systems, normalization and denormalization, star and snowflake schemas, and how to design models that can grow.
- [Introduction to Data Engineering Concepts | Data Warehousing Fundamentals](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-07/) (2025-05-02): What a data warehouse is and how it differs from an operational database: architecture, columnar storage, partitioning and clustering, and load patterns.
- [Introduction to Data Engineering Concepts | Data Lakes Explained](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-08/) (2025-05-02): What a data lake is, how schema-on-read works, how lakes differ from warehouses, and how raw, cleaned, and curated zones keep a lake from becoming a swamp.
- [Introduction to Data Engineering Concepts | Storage Formats and Compression](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-09/) (2025-05-02): How CSV, JSON, Avro, ORC, and Parquet differ, why columnar formats suit analytics, and how compression codecs affect file size and query speed.
- [Introduction to Data Engineering Concepts | Data Quality and Validation](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-10/) (2025-05-02): The dimensions of data quality, where quality breaks down in pipelines, how to validate data at each stage, and tools that catch problems before users do.
- [Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-11/) (2025-05-02): What metadata is, how data lineage helps with debugging and compliance, what data governance covers, and practices that build trust in data at scale.
- [Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-12/) (2025-05-02): How workflow orchestration goes beyond scheduling, what a DAG is, tools like Apache Airflow, and practices for retries, dependencies, and alerting.
- [Introduction to Data Engineering Concepts | Building Scalable Pipelines](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-13/) (2025-05-02): What makes a data pipeline scalable: parallel and distributed processing, less data movement, managing compute resources, and designing for change.
- [Introduction to Data Engineering Concepts | DevOps for Data Engineering](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-14/) (2025-05-02): How DevOps applies to data engineering: version control for pipelines, CI and automated tests, infrastructure as code, deployment, and incident response.
- [Introduction to Data Engineering Concepts | Cloud Data Platforms and the Modern Stack](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-15/) (2025-05-02): What the modern data stack is, how cloud platforms changed data engineering, the managed services they offer, and how to design data systems for agility.
- [Introduction to Data Engineering Concepts | Data Lakehouse Architecture Explained](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-16/) (2025-05-02): Why running a separate data lake and data warehouse causes problems, what a data lakehouse is, the open technologies behind it, and its main benefits.
- [Introduction to Data Engineering Concepts | Apache Iceberg, Arrow, and Polaris](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-17/) (2025-05-02): How Apache Iceberg tables, the Apache Arrow memory format, and the Apache Polaris catalog each solve a lakehouse problem and work together across engines.
- [Introduction to Data Engineering Concepts | The Power of Dremio in the Modern Lakehouse](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-18/) (2025-05-02): How Dremio fits a modern lakehouse: Iceberg clustering for data layout, Autonomous Reflections for speed, Polaris-based governance, and AI-ready data.
- [Using Helm with Kubernetes - A Guide to Helm Charts and Their Implementation](https://iceberglakehouse.com/posts/2025-02-using-helm-with-kubernetes/) (2025-02-19): What Helm is and when to use it on Kubernetes: Helm 3 architecture, chart structure, values.yaml, installing and upgrading releases, and handling secrets.
- [Intro to SQL using Apache Iceberg and Dremio](https://iceberglakehouse.com/posts/2024-11-intro-to-sql-with-dremio-and-apache-iceberg/) (2024-11-08): Learn SQL on Apache Iceberg tables with Dremio: set up Dremio, Nessie, and MinIO with Docker, then create, query, partition, branch, and time travel tables.
- [Introduction to Cargo and cargo.toml](https://iceberglakehouse.com/posts/2024-11-rust-cargo-toml/) (2024-11-05): A guide to Rust's Cargo.toml: package metadata, dependency versions, git and dev dependencies, optional features, build profiles, and workspaces.
- [Leveraging Python's Pattern Matching and Comprehensions for Data Analytics](https://iceberglakehouse.com/posts/2024-11-python-analytics-pattern-matching/) (2024-11-01): How to use Python's match statement and list, set, and dict comprehensions for data cleaning, categorizing, JSON mapping, filtering, and aggregation.
- [Data Modeling - Entities and Events](https://iceberglakehouse.com/posts/2024-10-data-modeling-events-entities/) (2024-10-30): The difference between modeling data as entities and as events, when each approach fits, key design choices, and how to combine both for full analysis.
- [Orchestrating Airflow DAGs with GitHub Actions - A Lightweight Approach to Data Curation Across Spark, Dremio, and Snowflake](https://iceberglakehouse.com/posts/2024-10-github-actions-dbt-airflow-data/) (2024-10-19): Trigger Airflow DAGs from GitHub Actions, ingest with Spark, build bronze, silver, and gold views in Dremio with dbt, and load the results into Snowflake.
- [A Deep Dive Into GitHub Actions From Software Development to Data Engineering](https://iceberglakehouse.com/posts/2024-10-intro-to-github-actions/) (2024-10-19): How GitHub Actions workflows, jobs, and runners work, how teams use them for tests, CI/CD, and security checks, and how to apply them to ETL pipelines.
- [Deep Dive into Data Apps with Streamlit](https://iceberglakehouse.com/posts/2024-9-deep-dive-into-data-apps-with-streamlit/) (2024-09-22): A hands-on guide to building data apps with Streamlit in a Python data science Docker image, covering widgets, layout, session state, caching, and deploys.
- [A Deep Dive into Docker Compose](https://iceberglakehouse.com/posts/2024-9-a-deep-dive-into-docker-compose/) (2024-09-21): How docker-compose.yml works: defining services, networks, and volumes, setting dependencies, and managing environment variables with .env files.

## Articles: Governance and Security

- [Guardrails for AI on Company Data](https://iceberglakehouse.com/posts/guardrails-ai-company-data/) (2026-09-10): The control surfaces that actually contain damage once an agent is fooled: identity, permissions, audit trails, and prompt injection at the query layer.
- [How Iceberg Catalogs Hand Engines Storage Access](https://iceberglakehouse.com/posts/iceberg-vended-credentials/) (2026-09-10): Credential vending end to end: the wire protocol, scoped access on each cloud, remote signing, credential lifetime on long jobs, and failures that look like bugs.
- [Deleting User Data From an Immutable Lakehouse: GDPR Hard Deletes on Iceberg](https://iceberglakehouse.com/posts/gdpr-hard-deletes-on-iceberg/) (2026-09-02): How to turn a logical delete on immutable Iceberg into a physical erasure across snapshots, versions, replicas, and downstream copies.
- [Metadata Platforms in 2026: DataHub, OpenMetadata, Atlan, and Catalog Convergence](https://iceberglakehouse.com/posts/metadata-platforms-in-2026/) (2026-09-02): How the technical catalog and the metadata platform are converging in 2026, and how to arrange the two layers for a lakehouse.
- [Governance-as-Code for the Lakehouse: Managing REST Catalog RBAC and Masking in Git](https://iceberglakehouse.com/posts/governance-as-code-lakehouse-rest-catalog/) (2026-08-25): Put REST catalog RBAC and masking in Git. How to review grants, apply them safely, and keep lakehouse access from drifting.
- [Policy-Aware Lakehouse Telemetry: Building Auditable AI Records on Apache Iceberg for the EU AI Act Era](https://iceberglakehouse.com/posts/policy-aware-lakehouse-telemetry-eu-ai-act/) (2026-08-19): Policy-aware lakehouse telemetry for the EU AI Act era: build auditable, immutable, governed AI decision records on Apache Iceberg.
- [Securing the Agentic Lakehouse Gateway: A Threat Model for Prompt Injection, Exfiltration, and the Firewall That Reads Sentences](https://iceberglakehouse.com/posts/securing-agentic-lakehouse-gateway/) (2026-08-19): A threat model for the agentic lakehouse gateway covering prompt injection, exfiltration, and the firewall that reads sentences.
- [Policy Layer for Autonomous AI Data Agents](https://iceberglakehouse.com/posts/policy-layer-security-policy-enforcement-autonomous-multi-agent-systems/) (2026-07-13): Why a prompt is not a security boundary for autonomous data agents, and how a runtime policy layer enforces query limits, egress quotas, and isolation pools.

## Articles: Other

- [Where Lock-In Went](https://iceberglakehouse.com/posts/where-lock-in-went/) (2026-09-10): The format war ended and exit cost did not: where lock-in relocated after open tables won, how to measure it, and which costs are worth keeping down.
- [The Filters We Build: Scams, Slop, and the Search for Signal, From Radio Ads to AI](https://iceberglakehouse.com/posts/cognitive-filters-media-evolution/) (2026-07-24): How every media era from radio to AI forces humanity to build new cognitive filters, the shield against scams and the sieve for finding signal in an ocean.
- [Open Table Format Benchmarks: Why They Require Critical Evaluation](https://iceberglakehouse.com/posts/2026-05-22-open-table-format-benchmarks-guide/) (2026-05-22): An in-depth analysis of open table format benchmarks comparing Apache Iceberg, Delta Lake, and Apache Hudi, detailing the pitfalls of standard benchmarks.
- [Apache Iceberg SCD Type 2 and CDC Patterns: Building Historical Lakehouse Tables](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-scd-type-2-cdc-patterns/) (2026-05-22): A deep dive into implementing Slowly Changing Dimension Type 2 (SCD Type 2) patterns and Change Data Capture (CDC) pipelines on Apache Iceberg, using.
- [Iceberg Catalogs Compared: REST, Glue, Hive, Polaris, Nessie](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-catalogs-explained/) (2026-05-22): Iceberg catalogs track each table's current metadata file and make commits atomic. Compare REST, Glue, Hive, Nessie, Polaris, S3 Tables, and Snowflake Horizon.
- [Iceberg MERGE INTO in Spark: Upserts, Copy-on-Write vs MoR](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-spark-dml-evolution/) (2026-05-22): Run Iceberg MERGE INTO, UPDATE, and DELETE in Spark with Iceberg 1.12.0, pick copy-on-write or merge-on-read per operation, and evolve schemas without rewrites.
- [Building a Multicloud Agentic Lakehouse Reference Architecture](https://iceberglakehouse.com/posts/2026-05-22-multicloud-agentic-lakehouse-reference-architecture/) (2026-05-22): A reference architecture for building an open, multicloud Data Lakehouse optimized for AI Agents using Apache Polaris, Apache Iceberg, and Dremio.
- [Common Misconceptions About Data Lakehouse and Apache Iceberg](https://iceberglakehouse.com/posts/2026-05-22-data-lakehouse-and-iceberg-misconceptions/) (2026-05-22): Common lakehouse and Apache Iceberg myths, corrected: why a lakehouse is more than a database on S3, what catalogs do, and why partitions don't lock you in.

## Apache Iceberg Knowledge Base

- [Knowledge Base Index](https://iceberglakehouse.com/iceberg/)
- [Amazon S3 Tables for Apache Iceberg](https://iceberglakehouse.com/iceberg/amazon-s3-tables/)
- [Apache DataFusion](https://iceberglakehouse.com/iceberg/apache-datafusion/)
- [Apache Polaris Catalog](https://iceberglakehouse.com/iceberg/apache-polaris-catalog/)
- [Apache XTable Translations](https://iceberglakehouse.com/iceberg/apache-xtable-translations/)
- [AWS Athena and Apache Iceberg](https://iceberglakehouse.com/iceberg/aws-athena-iceberg/)
- [Amazon EMR and Apache Iceberg](https://iceberglakehouse.com/iceberg/aws-emr-iceberg/)
- [AWS Glue Catalog for Apache Iceberg](https://iceberglakehouse.com/iceberg/aws-glue-catalog/)
- [Microsoft Fabric and Apache Iceberg](https://iceberglakehouse.com/iceberg/azure-fabric-iceberg/)
- [Bauplan](https://iceberglakehouse.com/iceberg/bauplan/)
- [BigQuery and Apache Iceberg](https://iceberglakehouse.com/iceberg/bigquery-apache-iceberg/)
- [Catalog Federation](https://iceberglakehouse.com/iceberg/catalog-federation/)
- [Catalog Namespaces](https://iceberglakehouse.com/iceberg/catalog-namespaces/)
- [CDC Log Ingestion Pipelines](https://iceberglakehouse.com/iceberg/cdc-log-ingestion-pipelines/)
- [ClickHouse](https://iceberglakehouse.com/iceberg/clickhouse/)
- [Columnar Memory Layouts](https://iceberglakehouse.com/iceberg/columnar-memory-layouts/)
- [Data Lakehouse](https://iceberglakehouse.com/iceberg/data-lakehouse/)
- [Data Lineage Tracking](https://iceberglakehouse.com/iceberg/data-lineage-tracking/)
- [Databricks and Apache Iceberg](https://iceberglakehouse.com/iceberg/databricks-iceberg/)
- [dbt and Apache Iceberg](https://iceberglakehouse.com/iceberg/dbt-apache-iceberg/)
- [Debezium CDC Engines](https://iceberglakehouse.com/iceberg/debezium-cdc-engines/)
- [Decoupled Compute and Storage](https://iceberglakehouse.com/iceberg/decoupled-compute-and-storage/)
- [Delta Lake UniForm Metadata](https://iceberglakehouse.com/iceberg/delta-lake-uniform-metadata/)
- [Apache Doris and Apache Iceberg](https://iceberglakehouse.com/iceberg/doris-apache-iceberg/)
- [Dremio Acceleration Engine](https://iceberglakehouse.com/iceberg/dremio-acceleration-engine/)
- [Dremio Aggregation Reflections](https://iceberglakehouse.com/iceberg/dremio-aggregation-reflections/)
- [Dremio and Apache Iceberg](https://iceberglakehouse.com/iceberg/dremio-apache-iceberg/)
- [Dremio Arrow Flight SQL](https://iceberglakehouse.com/iceberg/dremio-arrow-flight-sql/)
- [Dremio Column-Level Masking](https://iceberglakehouse.com/iceberg/dremio-column-level-masking/)
- [Dremio Columnar Cloud Cache (C3)](https://iceberglakehouse.com/iceberg/dremio-columnar-cloud-cache-c3/)
- [Dremio Coordinator Node](https://iceberglakehouse.com/iceberg/dremio-coordinator-node/)
- [Dremio Data Reflections Matching](https://iceberglakehouse.com/iceberg/dremio-data-reflections-matching/)
- [Dremio Engine Auto-scaling](https://iceberglakehouse.com/iceberg/dremio-engine-auto-scaling/)
- [Dremio External Queries](https://iceberglakehouse.com/iceberg/dremio-external-queries/)
- [Dremio Iceberg Metadata Sync](https://iceberglakehouse.com/iceberg/dremio-iceberg-metadata-sync/)
- [Dremio Join Co-segmentation](https://iceberglakehouse.com/iceberg/dremio-join-co-segmentation/)
- [Dremio LDAP Integration](https://iceberglakehouse.com/iceberg/dremio-ldap-integration/)
- [Dremio Metadata Caching](https://iceberglakehouse.com/iceberg/dremio-metadata-caching/)
- [Dremio Parquet Vectorized Reader](https://iceberglakehouse.com/iceberg/dremio-parquet-vectorized-reader/)
- [Dremio Physical Datasets (PDS)](https://iceberglakehouse.com/iceberg/dremio-physical-datasets-pds/)
- [Dremio Raw Reflections](https://iceberglakehouse.com/iceberg/dremio-raw-reflections/)
- [Dremio Reflections](https://iceberglakehouse.com/iceberg/dremio-reflections/)
- [Dremio Row-Level Security (RLS)](https://iceberglakehouse.com/iceberg/dremio-row-level-security-rls/)
- [Dremio Sabot Engine](https://iceberglakehouse.com/iceberg/dremio-sabot-engine/)
- [Dremio Spaces](https://iceberglakehouse.com/iceberg/dremio-spaces/)
- [Dremio SQL Runner](https://iceberglakehouse.com/iceberg/dremio-sql-runner/)
- [Dremio User Defined Functions (UDFs)](https://iceberglakehouse.com/iceberg/dremio-user-defined-functions-udfs/)
- [Dremio Virtual Datasets (VDS)](https://iceberglakehouse.com/iceberg/dremio-virtual-datasets-vds/)
- [DuckDB and Apache Iceberg](https://iceberglakehouse.com/iceberg/duckdb-apache-iceberg/)
- [Dynamic Filter Pushdown](https://iceberglakehouse.com/iceberg/dynamic-filter-pushdown/)
- [Apache Flink and Apache Iceberg](https://iceberglakehouse.com/iceberg/flink-apache-iceberg/)
- [Glue Catalog IAM Policies](https://iceberglakehouse.com/iceberg/glue-catalog-iam-policies/)
- [Glue Catalog Lake Formation](https://iceberglakehouse.com/iceberg/glue-catalog-lake-formation/)
- [Google Cloud BigLake](https://iceberglakehouse.com/iceberg/google-cloud-biglake/)
- [Google Cloud and Apache Iceberg](https://iceberglakehouse.com/iceberg/google-cloud-iceberg/)
- [Apache Gravitino](https://iceberglakehouse.com/iceberg/gravitino-catalog/)
- [Hive and Apache Iceberg](https://iceberglakehouse.com/iceberg/hive-apache-iceberg/)
- [Iceberg Access Control Patterns](https://iceberglakehouse.com/iceberg/iceberg-access-control/)
- [ACID Transactions in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-acid-transactions/)
- [Agentic Lakehouse](https://iceberglakehouse.com/iceberg/iceberg-agentic-lakehouse/)
- [Iceberg AI Readiness](https://iceberglakehouse.com/iceberg/iceberg-ai-readiness/)
- [Iceberg AI Semantic Layer](https://iceberglakehouse.com/iceberg/iceberg-ai-semantic-layer/)
- [Apache Airflow and Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-apache-airflow/)
- [Iceberg Apache Arrow Flight](https://iceberglakehouse.com/iceberg/iceberg-arrow-flight/)
- [Iceberg Audit Logging](https://iceberglakehouse.com/iceberg/iceberg-audit-logging/)
- [Iceberg Avro Metadata Format](https://iceberglakehouse.com/iceberg/iceberg-avro-format/)
- [Iceberg Bin-Packing Compaction](https://iceberglakehouse.com/iceberg/iceberg-bin-packing-compaction/)
- [Iceberg Bloom Filters](https://iceberglakehouse.com/iceberg/iceberg-bloom-filters/)
- [Iceberg Branching and Tagging](https://iceberglakehouse.com/iceberg/iceberg-branching-tagging/)
- [Iceberg Bucket Partition Transform](https://iceberglakehouse.com/iceberg/iceberg-bucket-partition-transform/)
- [Iceberg Catalog Migration](https://iceberglakehouse.com/iceberg/iceberg-catalog-migration/)
- [What is an Iceberg Catalog?](https://iceberglakehouse.com/iceberg/iceberg-catalog/)
- [Iceberg CDC (Change Data Capture)](https://iceberglakehouse.com/iceberg/iceberg-cdc/)
- [Iceberg Table Clustering](https://iceberglakehouse.com/iceberg/iceberg-clustering/)
- [Iceberg Column Mapping](https://iceberglakehouse.com/iceberg/iceberg-column-mapping/)
- [Iceberg Column Projection](https://iceberglakehouse.com/iceberg/iceberg-column-projection/)
- [Iceberg Table Compaction](https://iceberglakehouse.com/iceberg/iceberg-compaction/)
- [Iceberg Concurrent Write Handling](https://iceberglakehouse.com/iceberg/iceberg-concurrent-writes/)
- [Copy-on-Write (CoW) in Iceberg](https://iceberglakehouse.com/iceberg/iceberg-copy-on-write/)
- [Iceberg Cost Optimization](https://iceberglakehouse.com/iceberg/iceberg-cost-optimization/)
- [Iceberg Data Files](https://iceberglakehouse.com/iceberg/iceberg-data-files/)
- [Iceberg Data Lineage](https://iceberglakehouse.com/iceberg/iceberg-data-lineage/)
- [Iceberg Data Masking](https://iceberglakehouse.com/iceberg/iceberg-data-masking/)
- [Iceberg Data Mesh Architecture](https://iceberglakehouse.com/iceberg/iceberg-data-mesh/)
- [Iceberg Data Skipping](https://iceberglakehouse.com/iceberg/iceberg-data-skipping/)
- [Iceberg Date/Time Partition Transforms](https://iceberglakehouse.com/iceberg/iceberg-date-time-partition-transforms/)
- [Iceberg Decimal Type Widening](https://iceberglakehouse.com/iceberg/iceberg-decimal-type-widening/)
- [Iceberg Delete Files](https://iceberglakehouse.com/iceberg/iceberg-delete-files/)
- [Iceberg Deletion Vectors](https://iceberglakehouse.com/iceberg/iceberg-deletion-vectors/)
- [Iceberg DynamoDB Catalog](https://iceberglakehouse.com/iceberg/iceberg-dynamodb-catalog/)
- [Iceberg Encryption](https://iceberglakehouse.com/iceberg/iceberg-encryption/)
- [Iceberg Equality Deletes](https://iceberglakehouse.com/iceberg/iceberg-equality-deletes/)
- [Expire Snapshots in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-expire-snapshots/)
- [Iceberg Feature Store](https://iceberglakehouse.com/iceberg/iceberg-feature-store/)
- [Iceberg File Content Type](https://iceberglakehouse.com/iceberg/iceberg-file-content-type/)
- [Iceberg FileIO API](https://iceberglakehouse.com/iceberg/iceberg-file-io/)
- [Iceberg File Path Spec](https://iceberglakehouse.com/iceberg/iceberg-file-path-spec/)
- [Hidden Partitioning in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-hidden-partitioning/)
- [Iceberg Hive Catalog Lock Manager](https://iceberglakehouse.com/iceberg/iceberg-hive-catalog-lock-manager/)
- [Hive Metastore Catalog for Iceberg](https://iceberglakehouse.com/iceberg/iceberg-hive-metastore/)
- [Iceberg Identity Partition Transform](https://iceberglakehouse.com/iceberg/iceberg-identity-partition-transform/)
- [Iceberg Incremental Reads](https://iceberglakehouse.com/iceberg/iceberg-incremental-read/)
- [Iceberg JDBC Catalog Locks](https://iceberglakehouse.com/iceberg/iceberg-jdbc-catalog-locks/)
- [Iceberg JDBC Catalog](https://iceberglakehouse.com/iceberg/iceberg-jdbc-catalog/)
- [Iceberg Lakehouse Federation](https://iceberglakehouse.com/iceberg/iceberg-lakehouse-federation/)
- [Iceberg LLM Grounding and RAG for Structured Data](https://iceberglakehouse.com/iceberg/iceberg-llm-grounding/)
- [Iceberg Lock Manager](https://iceberglakehouse.com/iceberg/iceberg-lock-manager/)
- [Iceberg Maintenance Scheduling](https://iceberglakehouse.com/iceberg/iceberg-maintenance-scheduling/)
- [Iceberg Manifest Entry Schema](https://iceberglakehouse.com/iceberg/iceberg-manifest-entry-schema/)
- [Iceberg Manifest Entry Status](https://iceberglakehouse.com/iceberg/iceberg-manifest-entry-status/)
- [Iceberg Manifest File](https://iceberglakehouse.com/iceberg/iceberg-manifest-file/)
- [Iceberg Manifest List Schema](https://iceberglakehouse.com/iceberg/iceberg-manifest-list-schema/)
- [Iceberg Manifest List](https://iceberglakehouse.com/iceberg/iceberg-manifest-list/)
- [Iceberg Manifest Merging](https://iceberglakehouse.com/iceberg/iceberg-manifest-merging/)
- [MCP and Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-mcp/)
- [Medallion Architecture with Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-medallion-architecture/)
- [Merge-on-Read (MoR) in Iceberg](https://iceberglakehouse.com/iceberg/iceberg-merge-on-read/)
- [Iceberg Metadata File](https://iceberglakehouse.com/iceberg/iceberg-metadata-file/)
- [Iceberg Metadata Pruning](https://iceberglakehouse.com/iceberg/iceberg-metadata-pruning/)
- [Iceberg Metadata Table Files](https://iceberglakehouse.com/iceberg/iceberg-metadata-table-files/)
- [Iceberg Metadata Table History](https://iceberglakehouse.com/iceberg/iceberg-metadata-table-history/)
- [Iceberg Metadata Table Snapshots](https://iceberglakehouse.com/iceberg/iceberg-metadata-table-snapshots/)
- [Iceberg Metrics Mode](https://iceberglakehouse.com/iceberg/iceberg-metrics-mode/)
- [Iceberg Table Migration from Hive](https://iceberglakehouse.com/iceberg/iceberg-migration-hive/)
- [Iceberg Multi-Catalog Architecture](https://iceberglakehouse.com/iceberg/iceberg-multi-catalog/)
- [Iceberg Multi-Tenancy Patterns](https://iceberglakehouse.com/iceberg/iceberg-multi-tenant/)
- [Iceberg Natural Language Analytics](https://iceberglakehouse.com/iceberg/iceberg-natural-language/)
- [Iceberg Nested Type System](https://iceberglakehouse.com/iceberg/iceberg-nested-type-system/)
- [Iceberg Open Table Format vs. Delta Lake vs. Apache Hudi](https://iceberglakehouse.com/iceberg/iceberg-open-table-format/)
- [Iceberg Optimistic Concurrency Control (OCC)](https://iceberglakehouse.com/iceberg/iceberg-optimistic-concurrency-control-occ/)
- [Apache Iceberg ORC Format](https://iceberglakehouse.com/iceberg/iceberg-orc-format/)
- [Iceberg Orphan Files Penalty](https://iceberglakehouse.com/iceberg/iceberg-orphan-files-penalty/)
- [Iceberg Orphan Files](https://iceberglakehouse.com/iceberg/iceberg-orphan-files/)
- [Iceberg Parent Snapshot ID](https://iceberglakehouse.com/iceberg/iceberg-parent-snapshot-id/)
- [Apache Parquet and Iceberg](https://iceberglakehouse.com/iceberg/iceberg-parquet/)
- [Partition Evolution in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-partition-evolution/)
- [Iceberg Partition-Level Compaction](https://iceberglakehouse.com/iceberg/iceberg-partition-level-compaction/)
- [Iceberg Performance Tuning Guide](https://iceberglakehouse.com/iceberg/iceberg-performance-tuning/)
- [Iceberg Positional Deletes](https://iceberglakehouse.com/iceberg/iceberg-positional-deletes/)
- [Iceberg Predicate Pushdown](https://iceberglakehouse.com/iceberg/iceberg-predicate-pushdown/)
- [Iceberg Puffin Files](https://iceberglakehouse.com/iceberg/iceberg-puffin-files/)
- [Iceberg REST Catalog API Reference](https://iceberglakehouse.com/iceberg/iceberg-rest-catalog-api/)
- [Iceberg REST Catalog](https://iceberglakehouse.com/iceberg/iceberg-rest-catalog/)
- [Iceberg Rewrite Manifests](https://iceberglakehouse.com/iceberg/iceberg-rewrite-manifests/)
- [Iceberg Rollback Snapshot Procedures](https://iceberglakehouse.com/iceberg/iceberg-rollback-snapshot-procedures/)
- [Iceberg Table Rollback](https://iceberglakehouse.com/iceberg/iceberg-rollback/)
- [Row-Level Deletes in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-row-level-deletes/)
- [Schema Evolution in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-schema-evolution/)
- [Iceberg Sequence Number Inheritance](https://iceberglakehouse.com/iceberg/iceberg-sequence-number-inheritance/)
- [Iceberg Sequence Number](https://iceberglakehouse.com/iceberg/iceberg-sequence-number/)
- [Small File Problem in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-small-file-problem/)
- [Iceberg Snapshot Expiration Age](https://iceberglakehouse.com/iceberg/iceberg-snapshot-expiration-age/)
- [Iceberg Snapshot References](https://iceberglakehouse.com/iceberg/iceberg-snapshot-references/)
- [Iceberg Snapshot Summary](https://iceberglakehouse.com/iceberg/iceberg-snapshot-summary/)
- [Apache Iceberg Snapshot](https://iceberglakehouse.com/iceberg/iceberg-snapshot/)
- [Iceberg Sort-Based Compaction](https://iceberglakehouse.com/iceberg/iceberg-sort-based-compaction/)
- [Iceberg Sort Order](https://iceberglakehouse.com/iceberg/iceberg-sort-order/)
- [Iceberg Spark Procedure add_files](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-add-files/)
- [Iceberg Spark Procedure expire_snapshots](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-expire-snapshots/)
- [Iceberg Spark Procedure register_table](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-register-table/)
- [Iceberg Spark Procedure remove_orphan_files](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-remove-orphan-files/)
- [Iceberg Spark Procedure rewrite_data_files](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-rewrite-data-files/)
- [Iceberg Spark Procedure rewrite_manifests](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-rewrite-manifests/)
- [Iceberg Spark Procedure rewrite_position_deletes](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-rewrite-position-deletes/)
- [Apache Iceberg Spec v1 vs v2](https://iceberglakehouse.com/iceberg/iceberg-spec-v1-vs-v2/)
- [Iceberg Spec V3 File Encryption](https://iceberglakehouse.com/iceberg/iceberg-spec-v3-file-encryption/)
- [Iceberg Spec V3 Object-Store Storage Layout](https://iceberglakehouse.com/iceberg/iceberg-spec-v3-object-store-storage-layout/)
- [Apache Iceberg Spec v3](https://iceberglakehouse.com/iceberg/iceberg-spec-v3/)
- [Apache Iceberg Spec v4 (Current State)](https://iceberglakehouse.com/iceberg/iceberg-spec-v4/)
- [Iceberg Table Statistics (Puffin)](https://iceberglakehouse.com/iceberg/iceberg-statistics/)
- [Iceberg Streaming Ingestion](https://iceberglakehouse.com/iceberg/iceberg-streaming/)
- [Iceberg Table Design Best Practices](https://iceberglakehouse.com/iceberg/iceberg-table-design/)
- [Apache Iceberg Table Format](https://iceberglakehouse.com/iceberg/iceberg-table-format/)
- [Iceberg Table Metadata Schema](https://iceberglakehouse.com/iceberg/iceberg-table-metadata-schema/)
- [Iceberg Table Properties](https://iceberglakehouse.com/iceberg/iceberg-table-properties/)
- [Time Travel in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-time-travel/)
- [Iceberg Truncate Partition Transform](https://iceberglakehouse.com/iceberg/iceberg-truncate-partition-transform/)
- [Iceberg Upsert (MERGE INTO)](https://iceberglakehouse.com/iceberg/iceberg-upsert/)
- [Iceberg Views](https://iceberglakehouse.com/iceberg/iceberg-views/)
- [Apache Iceberg vs Apache Hudi](https://iceberglakehouse.com/iceberg/iceberg-vs-apache-hudi/)
- [Apache Iceberg vs Delta Lake](https://iceberglakehouse.com/iceberg/iceberg-vs-delta-lake/)
- [Write-Audit-Publish (WAP) Pattern](https://iceberglakehouse.com/iceberg/iceberg-wap-pattern/)
- [Iceberg Write Distribution Modes](https://iceberglakehouse.com/iceberg/iceberg-write-distribution/)
- [Iceberg Z-Order Compaction](https://iceberglakehouse.com/iceberg/iceberg-z-order-compaction/)
- [Z-Order Clustering in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-zorder/)
- [Apache Kafka and Apache Iceberg](https://iceberglakehouse.com/iceberg/kafka-apache-iceberg/)
- [LangChain and Apache Iceberg](https://iceberglakehouse.com/iceberg/langchain-iceberg/)
- [Microsoft Fabric OneLake](https://iceberglakehouse.com/iceberg/microsoft-fabric-onelake/)
- [Modern Data Stack (MDS)](https://iceberglakehouse.com/iceberg/modern-data-stack-mds/)
- [Nessie Git-like Branching](https://iceberglakehouse.com/iceberg/nessie-git-like-branching/)
- [Nessie Merging](https://iceberglakehouse.com/iceberg/nessie-merging/)
- [Nessie Tagging](https://iceberglakehouse.com/iceberg/nessie-tagging/)
- [Object Storage Prefix Hashing](https://iceberglakehouse.com/iceberg/object-storage-prefix-hashing/)
- [Open Table Format Comparison (Iceberg, Delta Lake, Hudi, Paimon)](https://iceberglakehouse.com/iceberg/open-table-format-comparison/)
- [Polaris Catalog Sharing](https://iceberglakehouse.com/iceberg/polaris-catalog-sharing/)
- [Polaris RBAC Model](https://iceberglakehouse.com/iceberg/polaris-rbac-model/)
- [Polaris Service Principals](https://iceberglakehouse.com/iceberg/polaris-service-principals/)
- [Presto and Apache Iceberg](https://iceberglakehouse.com/iceberg/presto-apache-iceberg/)
- [Project Nessie](https://iceberglakehouse.com/iceberg/project-nessie/)
- [PuppyGraph](https://iceberglakehouse.com/iceberg/puppygraph/)
- [PyIceberg: Python Library for Apache Iceberg](https://iceberglakehouse.com/iceberg/pyiceberg/)
- [Read Amplification](https://iceberglakehouse.com/iceberg/read-amplification/)
- [REST Catalog Credential Vending](https://iceberglakehouse.com/iceberg/rest-catalog-credential-vending/)
- [REST Catalog OAuth2 Token Flow](https://iceberglakehouse.com/iceberg/rest-catalog-oauth2-token-flow/)
- [Single Source of Truth (SSOT)](https://iceberglakehouse.com/iceberg/single-source-of-truth-ssot/)
- [Snowflake External Table Catalog Sync](https://iceberglakehouse.com/iceberg/snowflake-external-table-catalog-sync/)
- [Snowflake Iceberg Tables](https://iceberglakehouse.com/iceberg/snowflake-iceberg-tables/)
- [Snowflake Managed Iceberg Tables](https://iceberglakehouse.com/iceberg/snowflake-managed-iceberg-tables/)
- [Snowflake Open Catalog](https://iceberglakehouse.com/iceberg/snowflake-open-catalog/)
- [Space Amplification](https://iceberglakehouse.com/iceberg/space-amplification/)
- [Apache Spark and Apache Iceberg](https://iceberglakehouse.com/iceberg/spark-apache-iceberg/)
- [Spice.ai](https://iceberglakehouse.com/iceberg/spiceai/)
- [Split Planning Loops](https://iceberglakehouse.com/iceberg/split-planning-loops/)
- [StarRocks and Apache Iceberg](https://iceberglakehouse.com/iceberg/starrocks-apache-iceberg/)
- [Apache Superset and Apache Iceberg](https://iceberglakehouse.com/iceberg/superset-apache-iceberg/)
- [Time To First Byte (TTFB)](https://iceberglakehouse.com/iceberg/time-to-first-byte-ttfb/)
- [Trino and Apache Iceberg](https://iceberglakehouse.com/iceberg/trino-apache-iceberg/)
- [Unity Catalog Delta-Iceberg Compatibility](https://iceberglakehouse.com/iceberg/unity-catalog-delta-iceberg-compatibility/)
- [VeloDB](https://iceberglakehouse.com/iceberg/velodb/)
- [Vortex File Format](https://iceberglakehouse.com/iceberg/vortex-file-format/)
- [What is Apache Iceberg?](https://iceberglakehouse.com/iceberg/what-is-apache-iceberg/)
- [Write Amplification](https://iceberglakehouse.com/iceberg/write-amplification/)
- [Zero-Copy Cloning](https://iceberglakehouse.com/iceberg/zero-copy-cloning/)

## Events

- [Agentic Lakehouse Events](https://luma.com/agenticlakehouse): global meetups and webinars on agentic analytics
- [Data Lakehouse Hub Events](https://luma.com/DataLakehouseHub): global lakehouse meetups, linkups and webinars

## Community

- [Data Lakehouse Hub Slack](https://join.slack.com/t/thedatalakehousehub/shared_invite/zt-274yc8sza-mI2zhCW8LGkOh1uxuf8T5Q): practitioner community for lakehouse architecture
- [Data Events Slack](https://join.slack.com/t/data-events/shared_invite/zt-38vgrooy9-U9ral_gr3NAz_Siih1QwmQ): announcements for data conferences and meetups
- [Data & Tech Slack](https://join.slack.com/t/datatechcommunity/shared_invite/zt-12xrk4qmd-y~6jUFFd7kdaLhgLURKwoA): broader data and technology community
- [r/datalakehouseandai](https://www.reddit.com/r/datalakehouseandai/): subreddit for data lakehouse and AI discussion
- [Data Lakehouse Hub on LinkedIn](https://www.linkedin.com/company/data-lakehouse-hub/): company page for the Data Lakehouse Hub
- [Alex Merced Tech on YouTube](https://www.youtube.com/@AlexMercedCoder): software development and engineering channel
- [Alex Merced Data & AI on YouTube](https://www.youtube.com/@alexmerceddata): data lakehouse and AI channel