Iceberg Natural Language Analytics

Natural Language Analytics on Apache Iceberg

Natural language analytics (NLA) is the ability to query a data lakehouse by asking questions in plain English — “What was our fastest-growing product category last quarter?” — and receiving data-backed, accurate answers without writing SQL. Apache Iceberg is the ideal storage foundation for NLA because its governed, versioned, well-documented tables provide the authoritative data that LLMs need to generate trustworthy answers.

The Three-Layer NLA Architecture

Every production natural language analytics system over Iceberg has three layers:

1. Semantic Layer → business context (what does each table/column mean?)
2. LLM Layer     → natural language → SQL translation (grounded in semantic context)
3. Execution Layer → SQL → Iceberg data → results

Without the semantic layer, the LLM generates SQL against raw schemas it doesn’t understand — producing incorrect or hallucinated results. Without the execution layer, there’s no connection to actual data. Without the LLM layer, it’s not natural language.

Layer 1: The Semantic Layer

The semantic layer is the most important component for NLA accuracy. It translates raw Iceberg schemas into AI-understandable business context:

# Semantic catalog: register tables with business context
semantic_catalog = {
    "analytics.orders": {
        "description": "Canonical order table. Every order placed by customers since 2020-01-01.",
        "columns": {
            "order_id": "Unique identifier for each order. Never null.",
            "customer_id": "Foreign key to analytics.customers. Links order to customer profile.",
            "total": "Net order revenue in USD after discounts. Excludes tax and shipping.",
            "order_date": "Calendar date the order was placed (not shipped or delivered).",
            "status": "Order lifecycle status. Values: PENDING, PROCESSING, SHIPPED, DELIVERED, CANCELLED.",
            "region": "Geographic sales region. Values: AMER (Americas), EMEA (Europe/Middle East/Africa), APAC (Asia Pacific)."
        },
        "metrics": {
            "revenue": "SUM(total) WHERE status IN ('SHIPPED', 'DELIVERED')",
            "order_count": "COUNT(*) WHERE status != 'CANCELLED'",
            "aov": "SUM(total) / COUNT(*) WHERE status IN ('SHIPPED', 'DELIVERED')"
        },
        "business_rules": [
            "Only SHIPPED and DELIVERED orders count as revenue",
            "CANCELLED orders are excluded from all revenue metrics",
            "order_date is in UTC timezone"
        ]
    }
}

Layer 2: LLM Text-to-SQL (NL2SQL)

The LLM generates SQL from natural language, grounded in semantic context:

import anthropic

client = anthropic.Anthropic()

def natural_language_to_sql(question: str, table_context: dict) -> str:
    """Convert natural language question to SQL using LLM + semantic context."""

    context = f"""
    Table: analytics.orders
    Description: {table_context['description']}

    Columns:
    {chr(10).join([f"- {col}: {desc}" for col, desc in table_context['columns'].items()])}

    Pre-defined metrics:
    {chr(10).join([f"- {m}: {d}" for m, d in table_context['metrics'].items()])}

    Business rules:
    {chr(10).join([f"- {r}" for r in table_context['business_rules']])}
    """

    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1024,
        system=f"""You are a SQL expert. Generate a single, correct SQL SELECT query to answer
        the user's question using the following table context. Return ONLY the SQL query, no explanation.

        {context}""",
        messages=[{"role": "user", "content": f"Question: {question}"}]
    )

    return response.content[0].text.strip()

Layer 3: SQL Execution Against Iceberg

Execute the generated SQL against Iceberg:

import duckdb
from pyiceberg.catalog import load_catalog

def execute_on_iceberg(sql: str) -> pd.DataFrame:
    """Execute SQL against Iceberg tables, return results."""
    catalog = load_catalog("my_catalog", **{...})
    table = catalog.load_table("analytics.orders")

    arrow = table.scan().to_arrow()
    conn = duckdb.connect()
    conn.register("orders", arrow)

    return conn.execute(sql).fetchdf()

Full NLA Pipeline

def answer_business_question(question: str) -> str:
    """
    End-to-end natural language analytics over Iceberg.
    1. Get semantic context
    2. Generate SQL from question
    3. Execute against Iceberg
    4. Synthesize natural language answer
    """
    # 1. Get semantic context for relevant tables
    context = semantic_catalog["analytics.orders"]

    # 2. Generate SQL
    sql = natural_language_to_sql(question, context)
    print(f"Generated SQL: {sql}")

    # 3. Execute against Iceberg
    results = execute_on_iceberg(sql)

    # 4. Synthesize answer
    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1024,
        system="You are a business analyst. Synthesize the data into a clear, concise answer.",
        messages=[{
            "role": "user",
            "content": f"""
            Question: {question}

            Data:
            {results.to_markdown()}

            Provide a clear business answer.
            """
        }]
    )

    return response.content[0].text

# Example
answer = answer_business_question(
    "Which regions showed revenue growth above 20% this quarter compared to last?"
)
print(answer)

Dremio’s AI Agent: Production NLA on Iceberg

Dremio’s AI Agent is the production-ready implementation of NLA over Iceberg:

1. Semantic Layer: Dremio Virtual Datasets with Wiki documentation and column descriptions provide the business context.
2. NL2SQL: Dremio’s AI Agent uses the semantic layer to generate highly accurate SQL.
3. Execution: Dremio’s Intelligent Query Engine executes against Iceberg via Apache Polaris.
4. Governance: All queries are authenticated, authorized, and audit-logged.
5. MCP Integration: AI Agents in Claude, ChatGPT, or custom applications connect via MCP.

The Dremio approach eliminates the need to build and maintain a custom semantic catalog — the Dremio platform manages the business context layer that makes NLA accurate.

Common NLA Failure Modes (and How Iceberg + Semantic Layer Fixes Them)