Tag:data engineering

Articles tagged "data engineering", page 2.

• 28 MIN READ•May 22, 2026

  Iceberg Catalogs Compared: REST, Glue, Hive, Polaris, Nessie

  Iceberg catalogs track each table's current metadata file and make commits atomic. Compare REST, Glue, Hive, Nessie, Polaris, S3 Tables, and Snowflake Horizon.

  apache icebergcatalogsNessie
• 27 MIN READ•May 22, 2026

  Iceberg MERGE INTO in Spark: Upserts, Copy-on-Write vs MoR

  Run Iceberg MERGE INTO, UPDATE, and DELETE in Spark with Iceberg 1.12.0, pick copy-on-write or merge-on-read per operation, and evolve schemas without rewrites.

  apache sparkapache icebergdata engineering
• 19 MIN READ•May 31, 2026

  Data Platform AI Agent Tooling in 2026: 11 Vendors Compared

  How Dremio, Snowflake, Databricks, Fabric, AWS, Google Cloud, ClickHouse, Doris, Spice, Bauplan, and Qlik expose data to AI agents: MCP servers, agents, CLIs.

  AI AgentsData PlatformsMCP
• 32 MIN READ•Jul 6, 2026

  Agentic AI Standards in 2026: MCP vs A2A and the Full Stack

  MCP connects agents to tools; A2A connects agents to each other. A 2026 map of agentic AI standards: MCP, A2A, WebMCP, Apache Ossie (OSI), and agent payments.

  Apache Icebergdata engineeringlakehouse architecture
• 39 MIN READ•Sep 21, 2026

  Fast Classification Models, LLMs, and the Apache Iceberg Lakehouse

  How fast classification models like Jev and GLiClass compare with LLMs on fixed-label tasks, and how to run both together over Apache Iceberg tables at scale.

  Apache IcebergAILLM
• 31 MIN READ•Sep 10, 2026

  Migrating Into Iceberg Without Moving Data

  add_files, snapshot, and migrate compared: the three in-place paths into Iceberg, the reconciliation each requires, the layout traps, and the rollback story.

  Apache Icebergmigrationmetadata
• 31 MIN READ•Sep 10, 2026

  What Iceberg Table Maintenance Actually Costs

  A cost model for compaction, snapshot expiry, orphan cleanup, and manifest rewriting: what each operation spends, on which meter, and how to set a schedule.

  Apache Icebergtable maintenancecompaction
• 24 MIN READ•Sep 2, 2026

  Will AI Replace Data Engineers?

  What the evidence shows about whether AI replaces data engineers, which parts of the job compress, and which parts do not.

  AIData EngineeringCareers
• 31 MIN READ•Aug 24, 2026

  Iceberg Is Becoming a Library, Not Just a Table Format

  Iceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself.

  Apache Iceberglibrariesecosystem
• 31 MIN READ•Aug 19, 2026

  Zero-Copy Warehouse Modernization: Moving to Apache Iceberg Without Downtime

  A four-phase plan for moving a warehouse to Apache Iceberg without downtime: federate first, redirect new data, materialize selectively, and cut over gradually.

  Apache Icebergwarehouse migrationzero-copy
• 31 MIN READ•Aug 2, 2026

  Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy

  Apache Polaris 1.7.0 deep dive: idempotent writes, semantic models, stricter credential vending, orphan cleanup, and what the upgrade asks of you.

  Apache PolarisApache IcebergCatalog
• 31 MIN READ•Aug 2, 2026

  Designing Batch Pipelines That Write Well Into Apache Iceberg

  How to design batch pipelines that write well into Apache Iceberg: commit strategy, partitioning, sort order, write-audit-publish, and maintenance done right.

  Apache IcebergData EngineeringBatch Pipelines