Tag:data engineering
Articles tagged "data engineering", page 8.
- 3 MIN READ•Sep 16, 2025
The Endgame – Building an Autonomous Optimization Pipeline for Apache Iceberg
Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools.
Apache IcebergAutomationCompaction - 3 MIN READ•Sep 9, 2025
Managing Large-Scale Optimizations – Parallelism, Checkpointing, and Fail Recovery
Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability.
Apache IcebergOptimizationCompaction - 9 MIN READ•Sep 5, 2025
Unlocking the Power of Agentic AI with Apache Iceberg and Dremio
The three bottlenecks that stall agentic AI (data access, query speed, and semantic meaning) and how Apache Iceberg and Dremio address each one.
Data LakehouseData EngineeringAgentic AI - 4 MIN READ•Sep 2, 2025
Hidden Pitfalls – Compaction and Partition Evolution in Apache Iceberg
Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction.
Apache IcebergPartition EvolutionTable Optimization - 4 MIN READ•Aug 19, 2025
Designing the Ideal Cadence for Compaction and Snapshot Expiration
Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness.
Apache IcebergTable OptimizationCompaction - 6 MIN READ•Jul 3, 2025
How to Discover or Organize Lakehouse & Apache Iceberg Meetups
Guide on How to Be Part of the Lakehouse Community
Data LakehouseData Engineering - 10 MIN READ•Jun 23, 2025
What is an API? And Why Data Architecture Depends on Them
What an API is, how SOAP, REST, RPC, and GraphQL differ, and why data architecture runs on APIs like S3, JDBC, ADBC, Hive Metastore, Glue, and Iceberg REST.
Data LakehouseData Engineering - 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | What is Data Engineering?
What data engineers do, how raw data moves through a pipeline to become usable, how the role differs from data science, and how the data stack evolved.
Data EngineeringDatabasesData Lakes - 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion
Common data sources, batch versus streaming ingestion, structured, semi-structured, and unstructured data, and how to design reliable ingestion pipelines.
Data EngineeringDatabasesData Lakes - 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines
The difference between ETL and ELT pipelines, why ELT took off with cloud warehouses, and when transforming data before loading it is still the better choice.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Batch Processing Fundamentals
What batch processing is, how batch jobs run under the hood, the tools that build and schedule them, their limits, and where batch still beats streaming.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Streaming Data Fundamentals
What streaming data is, how producers, message brokers, and stream processors work together, real-time use cases, and the challenges of streaming pipelines.
Data EngineeringDatabasesData Lakes