Tag: Data Engineering
All the articles with the tag "Data Engineering".
- 27 MIN READ•May 22, 2026
Apache Iceberg SCD Type 2 and CDC Patterns: Building Historical Lakehouse Tables
A deep dive into implementing Slowly Changing Dimension Type 2 (SCD Type 2) patterns and Change Data Capture (CDC) pipelines on Apache Iceberg, using PySpark and Dremio.
apache icebergcdcscd type 2 - 24 MIN READ•May 22, 2026
Apache Iceberg Catalogs Explained: REST, Glue, Hive Metastore, Polaris, Nessie, and Snowflake
A deep dive into Apache Iceberg catalog architecture, comparing REST catalogs, AWS Glue, Project Nessie, Polaris, and Snowflake. Learn catalog role, credential vending, and cross-engine configurations.
apache icebergcatalogsNessie - 24 MIN READ•May 22, 2026
Maintaining Apache Iceberg Tables: Compaction, Snapshot Expiration, and Orphan File Cleanup
An in-depth guide to orchestrating maintenance operations on Apache Iceberg tables, covering bin-packing, sort-based, Z-Order compaction, snapshot expiration, and orphan file removal, with query acceleration details for the Dremio engine.
Apache IcebergCompactionData Engineering - 24 MIN READ•May 22, 2026
Apache Iceberg with Spark: Create, MERGE, Upsert, and Evolve Tables End to End
A comprehensive developer guide to configuring Apache Spark with Apache Iceberg, executing transactional writes, and managing schema evolution.
apache sparkapache icebergdata engineering - 21 MIN READ•May 22, 2026
Common Misconceptions About Data Lakehouse and Apache Iceberg
Addressing common search queries and reader confusion about Data Lakehouse architectures, Apache Iceberg catalogs, partitions, and lock-in.
data lakehouseapache icebergdata engineering - 5 MIN READ•Feb 18, 2026
Data Engineering Best Practices: The Complete Checklist
Best practices documents are easy to write and hard to use. They list principles without context, advice without prioritization, and rules without explaining...
data engineeringbest practiceschecklist - 5 MIN READ•Feb 18, 2026
Data Modeling Best Practices: 7 Mistakes to Avoid
A bad data model doesn't announce itself. It hides behind slow dashboards, conflicting numbers, confused analysts, and AI agents that generate wrong SQL. By ...
data modelingdata engineeringbest practices - 7 MIN READ•Feb 18, 2026
Semantic Layer Best Practices: 7 Mistakes to Avoid
Semantic layers don't fail because the technology is wrong. They fail because of design decisions made in the first two weeks : choices that seem reasonable ...
semantic layerdata engineeringbest practices - 5 MIN READ•Feb 18, 2026
Pipeline Observability: Know When Things Break
An analyst messages you on Slack – "The revenue numbers look wrong. Is the pipeline broken?" You check the orchestrator – all green. You check the target tabl...
data engineeringbest practicesobservability - 5 MIN READ•Feb 18, 2026
Data Vault Modeling: Hubs, Links, and Satellites
Dimensional modeling works well when your source systems are stable and your business questions are predictable. But what happens when sources change constan...
data modelingdata engineeringdata vault - 5 MIN READ•Feb 18, 2026
How a Self-Documenting Semantic Layer Reduces Data Team Toil
Every data team knows documentation is important. And almost every data team has a backlog of undocumented tables, unlabeled columns, and outdated descriptio...
semantic layerdata engineeringdocumentation - 5 MIN READ•Feb 18, 2026
Testing Data Pipelines: What to Validate and When
Ask an application developer how they test their code and they'll describe unit tests, integration tests, CI/CD pipelines, and coverage metrics. Ask a data e...
data engineeringbest practicestesting