Tag:data engineering
Articles tagged "data engineering".
- 20 MIN READ•Nov 16, 2026
The Iceberg and Parquet Spec Ambiguity Ledger, November 2026: Name Mapping, Nanosecond Buckets, FILE Inline vs URI and distinct_count
Four Iceberg and Parquet spec ambiguities from November 2026, what each implementation does, the proposed fix, and how to test your own readers.
Apache IcebergApache ParquetSpecification - 21 MIN READ•Nov 16, 2026
Iceberg v3 as the Default Format Version: An Implementation Readiness Report, November 2026
An implementation readiness matrix for making Iceberg v3 the default format version, covering Java, Go, C++, PyIceberg and Rust as of November 2026.
Apache IcebergFormat VersionDeletion Vectors - 30 MIN READ•Oct 27, 2026
Per-Developer Sandboxes on Apache Iceberg Branches: Lifecycle, Isolation Limits, and Cleanup
Give every developer and pull request a zero-copy Iceberg sandbox on branches: lifecycle, what branches do not isolate, and cleanup.
Apache Icebergbranchesdata engineering - 9 MIN READ•Oct 21, 2024
Parquet Metadata Explained: Footer, Row Groups, Statistics
Parquet metadata lives in the file footer: the schema, row group layout and per-column min/max statistics that let query engines skip data they do not need.
data lakehousedata engineeringapache parquet - 7 MIN READ•Oct 21, 2024
Read and Write Parquet Files in Python (PyArrow, Pandas)
How to read and write Parquet files in Python with PyArrow, FastParquet, and Pandas, including column selection, compression, and partitioned datasets.
data lakehousedata engineeringapache parquet - 9 MIN READ•Oct 21, 2024
Parquet Performance Tuning: 8 Best Practices
Tune Apache Parquet for speed and cost: row group and file sizes, partitioning, Snappy vs ZSTD compression, encodings, sorting, and predicate pushdown.
data lakehousedata engineeringapache parquet - 43 MIN READ•Sep 23, 2025
Data Lakehouse Guide 2026: Architecture and Ecosystem
What a data lakehouse is and how it works: Iceberg, Delta, Hudi, and Paimon table formats, the five core layers, catalogs, ingestion, and optimization.
Data LakehouseData EngineeringAgentic AI - 23 MIN READ•Sep 24, 2025
Open Table Formats Compared: Iceberg, Delta, Hudi, Paimon
Apache Iceberg, Delta Lake, Hudi, Paimon and DuckLake compared: metadata design, row-level updates, engine support, and when to choose each table format.
Data LakehouseData EngineeringAgentic AI - 5 MIN READ•Feb 18, 2026
Idempotent Data Pipelines: Patterns, SQL, and Pitfalls
An idempotent pipeline gives the same result no matter how often it runs. Use partition overwrite, MERGE upserts, and dedup to make retries and backfills safe.
data engineeringbest practicesidempotency - 6 MIN READ•Feb 18, 2026
What Is Data Modeling? Definition, Types, and Examples
Data modeling defines the entities, attributes, and relationships in your data. Learn the conceptual, logical, and physical levels and the main techniques.
data modelingdata engineeringdata lakehouse - 6 MIN READ•Feb 18, 2026
Conceptual vs Logical vs Physical Data Models (Examples)
Conceptual, logical, and physical data models explained with one e-commerce example: what each level contains, who owns it, and how one feeds the next.
data modelingdata engineeringconceptual model - 6 MIN READ•Feb 18, 2026
Star Schema vs Snowflake Schema: Differences and When to Use
A star schema keeps each dimension in one flat table; a snowflake schema splits dimensions into normalized sub-tables. Compare joins, speed, and use cases.
data modelingdata engineeringstar schema