Tag:data engineering
Articles tagged "data engineering", page 10.
- 6 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | The Power of Dremio in the Modern Lakehouse
How Dremio fits a modern lakehouse: Iceberg clustering for data layout, Autonomous Reflections for speed, Polaris-based governance, and AI-ready data.
Data EngineeringDatabasesData Lakes - 16 MIN READ•Nov 15, 2024
Deep Dive into Dremio's File-based Auto Ingestion into Apache Iceberg Tables
How to load JSON, CSV, and Parquet files into Iceberg tables with Dremio Auto-Ingest pipes: syntax, file format options, deduplication, and error handling.
data lakehousedata engineeringapache iceberg - 8 MIN READ•Nov 5, 2024
Dremio, Apache Iceberg and their role in AI-Ready Data
What makes data AI-ready, how Iceberg features like schema evolution and time travel support it, and how Dremio speeds data prep for AI workloads.
data lakehousedata engineeringapache iceberg - 26 MIN READ•Oct 31, 2024
Hands-on with Apache Iceberg & Dremio on Your Laptop within 10 Minutes
Run Dremio, Nessie, and MinIO with Docker Compose in about 10 minutes, create Iceberg tables, run transactions, and inspect the files and metadata.
data lakehousedata engineeringapache iceberg - 8 MIN READ•Oct 30, 2024
Data Modeling - Entities and Events
The difference between modeling data as entities and as events, when each approach fits, key design choices, and how to combine both for full analysis.
data lakehousedata engineeringdata modeling - 4 MIN READ•Oct 21, 2024
All About Parquet Part 01 - An Introduction
An introduction to Apache Parquet: how it differs from row formats like CSV and JSON, key features such as compression and schema evolution, and when to use it.
data lakehousedata engineeringapache parquet - 5 MIN READ•Oct 21, 2024
All About Parquet Part 02 - Parquet's Columnar Storage Model
How Parquet's columnar storage model works and why it speeds analytics through faster scans, better compression, and quick aggregation, plus when to avoid it.
data lakehousedata engineeringapache parquet - 6 MIN READ•Oct 21, 2024
All About Parquet Part 03 - Parquet File Structure | Pages, Row Groups, and Columns
Inside a Parquet file: how row groups, column chunks, and pages are laid out, what footer metadata stores, and how their sizes affect query performance.
data lakehousedata engineeringapache parquet - 6 MIN READ•Oct 21, 2024
All About Parquet Part 04 - Schema Evolution in Parquet
How Parquet handles schema evolution when you add, remove, retype, rename, or reorder columns, how it compares to other formats, and best practices.
data lakehousedata engineeringapache parquet - 6 MIN READ•Oct 21, 2024
All About Parquet Part 05 - Compression Techniques in Parquet
The compression codecs Parquet supports (Snappy, Gzip, Brotli, Zstandard, LZO), the speed and size tradeoffs of each, and how they pair with encoding.
data lakehousedata engineeringapache parquet - 7 MIN READ•Oct 21, 2024
All About Parquet Part 06 - Encoding in Parquet | Optimizing for Storage
How Parquet encodings work and when to use them: dictionary, run-length, bit-packing, delta, and plain, and how encoding pairs with compression.
data lakehousedata engineeringapache parquet - 7 MIN READ•Oct 21, 2024
All About Parquet Part 09 - Parquet in Data Lake Architectures
Why Parquet is the default file format for data lakes, how it underpins Iceberg, Delta Lake, and Hudi, and best practices for partitioning and pushdown.
data lakehousedata engineeringapache parquet