Tag:data engineering
Articles tagged "data engineering", page 9.
- 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Storage Formats and Compression
How CSV, JSON, Avro, ORC, and Parquet differ, why columnar formats suit analytics, and how compression codecs affect file size and query speed.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Data Quality and Validation
The dimensions of data quality, where quality breaks down in pipelines, how to validate data at each stage, and tools that catch problems before users do.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance
What metadata is, how data lineage helps with debugging and compliance, what data governance covers, and practices that build trust in data at scale.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration
How workflow orchestration goes beyond scheduling, what a DAG is, tools like Apache Airflow, and practices for retries, dependencies, and alerting.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Building Scalable Pipelines
What makes a data pipeline scalable: parallel and distributed processing, less data movement, managing compute resources, and designing for change.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | DevOps for Data Engineering
How DevOps applies to data engineering: version control for pipelines, CI and automated tests, infrastructure as code, deployment, and incident response.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Cloud Data Platforms and the Modern Stack
What the modern data stack is, how cloud platforms changed data engineering, the managed services they offer, and how to design data systems for agility.
Data EngineeringDatabasesData Lakes - 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Data Lakehouse Architecture Explained
Why running a separate data lake and data warehouse causes problems, what a data lakehouse is, the open technologies behind it, and its main benefits.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Apache Iceberg, Arrow, and Polaris
How Apache Iceberg tables, the Apache Arrow memory format, and the Apache Polaris catalog each solve a lakehouse problem and work together across engines.
Data EngineeringDatabasesData Lakes - 6 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | The Power of Dremio in the Modern Lakehouse
How Dremio fits a modern lakehouse: Iceberg clustering for data layout, Autonomous Reflections for speed, Polaris-based governance, and AI-ready data.
Data EngineeringDatabasesData Lakes - 16 MIN READ•Nov 15, 2024
Deep Dive into Dremio's File-based Auto Ingestion into Apache Iceberg Tables
How to load JSON, CSV, and Parquet files into Iceberg tables with Dremio Auto-Ingest pipes: syntax, file format options, deduplication, and error handling.
data lakehousedata engineeringapache iceberg - 8 MIN READ•Nov 5, 2024
Dremio, Apache Iceberg and their role in AI-Ready Data
What makes data AI-ready, how Iceberg features like schema evolution and time travel support it, and how Dremio speeds data prep for AI workloads.
data lakehousedata engineeringapache iceberg