Tag:Data Lakes
Articles tagged "Data Lakes".
- 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | What is Data Engineering?
What data engineers do, how raw data moves through a pipeline to become usable, how the role differs from data science, and how the data stack evolved.
Data EngineeringDatabasesData Lakes - 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion
Common data sources, batch versus streaming ingestion, structured, semi-structured, and unstructured data, and how to design reliable ingestion pipelines.
Data EngineeringDatabasesData Lakes - 4 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines
The difference between ETL and ELT pipelines, why ELT took off with cloud warehouses, and when transforming data before loading it is still the better choice.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Batch Processing Fundamentals
What batch processing is, how batch jobs run under the hood, the tools that build and schedule them, their limits, and where batch still beats streaming.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Streaming Data Fundamentals
What streaming data is, how producers, message brokers, and stream processors work together, real-time use cases, and the challenges of streaming pipelines.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Data Modeling Basics
Data modeling basics: OLTP versus OLAP systems, normalization and denormalization, star and snowflake schemas, and how to design models that can grow.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Data Warehousing Fundamentals
What a data warehouse is and how it differs from an operational database: architecture, columnar storage, partitioning and clustering, and load patterns.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Data Lakes Explained
What a data lake is, how schema-on-read works, how lakes differ from warehouses, and how raw, cleaned, and curated zones keep a lake from becoming a swamp.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Storage Formats and Compression
How CSV, JSON, Avro, ORC, and Parquet differ, why columnar formats suit analytics, and how compression codecs affect file size and query speed.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Data Quality and Validation
The dimensions of data quality, where quality breaks down in pipelines, how to validate data at each stage, and tools that catch problems before users do.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance
What metadata is, how data lineage helps with debugging and compliance, what data governance covers, and practices that build trust in data at scale.
Data EngineeringDatabasesData Lakes - 5 MIN READ•May 2, 2025
Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration
How workflow orchestration goes beyond scheduling, what a DAG is, tools like Apache Airflow, and practices for retries, dependencies, and alerting.
Data EngineeringDatabasesData Lakes