Tag:data engineering
Articles tagged "data engineering", page 11.
- 6 MIN READ•Oct 18, 2024
A Guide to dbt Macros - Purpose, Benefits, and Usage
What dbt macros are and how they cut repeated SQL: defining them, calling them in models, generating SQL with variables, and keeping them maintainable.
data lakehousedata engineeringdbt - 6 MIN READ•Oct 16, 2024
Data Lakehouse Roundup 1 - News and Insights on the Lakehouse
What's Going on in the Data Lakehouse Space
data lakehousedata engineeringstreaming - 17 MIN READ•Oct 15, 2024
Getting Started with Data Analytics Using PyArrow in Python
A practical PyArrow guide: core objects like Tables and RecordBatches, reading and writing Parquet, running computations, and using Arrow Flight.
data lakehousedata engineeringPython - 14 MIN READ•Oct 9, 2024
What is Three-Tier Data (Bronze, Silver, Gold) and How Dremio Simplifies It
What the bronze, silver, and gold data tiers are, the copy and sync costs of moving data between them, and how Dremio views and Reflections cut that work.
data lakehousedata engineering - 7 MIN READ•Oct 7, 2024
A Brief Guide to the Governance of Apache Iceberg Tables
How access control for Apache Iceberg tables works at the file, engine, and catalog level, and how Project Nessie and Apache Polaris governance compare.
data lakehousedata engineeringapache iceberg - 17 MIN READ•Oct 7, 2024
Exploring Data Operations with PySpark, Pandas, DuckDB, Polars, and DataFusion in a Python Notebook
Hands-on examples of loading and querying data with PySpark, Pandas, DuckDB, Polars, and DataFusion in one Docker notebook, plus querying Dremio.
data lakehousedata engineeringPython - 8 MIN READ•Oct 5, 2024
Ultimate Directory of Apache Iceberg Resources
A directory of Apache Iceberg learning material by topic: architecture, ingestion, migration, streaming, partitioning, maintenance, and catalogs.
data lakehousedata engineeringapache iceberg - 14 MIN READ•Oct 4, 2024
Change Data Capture (CDC) when there is no CDC
How to sync changing data when a source has no native CDC: timestamp and version columns, SQL to find changes, MERGE, and scheduled incremental batches.
data lakehousedata engineeringstreaming - 4 MIN READ•Sep 25, 2024
Virtualization + Lakehouse + Mesh = Data At Scale
How virtualization, a lakehouse, and data mesh ideas combine into one point of access with data curated by many domain teams, and where Dremio fits.
data lakehousedata engineering - 12 MIN READ•Sep 10, 2024
Why Data Analysts, Engineers, Architects and Scientists Should Care about Dremio and Apache Iceberg
How Dremio and Apache Iceberg address the daily problems of data engineers, architects, analysts, and data scientists, from data silos to governance.
data lakehousedata engineering - 43 MIN READ•Sep 10, 2024
Hands-on with Apache Iceberg on Your Laptop - Deep Dive with Apache Spark, Nessie, Minio, Dremio, Polars and Seaborn
Build an Iceberg lakehouse on your laptop: ingest with Spark into MinIO, track tables in Nessie, query in Dremio, then analyze with Polars and Seaborn.
data lakehousedata engineering - 7 MIN READ•Sep 1, 2024
5 Trends in the Data Lakehouse Space
Five lakehouse trends: storage vendors adding warehouse features, table format convergence, managed Iceberg upkeep, catalog wars, and unified platforms.
data lakehousedata engineering