2 posts tagged with "Airflow"

In recent years, the field of data engineering has undergone significant transformations. Tools like dbt (data build tool) have emerged as vital components of modern data engineering workflows. These technologies not only optimize how data teams operate but also enable collaboration across diverse roles, including data engineers, analysts, project managers, and stakeholders. This article, based on my experience and a recent talk, explores how data engineering has evolved, why dbt has gained traction, and how it addresses pain points in data workflows.

1. Introduction​

In my role at a traditional company undergoing digital transformation, I faced a common challenge: managing multiple pipelines across Windows and Linux. These pipelines, essential for generating daily dashboard reports, were difficult to monitor, debug, and scale.