When your team starts relying on data for daily operations and decision-making, trust becomes the foundation. But what does it mean to “trust the data”? And how do you build a system where data is accurate, timely, and easy to understand?
This article summarizes the ideas I shared in a recent internal tech talk, designed for both engineers and non-engineers alike.
In recent years, the field of data engineering has undergone significant transformations. Tools like dbt (data build tool) have emerged as vital components of modern data engineering workflows. These technologies not only optimize how data teams operate but also enable collaboration across diverse roles, including data engineers, analysts, project managers, and stakeholders. This article, based on my experience and a recent talk, explores how data engineering has evolved, why dbt has gained traction, and how it addresses pain points in data workflows.
In my role at a traditional company undergoing digital transformation, I faced a common challenge: managing multiple pipelines across Windows and Linux. These pipelines, essential for generating daily dashboard reports, were difficult to monitor, debug, and scale.
Big data means working with very large amounts of information. In one of my jobs, I had to handle 500TB of data and run more than 10,000 SQL queries every day. The old system we used was slow and had many problems, like some tasks taking over 24 hours to finish. In this blog, I will share how I solved these problems by using Spark and making the system faster and better.
嗨，大家好！歡迎來到我的第一篇部落格文章！🎉
我非常興奮能通過這個平台開始與大家分享我的想法和經驗。作為一個熱愛科技和程式設計的人，我覺得是時候在網路上為自己劃出一個小天地了。