Fascinating article and to the heart of my being as a DE engineer. I had the same conclusion a while back and wrote about the trends in data orchestration towards a declarative pipeline, which more modern tools, as you mentioned, will do, but I'm afraid not Airflow.
I'd say data consumers, such as data analysts and business users, care primarily about the production of data assets. On the other hand, data engineers with Airflow focus on modeling the dependencies between tasks (instead of data assets). How how can we reconcile both worlds?
In my latest article, I review Airflow, Prefect, and Dagster and discuss how data orchestration tools introduce data assets as first-class objects. I also cover why a declarative approach with higher-level abstractions helps with faster developer cycles, stability, and a better understanding of what’s going on pre-runtime. I explore five different abstractions (jobs, tasks, resources, triggers, and data products) and see if it all helps to build a Data Mesh. If that sounds interesting, make sure to check out https://airbyte.com/blog/data-orchestration-trends.
I'd say data consumers, such as data analysts and business users, care primarily about the production of data assets. On the other hand, data engineers with Airflow focus on modeling the dependencies between tasks (instead of data assets). How how can we reconcile both worlds?
In my latest article, I review Airflow, Prefect, and Dagster and discuss how data orchestration tools introduce data assets as first-class objects. I also cover why a declarative approach with higher-level abstractions helps with faster developer cycles, stability, and a better understanding of what’s going on pre-runtime. I explore five different abstractions (jobs, tasks, resources, triggers, and data products) and see if it all helps to build a Data Mesh. If that sounds interesting, make sure to check out https://airbyte.com/blog/data-orchestration-trends.