Well. I try for a middle ground. I am currently ditching both airflow and dbt. In Snowflake, I use scheduled tasks that call stored procedures. The stored procedures do everything I need to do. I even call external APIs like Datadog’s and Okta’s and pull down the logs directly into snowflake. I do try to name my stored procedures with meaningful names. I also add generous comments including urls back to the original story.
I forgot to mention in Snowflake, besides chron scheduled tasks, you can add dependent tasks that only run if the previous task succeeded. I have 40 tasks chained together that way. Each of my task calls a stored procedure. Within each procedure, I have Try Catch and a catch-all clause that Raiseerror.
