A few years ago a new guy at our DWH team tried to sell Airflow to the rest of the team. They invited me to listen to his talk as well, and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure.
Why would I need a glorified server-side crontab if something like MS DTS from 1998 could do the same, but better? Sure, Python is probably better than whatever DTS generated, but the ops don't care either way, since Airflow doesn't care what it's running.
Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem.
"What's the overall trend for job D's finish time, what is the main reason for that?" isn't Airflow's problem, it's your problem. "What jobs are on the critical path for job E?" isn't Airflow's problem, it's your problem.
"Job F failed for date T and then recursively restart everything that uses its results for date T" isn't Airflow's problem, it's your problem.
>Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem.
I guess that's one approach to job security. And why not make data egress manual too? Why transfer data through the network, when you can print them, mail the papers, and type them back in? Data input is not the computer's problem, it's your problem!
>Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem. "What's the overall trend for job D's finish time, what is the main reason for that?" isn't Airflow's problem, it's your problem. "What jobs are on the critical path for job E?" isn't Airflow's problem, it's your problem. "Job F failed for date T and then recursively restart everything that uses its results for date T" isn't Airflow's problem, it's your problem.
The whole idea of writing programs is making things automatable. That is, making them the computer's problem, not our problem. We get the higher level problem of writing the automation once, and fixing any bugs in our code, then we get to enjoy putting it to work for us...
I think you've misunderstood my point. I don't want all these problems to be mine, I want whatever job orchestrator I choose to solve them. Airflow very explicitly doesn't try to solve them, doesn't even try to solve them badly, it just runs jobs when scheduled.
Well, Airflow is not that great, but does try to solve these problems: it has retries, it shows "jobs are on the critical path" for another job, and how they're doing, and so on...
Why would I need a glorified server-side crontab if something like MS DTS from 1998 could do the same, but better? Sure, Python is probably better than whatever DTS generated, but the ops don't care either way, since Airflow doesn't care what it's running.
Something as simple as "job A must run after job B and job C, but if it doesn't start by 2am, wake up team X. If it doesn't finish by 4am, wake up team Y" isn't Airflow's problem, it's your problem.
"What's the overall trend for job D's finish time, what is the main reason for that?" isn't Airflow's problem, it's your problem. "What jobs are on the critical path for job E?" isn't Airflow's problem, it's your problem.
"Job F failed for date T and then recursively restart everything that uses its results for date T" isn't Airflow's problem, it's your problem.