I think what you are getting at is a "staged execution model", and I agree.
GNU make actually has this, but it's done poorly. It has build STEPS in the shell language, but the build GRAPH can be done in the Make language, or even Guile scheme. [1]
So the build GRAPH should be described as you say. It's declarative, but you need metaprogramming. You can think of it like generating a Ninja file, but using reflection/metaprogramming rather than textual code generation.
And then the build STEPS are literally shell. Shell is a lot better than Python for this use case! e.g. for invoking cmopilers and other tools.
[1] aside: Tensorflow has the same staged execution model. The (serial) Python language is used for metaprogramming the graph, while the the highly parallel graph language is called "Tensorflow".
GNU make actually has this, but it's done poorly. It has build STEPS in the shell language, but the build GRAPH can be done in the Make language, or even Guile scheme. [1]
----
I hope to add the "missing declarative part" to shell with https://www.oilshell.org.
So the build GRAPH should be described as you say. It's declarative, but you need metaprogramming. You can think of it like generating a Ninja file, but using reflection/metaprogramming rather than textual code generation.
And then the build STEPS are literally shell. Shell is a lot better than Python for this use case! e.g. for invoking cmopilers and other tools.
I hinted at this a bit in a previous thread: https://news.ycombinator.com/item?id=25343716
And this current comment https://lobste.rs/s/k0qhfw/modern_ci_is_too_complex_misdirec...
Comments welcome!
[1] aside: Tensorflow has the same staged execution model. The (serial) Python language is used for metaprogramming the graph, while the the highly parallel graph language is called "Tensorflow".