The way I would categorize build systems (and by extension, a lot of CI systems) is semi-declarative. That is to say, we can describe the steps needed to build as a declarative list of source files, the binaries they end up in, along with some special overrides (maybe this one file needs special compiler flags) and custom actions (including the need to generate files). To some degree, it's recursive: we need to build the tool to build the generated files we need to compile for the project. In essence, the build system boils down to computing some superset of Clang's compilation database format. However, the steps needed to produce this declarative list are effectively a Turing-complete combination of the machine's environment, user's requested configuration, package maintainers' whims, current position of Jupiter and Saturn in the sky, etc.
Now what makes this incredibly complex is that the configuration step itself is semi-declarative. I may be able to reduce the configuration to "I need these dependencies", but the list of dependencies may be platform-dependent (again with recursion!). Given that configuration is intertwined with the build system, it makes some amount of sense to combine the two concepts into one system, but they are two distinct steps and separating those steps is probably saner.
To me, it makes the most sense to have the core of the build system be an existing scripting language in a pure environment that computes the build database: the only accessible input is the result of the configuration step, no ability to run other programs or read files during this process, but the full control flow of the scripting language is available (Mozilla's take uses Python, which isn't a bad choice here). Instead, the arbitrary shell execution is shoved into the actual build actions and the configuration process (but don't actually use shell scripts here, just equivalent in power to shell scripts). Also, make the computed build database accessible both for other tools (like compilation-database.json is) and for build actions to use in their implementations.
I think what you are getting at is a "staged execution model", and I agree.
GNU make actually has this, but it's done poorly. It has build STEPS in the shell language, but the build GRAPH can be done in the Make language, or even Guile scheme. [1]
So the build GRAPH should be described as you say. It's declarative, but you need metaprogramming. You can think of it like generating a Ninja file, but using reflection/metaprogramming rather than textual code generation.
And then the build STEPS are literally shell. Shell is a lot better than Python for this use case! e.g. for invoking cmopilers and other tools.
[1] aside: Tensorflow has the same staged execution model. The (serial) Python language is used for metaprogramming the graph, while the the highly parallel graph language is called "Tensorflow".
Now what makes this incredibly complex is that the configuration step itself is semi-declarative. I may be able to reduce the configuration to "I need these dependencies", but the list of dependencies may be platform-dependent (again with recursion!). Given that configuration is intertwined with the build system, it makes some amount of sense to combine the two concepts into one system, but they are two distinct steps and separating those steps is probably saner.
To me, it makes the most sense to have the core of the build system be an existing scripting language in a pure environment that computes the build database: the only accessible input is the result of the configuration step, no ability to run other programs or read files during this process, but the full control flow of the scripting language is available (Mozilla's take uses Python, which isn't a bad choice here). Instead, the arbitrary shell execution is shoved into the actual build actions and the configuration process (but don't actually use shell scripts here, just equivalent in power to shell scripts). Also, make the computed build database accessible both for other tools (like compilation-database.json is) and for build actions to use in their implementations.