Most people only change one or two files at a time. Those files, and the few that depend on them, must be rebuilt upon a change. Incremental builds only do what's necessary. Your proposal always builds everything no matter how trivial the change. It does not scale.
Did I ever say it was the fastest way to build a program? No, I said it was the simplest. But you might be surprised at just how fast it can be; it can scale to programs of significant complexity. Incremental compilation made a lot more sense back when computers were a lot slower.
With large complex programs using complex incremental build systems, especially programs written in C++, you often end up spending most of your time processing the same header files over and over again. Some compilers use complex systems of pre–compiled headers that can paper over this cost, but building your whole program in a single compilation unit that only ever includes each file once eliminates it entirely without adding any additional complexity. If you want proof of this, go look at the efforts over the last year or two to shave a few minutes off the build time of the Linux kernel by carefully adjusting all of the header files, splitting them all into even smaller files so that each #include can bring in just what is needed. This avoids burdening the compiler with parsing things you aren’t going to use just to pull in a struct definition or two.
Plus, by having a single compilation unit you avoid the need to link a bunch of small object files together. Linking is the final step of building an executable, and it cannot generally be parallelized much. By reducing the amount of work the linker needs to do you can greatly speed up a step that can’t be sped up any other way.
But the real reason I recommend it is its simplicity. By spending less programmer time on the build system you can significantly reduce the overall development cost of the whole program. This is a huge benefit of Rust (and a lot of other modern programming systems) that is often overlooked. With a complex C++ program I might spend 10% of my overall development time monkeying around with the build system. If I write it in Rust I can avoid all of that; cargo can do everything I need while requiring only a very small amount of time configuring it. 10% of a project that takes a year is over a month, and with cargo that time expenditure is reduced to minutes or hours.
On the other hand, I have worked as a contractor for many years helping companies develop complex systems. It’s only a rough estimate, but I believe that I have earned over a hundred thousand dollars just by helping them with their complex build systems. A few days here, a few days there, it really adds up.