The Arm ecosystem is a shitshow for this reason, with massive efforts required to port to each new SBC. We hope to standardize RISC-V enough to avoid at least the worst of this (Linux drivers will still be a problem, but you should be able to boot a standard single image on most boards even if some of the hardware won't always work at the start).
So this sounds like a deliberate design choice. Like when Intel went from pins to balls shifting the complexity to the motherboard. There are always trade-offs. I guess the idea is that the overhead is worth the benefit to the ecosystem and if the vendor really needs to they can create a chip that is only conformant at a user level but not at a priveledged level and still better than starting a custom ISA from scratch.
