That's why high assurance systems (like L4/seL4) minimize privileged code to a tiny fraction of what you find in a regular system, which absolutely makes sense and provably improves security.
Apart from the obvious advantages, minimal privileged code also means that it becomes far more feasible to formally verify the privileged code, like it has been done for seL4, which in turn is an important cornerstone to ensure that isolation (or more generally, specification invariants) does not break in any reachable system state.
Something like Rowhammer can turn perfectly written code into malicioous gadgets for an attacker. Until ECC or preventative measures becomes widespread in consumer hardware, this attack will remain viable.
Additionally, cosmic rays may at any time write arbitrary data into memory. In these situations it is perfectly reasonable to crash the machine if essential datastructures were affected.
Or suppose you just received a MACHINE CHECK exception, the type of interrupt which essentially just tells the OS that the machine is no longer capable of operating in it's current state.
This isn't all about an attacker gaining privilege, the mere act of crashing the kernel in a hostile or other event (cosmic rays, voltage fluctuation) which leads to corruption or compromise of the continued operation of the machine is perfectly acceptable and normal.
Apart from the obvious advantages, minimal privileged code also means that it becomes far more feasible to formally verify the privileged code, like it has been done for seL4, which in turn is an important cornerstone to ensure that isolation (or more generally, specification invariants) does not break in any reachable system state.