Except that it would still be delivered as an opaque binary blob. The GP's suggestion isn't much different from Intel's proposal for abstracting drivers into EFI Runtime Services, and many people see those as a security risk (in a "you don't control your hardware" sense, not necessarily in a "it has bugs" sense).

Outside CPU, you have microprosessors inside the chipsets (like Intel ME), network cards, mass memory devices, etc. They all have binary blobs. Better that they don't have bugs.

One thing that seL4 requires is an mmu in order to support the isolation claims. The intel me type cpu would need to have some of these features that micorcontrollers don't usually have.