I've had no problem using consumer Nvidia GPUs with QEMU/KVM for this. All I needed to do was (1) use kvm=off (this does not turn off KVM, it just sort of hides it from the NVidia driver) and (2) on Windows, edit the registry to force enable Message Signaled Interrupts. When I tried it with an AMD card it would periodically crash the host when it was assigned to a Linux guest.

It's "very difficult" because the drivers are intentionally nerfed not to load in a virtual environment. Your statement is entirely correct.

One line config change to hide the hypervisor from the VM and it works. (Source: worked for me)