This is a nice idea actually for finessing the GPU driver issue - use the virtual GPU drivers. (As a bonus you still have access to macOS desktop apps if you need or want them.)
Personally I use tend to use Linux VMs as if they were remote servers, so I don't have a huge need for Linux GPU drivers.
One thing that helped with VMware Fusion is using host-local interfaces with static IP addresses. Otherwise I would tend to get disconnected from VMs when macOS disconnected from wi-fi, which was very annoying. This is less of an issue if you're using the VM window directly.
Personally I use tend to use Linux VMs as if they were remote servers, so I don't have a huge need for Linux GPU drivers.
One thing that helped with VMware Fusion is using host-local interfaces with static IP addresses. Otherwise I would tend to get disconnected from VMs when macOS disconnected from wi-fi, which was very annoying. This is less of an issue if you're using the VM window directly.