I guess that means not usable day-to-day. One infinite loop in any app and the whole OS will freeze forever. I didn't even know any modern OS could operate only with cooperative multitasking. Windows 3.1 in 1992 and PowerPC Mac OS until 2002 were the last mainstream OS's to use it...
For decades no one used "nix" to mean Unix and Unix-like OSes (they used "*nix" and occasionally "Lunix") then a package manager named Nix became important and now people confusingly start using "nix" to mean Unix and Unix-like.
I can’t really blame you, it’s a perfectly reasonable concatenation imo. Pretty sure I’ve at least rarely seen it over the years as well.
That said, he is correct that now “nix” is going to likely be popularly associated (especially with tech types that love excessive things) with the Nix language, package manager, and NixOS, which has really been gaining steam. Especially these past few years.
It was what I first thought of as well, given I use them.
VirGL is a poor solution to the pressing problem of virtualized graphics. It only really exists because the hardware makers AMD/Intel/Nvidia in their infinite greed refuse to support VFIO on all GPU's like how IOMMU is supported on nearly all CPU's.
It does get a little tricky if your GPUs are identical, but I've done this for years and maintain a guide for doing this (as well as the ACS-override patched kernel RPMs) for Fedora.
As far as concerns around stability with ACS override, I tend to only enable the override for the specific GPU (or other hardware) that I'm passing through and haven't encountered any stability problems or memory leaks that'd interrupt desktop or light server usage. I also used to run this for a bunch of white-box GPU hardware for a customer at a former job and it worked well for exploratory AI/ML workloads before investing in the big Nvidia DGX boxes. YMMV, of course!
It's not difficult but it misses the point. SIOV supports 1k's of VF's because that's what you need if you want a sandboxed app-per-VM security model. When statically compiled VM's are just as performant as containers but more secure.
Correct me if I am wrong on this, but to me, it would seem that something like VirGL would still serve a purpose with wider spreader full SR-IOV support on consumer GPUs, as VirGL could find application in many scenarios where a GPU vendor's drivers are not compatible with the guest.
Saying that, I do agree that vendors should enable support in customer GPUs and feel that their focus on protecting server sales is going to turn out misguided in the long term. Intel especially disappointed in this area, as they in the past did allow such functionality on their GPUs, but have recently removed that.
AMDs mainstream CPUs supporting server features such as ECC also have proven that such restrictions aren't necessary, and allowing this type of capability on mainstream platforms in no way harms enterprise sales.
That being said, any effort focused on GPU virtualization or drivers impresses me immensly and I very much appreciate the work done on VirGL.
>Correct me if I am wrong on this, but to me, it would seem that something like VirGL would still serve a purpose with wider spreader full SR-IOV support on consumer GPUs, as VirGL could find application in many scenarios where a GPU vendor's drivers are not compatible with the guest.
Pretty much every guest OS (windows, Linux, BSD) has drivers that would work with a native PCIe VF GPU device. MacOS still has AMD drivers but only up to RDNA 2. I can't think of any guest that would support a GL device but not have a native driver.
>Saying that, I do agree that vendors should enable support in customer GPUs and feel that their focus on protecting server sales is going to turn out misguided in the long term. Intel especially disappointed in this area, as they in the past did allow such functionality on their GPUs, but have recently removed that.
Intel supports SRIOV/SIOV on consumer CPU iGPU's (Xe, 11th, 12th, and 13th gen) but not dGPU's (A770, A750..) which is very frustrating. Indeed 'enterprise features' such as ECC or IOMMU on consumer chips have not affected server sales.
>That being said, any effort focused on GPU virtualization or drivers impresses me immensly and I very much appreciate the work done on VirGL.
GVT-g was intel's first crack at virtualization and is now abandoned. It was supplanted by Intel supporting SR-IOV which itself was succeeded by intel's SIOV.
Yeah, I would love to have api layer pass through as well as pcie layer pass through. API pass through would work well for things like containers or sandbox environments.
AMD supports VFIO on most of their cards. All of the RDNA based cards support it. Even some pre-RDNA ones too, and with a recent-ish driver NVIDIA's Geforce line supports it too without hacks.
Problem isn't really HW support, its that the software side is super glitchy and its not all that easy to configure and in most cases requires a 2nd GPU if you still want basic host functions.
Where as VirGL is much easier to get working and doesn't require specific HW support as far as I'm aware.
You are likely confusing what is better termed "PCIe passthrough" with virtual functions wherein one physical GPU presents itself on the PCIe bus as (dozen's in the case of SR-IOV or thousands in the case of SIOV) of GPU functions which can be passed to dozens or thousands of GPU enabled VM's.
Unfortunately AMD cards suffer from a reset bug, still, when used with passthrough.
The reset bug being that you can pass through the card fine, once. But if you try to pass it through again (or the card experiences an issue and needs to reset), they get caught in some kind of bad state and won’t work until power is removed and restored. Which requires a reboot or a only slightly less disruptive dance with system power states.
Incidentally, nvidia gpus are so good at resetting, they’ve probably done so without you noticing. If the screen ever goes black for a fraction of a second and returns in normal usage, it was probably because it reset itself.
The lower 6000 series lower than the 6800’s for example may or may not have the issue. It seems most “reference” cards are fine, but custom vendor cards often but not always have issues. My reference 6700 works fine, but a sapphire 6700 probably won’t.
And the 7000 series is also fucky in a new way somehow. Gnif knows far more about this than me, and has basically thrown up his hands at how AMD doesn’t care. He’s made occasional posts about it on https://forum.level1techs.com/
When it comes to splitting a gpu into virtual ones with SR-IOV/MxGPU, that’s not really a thing with AMD. Whereas Nvidia will happily give you what you want if you shovel some money into their gaping maw, AMD won’t even give business customers the time if day if you aren’t worth billions. They very deliberately do not want the unwashed masses to use MxGPU. See: https://www.reddit.com/r/VFIO/comments/eqvn9z/amd_mxgpu_or_s... for a summary of the years of hopelessness on this functionality.
You could also be in a fun grey area where your 6600XT will sometimes reset properly and allow the physical host to reclaim it for its own purposes. Or not and require a full forced shutdown and reboot to restore proper function.
Let's just say I'm very aware of AMD's issues in this area :P
Also looking glass is pretty great though it's use of the Desktop Duplication API seemed to carry with it a huge performance hit. Or rather did the last time I used it (it's been a while)
It's not really to different to giving your web browser access to your GPU (and by extension to random websites using WebGL). So yes, it's dangerous. But it is at least a threat which designers of GPUs are already considering. Although there have been interesting bugs where GPU memory hasn't been zeroed before allocating it to a new context and you could read previously written graphics memory to find secrets.
As long as 1 GL context on the guest side == 1 GL context on the host side then it _should_ at least be as safe as letting your web browser access your GPU but certainly not as safe as using an IOMMU to segregate a whole GPU solely for your VM.
I feel like a good half of machines I find with glitchy graphics drivers seem to show bits of textures from one application inside another application - indicating memory contents leakage between contexts. Chunks of webpages from Chrome appearing in 3D games seems common.
And those are accidentally caused leaks. As soon as someone starts storing actually sensitive data in graphics memory, I'm sure lots of methods to deliberately cause leaks will be found.
I've had a more extreme case in a dual boot configuration of some graphics corruption on my Linux desktop exposing a mirrored and discolored frame of my prior Windows desktop from before rebooting.
There is a project I am following to embed a Linux machine into mobile VR applications using virtio-gpu & his latest effort gets 200fps.
"VR Linux is now testing with working GPU acceleration via the kernel over virtio-gpu. This is a major step forward following on from the Shadertoy device that we made a few months ago. Now we can make OpenGLES Linux programs that are accelerated by the Quest's Adreno GPU."
VirGL is definitely an interesting project, but all one has to do to get GPU passthrough working (from a Linux QEMU host to any guest OS) is: 1.) research a cheap, secondary GPU that is natively supported by the guest OS, 2.) plug such a secondary GPU into a PCIe slot on the host and hook it up to the primary monitor with a secondary cable (D-Sub vs. DVI, etc.), 3.) setup Linux to ignore the secondary GPU at boot and configure a QEMU VM for the GPU passthrough. The whole process takes perhaps [edit: "a few"] hours and as works flawlessly, with no stability issues [edit: "at least with Asus motherboards"]. (Switching across the two GPU cables can be accomplished in software by using Display Data Channel /DDC/ utilities and switching keyboard/mouse can be accomplished by using evdev /event device/ passthrough.) More information: https://github.com/kholia/OSX-KVM/blob/master/notes.md#gpu-p...
All gaming laptops come with a secondary gpu. What i do on mine is a disable the dedicated gpu, enable passthrough, and voila - shitty windows can run virtualised with a dedicated gpu. Takes 5-10 mins to do so. Also if your laptop supports thunderbolt or usb4 then all you need is an egpu. But thats the more pricey solution.
I have an AMD 290 or whatever GPU to run NixOS (will be upgrading to Intel soon) and a 3060 that I pass into a Windows VM for gaming, I feel very rational.
Unused PCIe slots in a laptop is hard, haven't tried but I imagine Thunderbolt could work for this purpose.
Good point, but I believe that was a serious problem 10 years ago, while these days virtually any decent motherboard properly supports IOMMU, consumer-grade boards included - e.g. any Asus motherboard should work perfectly.
They might support IOMMU, but the default groups can be very annoying to work with so you have to use a patched kernel as well that "ignores" the actual groups.
IOMMU is a integrated CPU feature, so consumer motherboards do not affect reliability - they just make the setting available.
IOMMU is also required for various modern security features, and M$ requires it for certification nowadays to protect against DMA vulnerabilities (heard of thunderbolt?).
But forwarding real GPUs limits the number of VMs and can cause stability issues if unlucky - PCIe device bugs, especially reset bugs, are not unusual. Had problems with e.g. a forwarded rx580 that would require a hard reboot of the host to fix...
Things like Intel GVT-g and VirGL are better solutions, when they can be used.
I've been forwarding Sapphire Radeon RX 580 Pulse to both Windows and macOS for literally years and, except for the specific problem of host sleep/wake, had no problems whatsoever. Perhaps try an Asus motherboard?
> Things like Intel GVT-g and VirGL are better solutions, when they can be used.
Sure, when software solutions are available and stable (and there's no need for near-native GPU performance), they are definitely easier to work with. However, as of today, GPU passthrough is probably the only solution available for a daily driver.
I’m not sure what you mean by forwarding, but if you mean regular gpu passthrough and the reset bugs with amd gpus, then for the rx580 vendor-reset should typically work: https://github.com/gnif/vendor-reset
Does not work with the lower 6000 (below 6800) and 7000 series that can also have reset issues.
> The whole process takes perhaps one or two hours and as works flawlessly, with no stability issues.
Good joke, that really made me laugh :)
I tried this on my ASRock X370 Taichi a while back. Turns out that there is a bug in older bios versions and the whole thing just freezes when starting the QEMU VM. Then there is an intermediate bios version with which I actually managed to get it working. Unforunately I later upgraded my CPU and had to install a new bios and this again completely breaks the IOMMU groups. Probably spend a few days to get everything running, including downgrading from a non-downgradeable bios version.
And even when it was working it was a pain to use. Want to use the passthrough GPU in Linux? Now I have to dual-boot QEMU VMs or disable the passthrough, reboot, then enable it again, reboot one more...
I've been forwarding an AMD GPU to both Windows and macOS for literally years across multiple Asus motherboards and, except for the specific problem of host sleep/wake, had no problems whatsoever, even considering I work under GPU-passthrough VMs whole-day, every-day. Perhaps try a recent Asus motherboard?
> have to dual-boot QEMU VMs or disable the passthrough
Yes, you would have to buy as many cheap, secondary GPUs as the number of virtual machines that you want to run in parallel.
> I really want proper GPU virtualisation...
Sure, I don't blame you - my point was that the only truly usable GPU virtualization solution available today is GPU passthrough and that GPU passthrough is much easier to setup than it is commonly perceived.
And if I don't want to or can't afford to buy new hardware?
> Sure, I don't blame you - my point was that the only truly usable GPU virtualization solution available today is GPU passthrough and that GPU passthrough is much easier to setup than it is commonly perceived.
Okay, but for the poster you're replying to it is not available on their hardware.
I also have an AM4 ASUS board (is that recent enough?) and earlier this year, ASUS decided to completely remove any mention of this board from their site, as if it never existed. So no bios updates for me I guess? No idea if it is up to date or not or if my CPU is even supported...
> Yes, you would have to buy as many cheap, secondary GPUs as the number of virtual machines that you want to run in parallel.
Except that cheap GPUs are... cheap and not very powerful, so depending on what I want to do I would have to buy a bunch of expensive and powerful GPUs (or go back to rebooting VMs to switch). And there are only so many PCIe slots on my board (2x8 and 1x4, the latter of which is already in use by a non-GPU card). Running them in an x1 slot also doesn't sound like a great idea.
I have tried to run it but gave up in the end after breaking bios updates and not wanting to spend even more money on another fast GPU.
> earlier this year, ASUS decided to completely remove any mention of this board from their site, as if it never existed.
Maybe your board was only disappeared on some locales? Maybe see if you can find it on asus.cn or one of their other regional sites (translation service required, but you can probably muddle through)
I haven't seen that in a long time, amd640 super7 chipsets got disappeared, but back then you could still get the bios updates via ftp, the boards just dropped off the website.
There's also virgl_test_server which runs the virgl protocol over a socket.
This can be used for OpenGL inside containers, by bind mount-ing the socket into the container (/tmp/.virgl_test).
Also useful for debugging with rr, even with the nvidia driver you use OpenGL over virgl with the environment vars __GLX_VENDOR_LIBRARY_NAME=mesa LIBGL_ALWAYS_SOFTWARE=1 GALLIUM_DRIVER=virpipe
I've never been able to get this to work under UTM on an ARM Mac; if anyone knows the magic qemu and VM settings, please drop me an email (email in profile).
It would be nice to have fullscreen, full res 3d accelerated VM guests finally. I've never been able to get this to work as smoothly as I think it should.
I've always found it interesting, and a bit unfortunate, that it seems like things like apitrace are generally pretty reliable, but tools like VirGL seem to often have trouble with typical programs. It seems like the problem of accurately serializing what is being done to the GPU using an API is possible, but in practice there is clearly a lot more to it than that. (Of course, in case of remoting, you do not have the benefit of observing the interactions with a known-good driver and with direct memory mapping/etc. So, that probably answers the "why" right there.)
It's been a while, though. Maybe this works really well... I'm curious to hear if anyone has more recent experience messing with these things.
That's incorrect, the goal of VirGL is pass the GL commands themselves (Gallium intermediate representation of them, actually), from the guest to the host, so they are executed on the host accelerated Mesa stack. It is not about virtualising the GPU hardware itself.
So it makes sense to one day do the same for Vulkan, which is much lower level and might be able to extract more performance.
I tried to use it 1.5years ago when playing CSGO inside a VM. It worked, but the delay (between mouse movements and changes on the screen) was quickly raising to something like 0.5-1s.
It required some host daemon to be cooperating with qemu back then (virgldeamonsthsth).
The recent moves of Apple hypervisor being able to use Rosetta on intel cpu code it its already translated(Qemu does cpu intel translation) means that one can use Qemu to simulate x86 64 on Apple Silicon....
See LLVMpipe a little ways up the list in the left hand pane. It should do exactly what you're looking for.
It's worth noting the terminology "passthrough" means something else in this particular context and VirGL is not considered a passthrough solution (as noted in the "out of scope" section). VirGL is a form of paravirtualization often called "API remoting".
Yes, llvmpipe or lavapipe is awesome. It is fully Vulkan 1.3 compliant and the performance is more than adequate for GUI apps, simple 3d apps or running your gfx tests in a CI environment without a GPU.
uh, this doesn't look like what those gpu passthrough things like vfio do, is it a different thing? Or I am more interested in, can this do what vfio do, run games in almost native performance in windows guest vm?
> run games in almost native performance in windows guest vm?
No, vfio is still the most recommended option.
This is essentially something like VirtualBox or VMWare 3D acceleration - the host is still the "owner" of the GPU, and has to juggle receiving OpenGL commands from the guest and sending the render results back to it, with lots of overhead. But easier for users than setting up passthrough or using proprietary GPU virtualization solutions.
People pointed out that there is a similar project for Vulkan, already being used in production for ChromeOS, called Venus. Should be the one to watch nowadays.
Virgl just sends opengl commands to the host. And it also has to copy some buffers back and forth. So it won't be as fast as vfio. But, there is a new feature called VirtGPU DRM native contexts which eliminates some of this overhead and run at near native speeds. Unfortunately it's only implemented for adreno/msm GPUs for now.I think there is work on the way for Intel and AMD GPUs.
It's been a while since I've really looked, but I seem to remember that the major contributors are either employed by various hardware/software vendors (Google, Valve, Intel, AMD, VMWare) or do contract/consulting work on the Linux graphics stack (LunarG), not unlike the Linux kernel.
[1]https://github.com/virtio-win/kvm-guest-drivers-windows/pull...