Semi-related question... I recently spent an evening trying to familiarize myself with virtualization tooling and my initial impression was that the CLI experience for kvm and hyper-v was rather clunky compared to let's say docker, gcloud, ignite and kubectl. Also a lot of the learning material seems to be oriented towards GUI. It could be that I just haven't spent enough time with kvm and hyper-v. I've spent a LOT of time on those other tools so maybe I'm just more used to them.
Is there good CLI tooling available for virtualization? I would say that I'm looking for the "on-prem" experience where I start by configuring a virtual network, download some unmodified ISOs directly from the operating system vendor, provision my headless virtual machines with a combination of Powershell, Answer files, Bash and Kickstart. I then provide services like DHCP and DNS using my virtual machines rather than the built-in mechanism provided by the virtualization tooling.
There's a lot of useful command-line tooling for KVM- and QEMU-based virt. Here's a small selection of them:
• virsh — This[1] is libvirt's shell interface; and gives you access to the rich set of libvirt APIs.
• virt-builder — Use this for rapidly building minimal or customized virtual machines; it's greatly flexible; check out its man page[2]. And here's[3] a quick example that connects both virt-builder and virsh together.
• virt-install — Use this if you don't like the default build of the template images from virt-builder; it lets you create "headless" servers via 'kickstart' and Linux OS trees from the command-line.
• guestfish and libguestfs suite[4] — This rich set of tools help you in a variety of use-cases: repairing your broken disk images, editing, cloning, debugging disk images, and more. It has saved my behind a lot of times.
• qemu-img[5] – This Swiss Army knife lets you powerfully manipulate disk images (QCOW2, raw, et al) offline. Example operations include: create images, backing chains, offline snapshots, disk image merging, ability to convert disk images from one format to another, and more.
virsh and/or lxc is all I ever use. All that cloud stuff, docker, kub-whatever is overkill to run a few VMs or containers at a small scale on-prem. KISS.
In terms of speed, how is virt-builder compare to say Terraform?
Terraform is dog slow sometimes due to cloud provisioning. Would be nice to just be able to build the VM locally then push the AMI/VHD into AWS or Azure.
virt-builder can usually build out a disk image in 15-60 seconds if the template has already been downloaded and you're running on baremetal. Might be 120 seconds if you have to use nested KVM or TCG.
$ virt-builder fedora-33
[ 1.5] Downloading: http://builder.libguestfs.org/fedora-33.xz
[ 2.3] Planning how to build this image
[ 2.3] Uncompressing
[ 8.6] Opening the new disk
[ 13.8] Setting a random seed
[ 13.8] Setting passwords
virt-builder: Setting random password of root to mZbPJw9d1ZHgowBk
[ 14.8] Finishing off
Output file: fedora-33.img
Output size: 6.0G
Output format: raw
Total usable space: 6.0G
Free space: 4.7G (79%)
I don't know how Terraform works, but `virt-builder` will cache the xz-compressed template images locally on your first pull. So for example, if you build a Fedora image:
Then the (xz-compressed) template Fedora 32 image will be cached under ~/.cache/virt-builder. So your subsequent Fedora 32 image provisioning will be much faster.
You might want to give it a whirl and see if it satisfies your needs; `virt-builder` should be available on most major Linux distributions.
I stopped bothering with libvirt and other frontends for local dev environments. Now I use shell scripts to start my VMs. I only create one per month or two so it isn't that much of a hassle.
I second this! I have a shell script to download if necessary Debian, run its installer with setting matching production and then run VM with necessary development directories exposed to it. I tried initially to do that with virsh, but then I gave up. Raw qemu commands in fact rather straightforward indeed.
Yeah, if you have a few virtual machines for specific purposes, there's not much point bothering with some abstraction on top of qemu.
I also have just a few 20 line scripts to run my VMs, and that's all. Works with no changes/hassle for 5 years already or more. No extra SW to learn. Man qemu gives me all the answers. Easy.
I guess if you want to manage some complicated setup and dynamically add/remove disks, network cards, or whatever, all the time, migrate machines, etc., some solution like virsh would be good, but for having everything on one bridge, in one subnet, to run some throwaway VM with Windows/browsers for web testing, the simpler setup is so nice.
You're comparing VM tooling with container tooling, there's going to be substantial differences in level of abstraction. I might suggest you look at the libvirt platform, it's a wrapper around KVM essentially, but there's virsh which I've found reasonably straight forward to work with.
Yes, virsh is really a nice tool to work with. It has multiple times saved my ass when the graphical virt-manager refused to work (for whatever X11-reason).
For small installations or a single host, plain old virsh is probably what you are looking for. Point it to an iso and you're good to go. Not sure if it does any post install provisioning, but that's what kickstart and ansible is for.
Larger virtualization installations will already have a management layer, something like oVirt or vSphere, that has similar tools available.
I had some QEMU VMs running for a while on a home server that were more or less started by command line. You can specify everything as command line parameters.
I wrote a script that pulled some info from a sourced-in bash "config file" for a given VM (such as amount of CPUs, RAM, and where the disks were), executed the appropriate ip commands to create the taps needed for network access (including a private inter-VM network), and then built/ran the long QEMU command with it in a screen session.
Which file is the ISO for the virtual CD ROM is just another QEMU command line parameter.
I had my own bind running in a VM and created a view on the same subnet as the private VM network. Since this was only a few VMs I didn't bother with DHCP, I just statically assigned IPv4s. I used VNC to setup the OS in them.
I'm not as familiar with Hyper-V but certainly for KVM you would probably interact with it (indirectly) using virsh which I find pretty capable. No sane human should be interacting directly with the qemu command line if they can help it, it's written more for consumption by other software.
I'd recommend Packer and Terraform for this. Packer to provision the machine images from unmodified vendor ISOs and then Terraform to provision the network and VMs.
I include the blog link because there is some nuance in how to get the path right for community Terraform providers that aren't in the Hashicorp registry. The documentation on the GitHub project isn't quite up to date with respect to how the latest versions of Terraform expect the plugin paths to be set up.
I've done this pretty successfully with all the major Linux distros minus Arch, which requires some bootstrapping to get an iso that Packer can work with (no such thing as an answers file for Arch). It's not that big a deal, though. Just find some instructions on how to create and mount a cloud-init iso in addition to the installer iso and use that to add an ssh public key so you can script the installation steps externally. I actually think Packer can do this, but I just haven't gotten it to work yet and have relied on shell scripts.
Hyper-V actually has a very comprehensive PowerShell module that is pretty well documented, by the way: https://docs.microsoft.com/en-us/powershell/module/hyper-v/?.... I've found it pretty easy to use and actually got the Arch auto-provision working on Hyper-V in Windows before I got it working in KVM in Linux.
Another thing is you can just use the cloud images and cloud-init for bootstrapping everything pretty easily, even on-prem. cloud-init has a "no cloud" config option, as mentioned above, where you just mount an iso with the config data as a DVD drive and cloud-init will find it automatically when the distro iso boots.
This guy has a pretty comprehensive example of how to set up a kubernetes homelab entirely using the libvirt Terraform provider from Ubuntu cloud images bootstrapped with cloud-init: https://github.com/zloeber/k8s-lab-terraform-libvirt
I use Hyper-v for years and I rarely create new VMs, this is why I use GUI, it is convenient if you don't do this often enough to remember the syntax and params. Most people that I know are in the same boat, but for mass deployments or automated deployments PowerShell is the way to go.
Sorry for the approximative English, I'm losing my vocabulary with WFH and I'm pretty sleep-deprived.
I've been building a game streaming platform for influencers for the past months. We scaled to 9k users in a month.
We have android x86 running on QEMU-KVM and there are a bunch of issues I need to address.
- virsh (and virt-manager) always add a PS2 mouse to the guest. The issue is PS2 mouses have a relative movement and not an absolute one -> When you use a mouse or the touch screen through VNC you have to move the pointer to a position instead of directly clicking/capturing the mouse
- My VM resolutions are 720x1080 but VNC always launches in 1920x1080. I have black rectangles on both sides of the Screen.
- Streaming the app viewport with VNC has some (small) latency, any way to fix it?
- Can we host an arm android emulator on QEMU? (armeabi-v7 or v8)
- We had a sales presentation from Canonical for their anbox cloud solution, but they never returned to us with a proposal. Is there any other way for us to have the Android VMs in K8S? Or even Lxc if possible, we are flexible. Also, the juju charms for anbox is nowhere to be found
If you want to get in contact, my email is jadiaheno[at]ludexgames.com
I can even show you a demo, we have influencers increasing their revenue with our Beat the boss campaign. We're not VC backed.
For the PS2 mouse, you can add a USB controller (best is XHCI, as it's more friendly to virtualization and needs little or no CPU usage in the host) and an emulated USB tablet. You can also use the virtio-tablet device if your VM has drivers for it.
Hosting ARM is possible but performance will be ~10 times worse.
- virsh (and virt-manager) always add a PS2 mouse to the guest. The issue is PS2 mouses have a relative movement and not an absolute one -> When you use a mouse or the touch screen through VNC you have to move the pointer to a position instead of directly clicking/capturing the mouse.
I have been using android x86, quite successfully even with touch input. When I did it I didn't use virsh, instead I just used ffmpeg directly via CLI.
>> - Can we host an arm android emulator on QEMU? (armeabi-v7 or v8)
It is very possible to run arm code on a x86 machine, this would however be slower of course.
If you are running on a VPS you might be able to get one with an ARM CPU that supports virtualization.
>> - Streaming the app viewport with VNC has some (small) latency, any way to fix it?
There would be other ways that you can use to remote connect to it such as spice. You might also want to consider running it on an headless X server and then use basically any remote control program.
>> - - My VM resolutions are 720x1080 but VNC always launches in 1920x1080. I have black rectangles on both sides of the Screen.
This sounds as one of the points in the changelog, maybe try to use the newest version?
Hey I don't know how much help this is, but anbox is a very rough solution. The bootup time for the android container is on the order of minutes, and lits does not work.
You can try to get support in #anbox on freenode, though. There are postmarketOS people that know how to get the system working. I've never seen someone use anbox with qemu, though. It has always been same arch.
Hey Thanks for the reply. I will try to use postmarketOS on qemu. I'm wondering if there is any ec2 machine than can support (qemu-)aarch64, and also if AAA games can run on postmarketOS.
It defaults to the GTK display: https://down.loaded.ie/Lb9RDi3.png. See the -display section in QEMU(1) for details and other options. My favorite is -nographic, which forces VGA output to the terminal; very useful when there is no X session or similar.
Edit: I realised GUI might mean creating and managing virtual machines; virt-manager might be good? Take a look at https://wiki.archlinux.org/index.php/Libvirt for the setup needed. (I think it's easier to learn to use vanilla QEMU, though.)
You're being unfairly downvoted, so allow me to actually answer your first question.
QEMU doesn't have much of a GUI to speak of. It's a virtualization framework with some CLI tooling on top of it. Frontends/management UI's/etc are left as an exercise to the community so you can find one that suits your needs.
A recent Windows build used GTK3 decorations; I was very surprised to see that appear. You are right that it is quite low on functionality, its menu allows powering off and rebooting the VM, and gives access to the serial console as well as a command-line interface for the emulator.
QEMU does have a quite primitive GUI while it's running that lets you change disks and so on. But really, it's not a graphical emulator. You specify everything via command-line switches. (They're fairly intuitive, mind you. “-hda drive-c.img -m 16 -cpu 486 -soundhw sb16,adlib -vga cirrus” does what it sounds like.)
There are of course tools that provide a friendlier GUI wrapper if the command-line isn't your thing.
To pass files into VM I use lsyncd. It synchronizes changes to even big source trees over plain ssh and rsync. Of cause with big tree the initial copy takes time, but there is no security implications of exposing files to potentially untrusted VM.
There's an app that wraps QEMU called UTM. It's open source but also you can pay $10 and get it on the app store.
I have had a lot of trouble with machines getting corrupted, not working at all, locking up, but I did get a Debian machine working that runs ARM Linux on my Macbook Air M1.
+1 for UTM - The website for mac (https://mac.getutm.app/) has some recipes and pre-built image for Windows and the main Linux distros. I had okay results with emulating Windows -- nothing usable in my day to day, though.
HVF does build with Nix (with an overlay), so the build experience doesn't suck. [1] for the base, then apply [2] to SLIRP if you don't want to use HVF's network adapter (which needs root). I'm happily running FreeBSD and Linux VMs on my Mac Mini with this
Can something like this be used to, say, build emulators for different game consoles? And do the popular emulators use this as a building block? Asking this as someone whose only experience with emulation and virtualization is running games on VisualBoyAdvance and running Ubuntu on VirtualBox
Are both original Xbox emulators built off of QEMU (Xemu is a fork of XQEMU). I've only used Xemu, but performance was pretty good for the games I tried on it (it doesn't have a way to upscale rendering yet though).
You could build emulators for consoles, but QEMU is not cycle accurate. Meaning some operations may finish faster or slower than they would on the original console. Some games may rely on that timing for things like physics engines and break in weird ways.
Yes. xbox360 games running on xbox one often run with a custom emulation layer that includes a bunch of monkey patch fixes to make the game work right.
its not really meant for that, but there wouldn't be anything to stop someone from creating the hardware emulation, as far as I know. Still, purpose-built console emulator software is sure to be far better for the task.
I have heard about QEMU for years. Something about virtualization and Linux, this is what I've started to associate it to.
My Workstation runs on Windows, and up until some years ago I was using VMware Workstation for virtualization, and then moved over to VirtualBox, which I've also started to use on a Windows-based home server. I've always used both of them via the GUI.
Now, if I want to build a beefy Linux server (headless) and run some virtualization on it, would I then use QEMU and get the same thing I've been getting with VirtualBox on Windows?
How do both of these compare performance- and feature-wise? Are they alternatives, or do they server different purposes?
Qemu is a whole-system or CPU emulator, originally for x86 PCs but now encompassing other systems and architectures.
Where the virtualization bit comes in is with kvm. Kvm is a Linux kernel module that lets you take advantage of x86 CPU hardware virtualization. Qemu can use kvm to provide the virtual CPU while it itself emulates the rest of the PC hardware, resulting in a complete virtual system.
That said, qemu can also emulate the CPU itself, allowing for example an x86 OS to run on an ARM system, or vice versa, but that of course is slower.
TSO is Total Store Ordering, which refers to the memory model of x86_64. For Rosetta 2, Apple will switch the M1 processor's memory model when emulating x86_64.
I wish you luck! I've been reading a lot about the MiSTer project lately and it is getting harder and harder to resist taking the plunge. There is some amazing work going on there lately.
For those that missed it, here is a recent article on the project...
Last I played with the coldfire and 68k emulations in qemu to try to get it to boot EmuTOS (an Atari ST operating system) I found a lot of the I/O devices lacking, even just to run headless.
It was sufficient to boot Linux in some configurations but not complete enough of an emulation to run other things. This is in general the story with many things in QEMU.
I didn't keep notes, I suppose I should have. I might revisit this again someday.
It's been probably about 8 years or so since I touched QEMU, but at the time the performance was unusable (on a modest system) compared to a very usable virtualbox or vmware. IIRC, I was just trying to run win XP on Windows 7.
How do things stand these days?
Also, given that QEMU is somewhat platform independent, how well does it run an x86 OS on top of an ARM chip?
If anyone knows how to improve the graphics performance on Windows, please share. Compared with other hypervisors like Virtualbox or ESXI, there’s no graphics driver for Windows VMs with QEMU.
Is there good CLI tooling available for virtualization? I would say that I'm looking for the "on-prem" experience where I start by configuring a virtual network, download some unmodified ISOs directly from the operating system vendor, provision my headless virtual machines with a combination of Powershell, Answer files, Bash and Kickstart. I then provide services like DHCP and DNS using my virtual machines rather than the built-in mechanism provided by the virtualization tooling.