> So vm86 mode is a special mode in 32-bit CPUs that lets you execute old 16-bit code in the context of a normal process.

It's in 64 bit processors as well. Windows didn't cut 16-bit execution due to the processors not supporting it from long mode. The issue was software constructs like an increase in max HANDLE table size increasing in the transition to 64bit.

Yeah, as far as I know, the real problem is that Microsoft would either develop the 16 but compatibility layer all over again, or run NTVDM on top of the 32 bit compatibility mode (WOW64) which would also be a tonne of work.

I wonder if you could write something like a program loader that uses a hypervisor to execute 16 bit programs natively (as opposed to using dosbox with its inefficient emulation) with some kind of disk level compatibility layer.

You can in fact. Michael Steil talks about his adventures using hypervisor.framework to run DOS applications here: https://www.pagetable.com/?p=764

Executing real mode instructions inside virtualization extensions wasn't supported until Sandy Bridge/Westmere (called "Unrestricted Guest"). AMD Zen has it as well, not sure if any of the FX processors had it. Before then, the real mode init of a virtual machine had to be handled via software emulation.

edit - remove nitpick about this not being vm86 mode because I missed the hypervisor bit in the parent comment.

All of the AMD SVM cores supported real mode in guests as far as I know, and Sandy Bridge is a decade old at this point. But yeah, that was really annoying, retconned "restricted guest" would happily let you put real mode segment info into the VMCS, but just silently would be broken in goofy ways. I tore my hair out for a while writing a hypervisor back then that was intending on running some real mode code until I figured out what was going on.

My question is -are these extentions part of the features in newer CPUs that are being exploited like Spectre etc or is it completely unrelated. I ask because if it is, then it may be a feature (or set of features) that's going to be going away.

[edited in hopes of being clearer]

It's unrelated to spectre-like vulnerabilities.

There is an unofficial build of NTVDM for x64 Windows available: http://www.columbia.edu/~em36/ntvdmx64.html

I wouldn't actually use it for anything but it is interesting to read about.

There is winevdm, but I don't know how that works, just that it does seem to mostly work.