People speculate in the comments that Microsoft could provide a 16bit emulator in Windows for DOS applications. In fact they have an x86 emulator in Windows, but not for DOS.
It's used to run int10h calls (VGABIOS) on early initialization, funnily even on UEFI.
I don't understand why they gave up on DOS compatibility for 64-bit Windows.. the effort could not have been all that much for them (either with an emulator or x86 tricks). The benefit is huge: they can continue the lock-in of ancient DOS programs. For example, I can report that OrCAD for DOS works better in 32-bit Windows XP better than it ever did in MS-DOS.
Likewise, I don't understand why vm86 mode can not run directly from 64-bit mode in x86. This seems very like an architectural mistake to me.
As I remember, when an x86-64 chip is running in 64 bit mode, it can run 32 bit code, but the option necessary to run in 16 bit mode was removed. To run DOS they'd basically have to have a full emulator for a 286, like a limited version of bochs.
I don't know what mechanisms they use, but VMware's various products run 16-bit code just fine on x86-64 chips. Are you saying they added a full 286 emulator to their virtualization products?
Essentially yes. But I guess they simply kept their virtualization code from pre-hardware virtualization times around. 286 shouldn't require much maintenance :-)
This has nothing to do with VDM; the emulator is only accessible from kernel mode (it's in HAL.dll) and is not meant to be used by usermode applications. See [1,2].
There's also a similar emulator in X, that allows drivers for PC-specific graphics cards that used x86 initialisation in their video BIOS to work on non-x86 platforms, like SPARC.
It's used to run int10h calls (VGABIOS) on early initialization, funnily even on UEFI.