I was an Amazon EC2 Specialist SA in a prior role, so I know a little about this.

If EC2 were like your home server, you might be right. And an EC2 bare metal instance is the closest approximation to that. On bare metal, you've always been free to run your own VMs, and we had some customers who rolled their own nested VM implementations on it.

But EC2 is not like your home server. There are some nontrivial considerations and requirements to offer nested virtualization at cloud scale:

1. Ensuring virtualized networking (VPC) works with nested VMs as well as with the primary VM

2. Making sure the environment (VMM etc) is sufficiently hardened to meet AWS's incredibly stringent security standards so that nesting doesn't pose unintended threats or weaken EC2's isolation properties. EC2 doesn't use libvirt or an off-the-shelf KVM. See https://youtu.be/cD1mNQ9YbeA?si=hcaZaV2W_hcEIn9L&t=1095 and https://youtu.be/hqqKi3E-oG8?si=liAfollyupYicc_L&t=501

3. Ensuring performance and reliability meets customer standards

4. Building a rock-solid control plane around it all

It's not a trivial matter of flipping a bit.

There's no better way to get good information that is right, than to say something that is misguided and/or wrong.

Thanks for the well-reasoned response.

I always enjoy the color you add to these conversations. Thanks!

I always enjoy the color you add to these conversations in your newsletter.

It's provided many a chuckle.

Thanks!

Seriously curious, don’t Firecracker VMs already run on EC2 instances under the hood when they host Lambda and Fargate?

Since I don't work for AWS I'm allowed to say that at the scale of millions/billions of microVMs you're better off running them on bare metal instances to avoid the overhead of nested virtualization.

I used to work for AWS and I’m allowed to say the same thing. ;-)

If I remember correctly, Firecracker VMs don’t have the same security guarantees as EC2 instances. I think I remember that AWS doesn’t put multiple accounts lambdas either on the same bare metal server or VM. I can’t remember which

There is no way a random small account running a single serverless function gets a whole bare metal server dedicated to them.

Unfortunately I'm not at liberty to dive deep into those details. I will say that Firecracker can be used on bare metal EC2 instances, whether you're a public customer or AWS itself. :-)

I guess I should have peeked at the source code when I was there…

No need, at least when I was there when the day was still one, before the pandemic. And well, Firecracker is open source.

A few of the best technical presentations that I've watched were at a pre-SKO event. Nitro, Graviton and Firecracker.

Great engineering pieces, the three of them.

All that sounds like it would better be a contribution to KVM from the get go rather than invent stuff that eventually showed up in KVM anyway

it's been in kvm since the mid 10s

and in Xen (which they used to run) for at least as long

I meant in general, as the linked presentation talked about many features, not just nested virt

Nitro is very interesting stuff

the only thing I know about nested virtualization is from the libvirt/KVM world too:

* you are right, it just works

* but there were scary notes about the stuff which might happen when you live migrate a virtual machine between hypervisors and the machine has nested virtual machines inside it. I remember the words "neither safe nor secure"

> * but there were scary notes about the stuff which might happen when you live migrate a virtual machine between hypervisors and the machine has nested virtual machines inside it. I remember the words "neither safe nor secure"

Google does this to customer VMs in production all the time..