All the modern Pis can now boot solely over USB, and the Pi 4's USB 3.0 ports mean it actually has decent IO speeds for general use (and especially for things like transferring files over the network faster than a couple MB/sec!
Now that I have the HQ camera, I'm trying to figure out a good way to mount that, a Pi 4, and a USB SSD so I can have a much faster/nicer time lapse rig.
Based on some of your work (thanks) I've got 4xPi4-4gb booting from a fifth Pi4-2gb that is running dnsmasq (PXE only) and NFS for / and /boot. Boot order is 0xf412 - network, SD, Usb, repeat. The /nfs/tftpboot/<host_serial> is a soft link to the boot directory inside /nfs/<imagename>-<hostname> Each host gets its own directory - haven't looked into a CoW filesystem yet.
I've got a hacky Ansible script that will reboot a specific host from network, dd a fresh Raspios image onto the sd card, mount the root filesystem, copy over the ssh keys, reboot the pi, do the usual apt update/upgrade and a couple other tweaks, reboot from the network, dd the sd card back to an image and finally run Pishrink.sh on it. It's fascinating to watch in a Rube Goldberg kinda way.
I would add that net bandwidth seems quite respectable though not USB3 speeds. One issue i've run into is that K3S does not start - mostly likely due latencies with networked disks
That HQ camera is pretty awesome. I bought one to build a film scanner with. For those who don't now, it's a camera module for Raspberry Pi that comes with a 12mp sensor that supports attachable camera lenses(sold separately). It also comes with a really nice guidebook.
I wrote about my experience and testing with a cheap Kingston SSD last week: https://www.jeffgeerling.com/blog/2020/im-booting-my-raspber...
Now that I have the HQ camera, I'm trying to figure out a good way to mount that, a Pi 4, and a USB SSD so I can have a much faster/nicer time lapse rig.