I did a lot with RPi's back in the 2-3 era... My SDCards were constantly getting nuked by slightly noisy power sources when running timelapse photography over the course of days. My workaround was to use the SDCard for boot only, but running everything else off a USB drive. The sdcard then only ran for a couple minutes at startup, in read-only mode, and never had a chance to screw up a write.
This worked great, and allowed me to successfully run a four-month timelapse, where I didn't have access to the device. (You can see it here: https://www.youtube.com/watch?v=cyh0QJT0rAQ )
All the modern Pis can now boot solely over USB, and the Pi 4's USB 3.0 ports mean it actually has decent IO speeds for general use (and especially for things like transferring files over the network faster than a couple MB/sec!
Now that I have the HQ camera, I'm trying to figure out a good way to mount that, a Pi 4, and a USB SSD so I can have a much faster/nicer time lapse rig.
Based on some of your work (thanks) I've got 4xPi4-4gb booting from a fifth Pi4-2gb that is running dnsmasq (PXE only) and NFS for / and /boot. Boot order is 0xf412 - network, SD, Usb, repeat. The /nfs/tftpboot/<host_serial> is a soft link to the boot directory inside /nfs/<imagename>-<hostname> Each host gets its own directory - haven't looked into a CoW filesystem yet.
I've got a hacky Ansible script that will reboot a specific host from network, dd a fresh Raspios image onto the sd card, mount the root filesystem, copy over the ssh keys, reboot the pi, do the usual apt update/upgrade and a couple other tweaks, reboot from the network, dd the sd card back to an image and finally run Pishrink.sh on it. It's fascinating to watch in a Rube Goldberg kinda way.
I would add that net bandwidth seems quite respectable though not USB3 speeds. One issue i've run into is that K3S does not start - mostly likely due latencies with networked disks
That HQ camera is pretty awesome. I bought one to build a film scanner with. For those who don't now, it's a camera module for Raspberry Pi that comes with a 12mp sensor that supports attachable camera lenses(sold separately). It also comes with a really nice guidebook.
I absolutely concur with this setup. Power supply and SD Card can make or break your RPi experience. I've been running a Pi 2 for ~5 years in this setup i.e. Sony 2A power + Toshiba SD Card + USB drives for Git (SD card and USB drives encrypted).
It's been used only as a git server 24*7 for substantial amount of code, I reboot it once a year. Using it as a Git server started out as an experiment, but worked out great until the code became large and hung-up occasionally during commit due to CPU/Memory bottleneck (in-spite of adjusting relevant Git flags).
Recently I've started putting it to other usages, so not sure whether the reliability would be the same. My RPi3 needed official power supply and better cooling to extract similar performance/reliability ; Although it officially supports booting via USB I didn't have much luck in getting it work and had to use SD Card for boot and USB SSD as Arch Linux ARM/Btrfs root[1].
Having worked on firmware writing to SD cards, I can tell you that the big problem comes from allocating space for writes. That’s when the card disappears on you, as it looks for space it will potentially shuffle data around behind the scenes. Current spec I believe says max latency is 200ms. This is the operation that will abuse this part of the spec. If you want to see an SD card flail, just try to write one byte at a time. This will cause headaches no matter how wear leveling is implemented in the SD card. And will likely significantly shorten the lifespan of the card. If you can make sure you only ever write multiples of 4096 (probably card dependent) you’ll be much happier using SD cards. But even then, you might have a 200ms window where you can lose or corrupt data if your power to the card fluctuates significantly.
IIRC SD cards weren't made to handle the read/writes of running an operating system off them. Which is why you see them break so often when used with Raspberry Pis
My understanding - at least for the original Pi - was that there was a design flaw where the sd card was not protected from power irregularities in the same way the rest of the device was. There's a lot of argument over whether it's a hardware design flaw or a problem with sd cards generally (perhaps the usb spec has better buffered writes or something)... It's apparently much more a problem for long-running applications with high write frequency.
To avoid these problems I am using an overlay FS (just reads from flash and writes to memory) and after a reboot all changes are gone. That might not be suitable for everyone but works great for stuff like mqtt brokers or devices which pull their configuration on boot.
This is how I am using my Raspberry Pi (running Pi-hole) too: overlayFS. It means that I lost the statistics after a reboot, but it also mean that I don't care if the power in the house is suddenly lost either. I also enabled watchdog support so my Raspberry Pi is probably the most reliable part of my network.
Same here. I run temperature monitors on mine. Every five minutes a cron-job copies data to a /boot subdirectory which is still r/w. On power loss, the most I can lose is those five minutes.
I was going to ask if you had considered battery power, since a battery is basically the cleanest power source you can possibly get. But, your link makes it seem like basically a "set it up and check on it in X months" type project.
This worked great, and allowed me to successfully run a four-month timelapse, where I didn't have access to the device. (You can see it here: https://www.youtube.com/watch?v=cyh0QJT0rAQ )