Unless you want super lower power consumption, you're better off with an second hand ex-office micro computer like an Intel NUC or the various Dell/HP/Lenovo VESA mountable PCs.
Almost all of them have replaceable RAM and happily support an M2 and a 2.5" SSD if you want to RAID the storage, and you won't be relying on an SD card either.
~$100 USD will usually get you enough to run a bunch of VMs, not just a VPN.
Don't ignore the advantages of buying a used laptop (e.g. a ThinkPad), either! Battery backup against power loss, super power efficient, keyboard and screen for when your networking or SSH config is screwed up...
Not going to put a battery powered device not made for 24/7 into a corner of my flat, to be running 24/7 and be forgotten about. I've seen one too many puffed up batteries in a laptop to risk that. Nope.
When networking isn't working, SBCs have serial consoles for that.
These days power bricks are not external power sources anymore but merely charging devices.
CPU spikes can create power draws that could not be handled by power bricks as the system is designed as a whole, including a working battery to absorb such spikes, allowing for a downsized power brick (cheaper, simpler, smaller).
Gimping the CPU allows the machine to reliably operate in absence of battery, otherwise CPU spikes would result in power loss.
FWIW, ThinkPads support setting custom battery charge thresholds, which lowers the risk in this scenario. It also prolongs the battery life as the battery is not kept charged at 100% for extended periods of time.
Laptops, at least in my experience, don't turn themselves back on after a power cut so you have to pay attention to them. It was a pain for a CCTV server I had.
And if you care about power consumption, there are bunch of rockchip rk356x based SBC's with reasonable PCIe and SATA implementations popping up every other day, with 2-8GiB or RAM available for less than $100.
Can you recommend one or give hints on what you have to consider when shopping for a board like this? I mean, the thing about the Pi is that you know what you will get and what use cases are supported.
Linux 5.18 will have reasonably complete support for headless uses of quartz64-a. It's best to check https://elixir.bootlin.com/linux/latest/source/arch/arm64/bo... for whether the board is supported mainline, and how much of the hardware is enabled. Other than quartz64-a, and some bannana pi, it looks like not many others are supported, yet.
Something with an i5-6500T ($100-$120 in an 8/256 config) is insane bang-for-the-buck relative to a Pi. Especially with the tendency of Pis to multiply.
I've been lazier - I've just been buying ~$120 "Minisforum" PC's (4 now, for various people I've set them up for). They work great, but their CPU performance compared to the i5-6500T leaves a lot to be desired (1000/1400 vs 1800/4800 single/multi). They still kick a Pi's butt, and they are x86 which makes lots of things easier.
My main hesitation with buying the used USFF has been the Intel Graphics generations - I don't know enough about it, but I do use my mini PC for Plex, and have a hard time comparing the integrated graphics, which is critical. These N4000 destroy the i5-3450 I have in my NAS. It was cheaper to buy an N4000 and put Plex on it over network access, than it was to completely upgrade my NAS to a new generation of hardware, and the newer QuickSync version can actually encode with Plex.
My understanding is that QSV transcoding stopped being trash with Skylake and newer, but I'm basing that off of this thread and use an i5-10400 for my own.
My original "Pi for PiHole" replacement 5-6 years ago was a used AMD GX-415GA NUC-alike that I spent less than $40 on and half was for an SSD. I keep it around still because it cold boots incredibly fast and if there's anything you want to boot really fast, it's the thing running your DNS.
Almost all of them have replaceable RAM and happily support an M2 and a 2.5" SSD if you want to RAID the storage, and you won't be relying on an SD card either.
~$100 USD will usually get you enough to run a bunch of VMs, not just a VPN.
Server the home (no affiliation) had a brief write up on some of them: https://www.servethehome.com/introducing-project-tinyminimic...