I was doing just this for a while but stopped because my Mac Mini was too old for the last few OS releases. I then switched to using the Windows iCloud client but a bug from ~2 years ago that consumes tremendous CPU cycles made that less than ideal. (The best you can do is lock it to a single thread, which will then use 100% 24/7)
Now I just don't backup my iCloud, though I do remove everything older than one year every new years to my home server which follows a good 3-2-1 backup strategy.
TL;DR: If you go this route, try to get a Mac Mini that can run a supported macOS for some time to come.
I've been using Windows iCloud client to backup to a VPS. It works okay for files, but for photos it pegged disk usage even when there weren't any new photos to download and my VPS provider wasn't happy. So far my solution is Windows iCloud client for files, and then OneDrive on my iPhone for photo backup, with OneDrive again on the VPS.
I like how the backup is outside my house, but I'm about to add Yubikey to my iCloud account and I'm not sure the Windows iCloud client is going to like that.
I was actually running ESXi on my Mini for a while and successfully installed macOS in a VM on it. The performance was horrendous though, so much of macOS depends on GPU acceleration which I didn't get. I think I've read newer macOS builds don't even have a software video fallback, though that might just be the Apple Silicon builds which wouldn't apply to me.
It was definitely a fun project even if not terribly useful.