Anyone using ZFS on a daily-driver desktop computer?
Fedora + BTRFS + Deja dup daily backups have been my default for months now, but I am tempted to give it a go to perhaps Ubuntu JJ + ZFS + Deja dup. Not that I have any issues with my current setup. It's just for the sake of, you know, try new things.
I've used ZFS on my daily desktop computer for almost 3 years. I'm using it with ArchLinux. The overall experience is great but here are some notes:
* The cpu usage can be higher if you use compression.
* I used to setup the main pool with SSD cache with HDD data store. It's okay for most use case except slow first time open of applications. But for demanding games it usually has very bad performance.
* ZFS also manage cache by itself so it needs some memory. Personally I'd like to set a cap for it, because otherwise when I open some memory hungry program it tries to write cache onto disk which sometimes makes the system slow. This happens when I only use SSD as cache. Not sure if it can be better with all SSD solution.
* I ran into problem with docker as well. But after change some configurations it works perfectly.
I have been using ZFS on my daily-driver Ubuntu laptop for six months or so. Also been using ZFS as the backing store for a heavy update postgres database for a few years before that. Haven't had any issues with it at all, generally it seems like a very solid bit of software.
I'm not familiar with Deja dup, but the usual method of backing up a ZFS filesystem is to snapshot it and then stream the snapshot (just a file, can be compressed) into the remote store of your choice.
Streaming ZFS snapshots to a file and storing that in a remote of your choice is not the usual way of backing up ZFS data at all (it’s actually recommended against).
The usual way is ZFS send and receive to another ZFS server. But almost no cloud storage provider supports that, other than rsync.net (and zfs.rent), which starts with $60/month. So you have to set up and manage another ZFS server with secure remote access, PIA if you ask me.
It seems to work well. Am I carrying some unrecognised risk that I wouldn't be carrying if I were backing up into another ZFS filesystem? How does that risk balance against the risk of data loss on the backup server? i.e. I am pretty sure GCS won't lose my data, but not quite so sure about the other services you mention.
If your monolithic backup suffers single a bit error, it's unrecoverable. If you zfs recv, though, you can recover all pristine files. And if you zfs recv to a redundant filesystem (mirror, RAID-Z), you can periodically zfs scrub and recover from those errors completely.
I've been running ZFS continually since about 2009, currently using its replication as a major part of the disaster recovery strategy for my employer, and in the intervening time having been through a number of disk failures, controller failures, etc.
Just chiming in here to say that I endorse every aspect of the answer that 7e has given you.
Have you restored any data using this system? A backup isn't a backup unless you've practiced a whole cycle.
Snapshots are incremental, so management of your retention period is an active process and may require combining older snapshots together when you drop them.
With a zfs server on the other end you can zfs send / recv the other way to restore, and it correctly manages deleted snapshots.
They are but, unless the -i flag is specified (it isn't in my example above) then zfs send will send a complete replication stream, not an incremental.
ZFS receive checks the stream to ensure correctness. The stream can be easily corrupted, due to even a single error, if ZFS send is used without ZFS receive.
The error occurs in transit or on client side, not necessarily on remote.
Also managing a chain of incrementals can get unwieldy.
Thank you for answering! Deja dup it's just a GUI over duplicity (https://en.wikipedia.org/wiki/Duplicity_(software)). It allows me to have encryption, differential backups and restoring files in a granular way (even from Gnome context menu) with no effort. Very pleasant experience similar to Time machine in MacOS. Anyways, I'll definitely give it a go to ZFS.
using it on my notebook with native encryption and I'm quite happy - there are some things missing / work in progress for quite a while - unfortunately it looks like it's not a priority at the moment for the current sponsors or there is a lack of time - not a criticism of zfs but I'd love to see these features merged sometimes in the future:
- no overlayfs yet (https://github.com/openzfs/zfs/pull/9414) - the docker zfs driver is very slow and buggy - best to create a sparse zvol with ext4/xfs/btrfs and use this for /var/lib/docker
- async dmu (https://github.com/openzfs/zfs/pull/12166) - complicated patchset would transform a lot of operations into callbacks and probably increase performance for a lot of workloads.
- namespace delegation was recently merged: https://github.com/openzfs/zfs/commit/4ed5e25074ffec266df385... with idmapped mounts it should be possible to have zfs datasets / snapshots inside a unprivileged lxc-container which could be super cool for lightweight container things.
caveats:
- no swap on zvol at the moment (https://github.com/openzfs/zfs/issues/7734) - this is some hairy memory allocation problem beneath it and I don't really use swap - just something to be aware off.
That's great. I had a conversation with the owner of rsync.net about why they require zfs-enabled customers run on a separate bhyve VM (thus requiring a large minimum). Iirc it boils down to zfs destroy permissions somehow affecting other tenants. I see mention of zfs destroy in the linked patch, I'm curious how that relates to rsync's experience and requirements.
I'm using ZFS on my desktop (Arch Linux). 1 'Pool' is just an SSD that my OS root drive. (Booting off ZFS is a bit harder but putting /boot on your ESP works).
And a hard disk array with 4 drives in raidz (like raid 5).
Main advantages:
* I moved my root drive to another drive and did it entirely online without needing to cp or dd everything just add new drive to pool, remove old drive (and wait until it's done, but you can keep using it while it copies)
* It's way faster than MD, md spends 10s of hours initializing or resilivering big drives. zfs just formats it and your good to go.
* Subvolumes and snapshops are great. You can give a folder different file system properties. (Eg make /var/log compressed and disable it for /var/cache/pacman because it's already compressed)
* Docker is quicker.
* Although BTRFS has snapshots I do find ZFS's more intuitive. (eg subvolume can be nomount in ZFS which allows you to organize and apply options recursively easier)
Check for bugs in Ubuntu ZFS before installing updating. They do it differently than other OpenZFS options.
I'm not currently using ZFS send for backups. So I can't compare it to Deja dup.
Yes, I have Arch on a ZFS root pool. rEFInd + ZFSBootMenu to boot it.
The only thing I don't use ZFS for is a Windows (gaming) VM. I tried numerous different setups to keep the NTFS disk image on a zpool, but it's ultimately pretty hard to beat the performance of PCIe passthrough for a modern NVMe drive. I use syncoid/sanoid + ZFS delegations for backup management.
No real complaints except that it complicates updating Arch: kernel updates usually have to be delayed 24-48 hours while the ZFS packages get rebuilt, or in some cases patched upstream. syncoid/sanoid also poses a problem because it relies on a few Perl packages from the AUR that are a pain to rebuild.
I know the Arch devs loathe people doing partial updates, but occasionally delaying a kernel update for the sake of ZFS has yet to bite me in the ass. I would strongly advise against DKMS, though. Use a binary module or build the package yourself. The majority of problems I had with Arch+ZFS boiled down to misuse of zfs-dkms.
Specifically zfs-dkms + the stable kernel package ("linux") can often be an issue because I have seen the kernel version updated by Arch without ZFS support for that version (obviously not something that the Arch kernel package maintainers should care about, but it's a bit inconvenient).
zfs-dkms + "linux-lts" (longterm kernel) seems to work better than zfs-dkms + linux (stable kernel) if you're intending to do frequent system updates - I've been running this on several machines for years with no issues.
Yes, I have been using it for the bulk storage part of my desktop for the last 7 years, in a 3-drive RAID-Z configuration. I don't think any of the drives in the pool are the originals, as they have been swapped out for larger models as and when they started showing errors, with a scrub happening once a month. I have OS, swap, /home, /var, /tmp and ZFS log on an NVME drive separate - the ZFS pool is only used for the large stuff.
Separately, at work I managed to persuade them to configure our two new servers with ZFS, each with 168 drives for a total (after RAID) of 1.5PB each. Works like a charm, and is quite happy to saturate a 10Gb ethernet connection when accessing from another server, with transparent compression turned on. I would not hesitate to recommend ZFS to anyone.
I would worry a little about going above 90% space usage. ZFS slows down fairly dramatically when it starts running out of contiguous space and there is no defragmentation tool.
On the other hand it's not that hard to export and re-import a ZFS file system from a workstation or laptop to defragment manually - it's harder to do that with a NAS with lots of drives when you don't have a second set of drives just hanging around.
I have been using ZFS as root FS since the first release of ZFS on Linux (2013) on all my machines (laptops, NAS, server) and I can't imagine going back.
Checksums if the mobile hardware is damaged, snapshots if you delete something in your daily driver, native encryption if laptop is stolen, backup of work data via ZFS send if you have a ZFS server, compression to use limited disk space in laptops, errors can be fixed with copies=2, etc.
ZFS still gives benefits with one disk. Ignoring the data integrity stuff (which is better with ZFS even if you have no ECC), you also get snapshotting, potentially boot environments, you can backup your data to another machine using zfs-send, you get native encryption (including encrypted backups for free), highly performant compression, and (depending on your system) a nice memory cache algorithm.
I have, for more than two years, and my remote backup has been an encrypted ZFS for a year and a half. [1]
The scripts in that post are a little outdated, so if you would like to see updated ones, contact me [2].
The end result is a setup that:
* Has parallel upload to saturate my slow upload connection.
* Allows resume with `make -j<cores>`
* Allows me to backup things that are only on my server, such as my Gitea instance [3], to my desktop, thus having my server and desktop serve as backups to each other.
* Allows me to delete snapshots older than a certain number of snapshots (set to 60).
* Has automatic loading and unloading of encryption keys, to reduce the exposure time. (However, I wish raw sends on encrypted datasets worked. But it is nice that I can load the keys and download individual files instead, which I've already had to do.)
Perhaps I should add an update post to [1]. If there's enough interest, I will.
Edit: I should make clear that my ZFS is only my home directory. I do not use it as a root or boot filesystem because it runs off of a mirror of hard drives, and my root/boot drive is an SSD and can be destroyed without pain because all of my configs are in a repo in my home directory. Thus, it would be trivial to rebuild my system.
I installed Ubuntu 20.04 with ZFS for the root and user storage, with ZFS encryption, and frankly found the performance to be annoying. Keep in mind: I'm a huge ZFS fan. One of the packages I had installed, I forget which, made basically all apt sessions take much, much longer because of snapshotting, and my system load was basically always 1-3. I ended up reinstalling with ext4+LUKS when 22.04 came out. This is on a Dell XPS 15 with i7-10750H and 32GB RAM and "PC611 NVMe SK hynix 1TB" SSD.
It's a different layout of RAID, it's also not expandable vdev for now since it's made for enterprise. RAIDZ expansion is here https://github.com/openzfs/zfs/pull/12225
Fedora + BTRFS + Deja dup daily backups have been my default for months now, but I am tempted to give it a go to perhaps Ubuntu JJ + ZFS + Deja dup. Not that I have any issues with my current setup. It's just for the sake of, you know, try new things.