# ZFS on Linux



## achix (Oct 19, 2022)

Hello
some guys are concerned at work about stability, reliability of ZFS on Linux.
What's your take my dear FreeBSD gurus?


----------



## SirDice (Oct 19, 2022)

Maybe you should ask this on a Linux forum?


----------



## Alain De Vos (Oct 19, 2022)

I dual boot freesd-on-zfs , gentoo-linux-on-ext4.
When i boot gentoo-linux i mount the freebsd-zpools & this works fine and perfect.
[ Personally i would not put the linux-kernel on zfs. This can be painfull booting ].
But zfs works fine as out-of-linux-kernel-source-tree.
Openzfs shares a source-tree between freebsd & linux , so this guaranteed to work.


----------



## Crivens (Oct 19, 2022)

I would be more concerned with the stability of the user land than ZFS here. How long is your installation going to run?


----------



## Alain De Vos (Oct 19, 2022)

Why is openzfs not part of base. Would it be not more logic ?


----------



## SirDice (Oct 19, 2022)

Alain De Vos said:


> Why is openzfs not part of base.


It is on 13.0 and above.


----------



## achix (Oct 19, 2022)

SirDice said:


> Maybe you should ask this on a Linux forum?



Because I know you guys and I trust you (more).


----------



## achix (Oct 19, 2022)

Crivens said:


> I would be more concerned with the stability of the user land than ZFS here. How long is your installation going to run?



this is about an installation on vessels (ships) traditionally aimed to run at about 10 years. Now we are moving to proxmox virtualization and those guys fully support and endorse ZFS.
However some ppl in the company seem concerned since their expertise lies on EXT4/ mdadm, etc.

I use ZFS in my current FreeBSD machine (13.1-RELEASE) and its many levels superior to anything I had worked with in the past.


----------



## Alain De Vos (Oct 19, 2022)

Example.
With zfs i take every 15 minutes an incremental snapshot of my user-home-directory.
So i never lose data older then 15 minutes or can restore it to any place each 15 minutes before.
Note: JFS,XFS&EXT4 are also not bad filesystems. Personally i find zfs superior.


----------



## Erichans (Oct 19, 2022)

One fundamental difference between ZFS on Linux and ZFS as part of FreeBSD is that the ZFS part of FreeBSD has its ZFS development in sync with its kernel development. That is one consequence of having a FreeBSD that is developed as a whole where a base install is a complete OS and a base userland that is being achieved by one team. Linux is just a kernel and ZFS is denied access to a shared kernel development process (ask Linus for reasons o.a. such as the stated differences in licence models). Because of the ZFS kernel module on Linux it is dependant on its kernel ABI and therefore the OpenZFS team has to play catch-up when the Linux kernel ABI changes. Linux _distros_ have to combine a chosen Linux kernel and an accompanying ZFS.


----------



## Alain De Vos (Oct 19, 2022)

The same thing can be seen with compilers.
linux goes good with gcc,
while freebsd goes good with clang.
This is related to the license.


----------



## forquare (Oct 19, 2022)

I've been using ZFS on my Arch desktop for a little over two years I think.
Generally everything has been fine, current setup is XFS on a NVMe for boot, one zpool made up of two SSDs (mirrored) for home directories, a second zpool made up of two larger HDDs (also mirrored) for media files.
I've been using this setup for about a year and a half.

Before that I was using the two SSDs as a boot disk as well as home directories, the OS partition on each was part of a zpool which mirrored both partitions.  I'm pretty sure back then I used systemd-boot to boot into Arch installed on ZFS.

What issues have I had?  Maybe twice the ZFS packages have changed names and caused some weirdness - ZFS is not "built into" Arch in the same way as it is in something like Ubuntu, so that may not happen there.
Oh, and once I went to upgrade the kernel and the ZFS kernel modules weren't ready yet - I've had this problem way more with my graphics driver.  Again, probably not so much of an issue on the more "managed" distros.

Why did I stop using ZFS on root?  Mainly because it didn't have the same advantages as ZFS on root on FreeBSD (e.g. no boot environments), I was afraid it might break (not that anything ever cropped up to suggest it would), and I was looking to make my setup a little more streamlined - no backups/snapshots of the root disk, if it becomes borked just reinstall + run ansible to set everything back up.

Overall, ZFS on Linux has been fine.  I do similarly to Alain De Vos where my home directory gets snapshotted every 5 minutes and other file systems get hourly snapshots.


----------



## achix (Oct 19, 2022)

Any opinion on the Linus Torvalds stmt back in 2020 : "do not use ZFS"?


----------



## Alain De Vos (Oct 19, 2022)

Yes, each priest speaks for his church.
Which were the arguments ?


----------



## achix (Oct 19, 2022)

Alain De Vos said:


> Yes, each priest speaks for his church.
> Which were the arguments ?



Sorry no idea. Its just an argument by a colleague that Linus said so.


----------



## Alain De Vos (Oct 19, 2022)

Don't forget to tell me when you have.
Do you have link to his speach,it makes me curious.


----------



## Alain De Vos (Oct 19, 2022)

I found this but the argument is not technical,








						Linus Torvalds Doesn't Recommend Using ZFS on Linux
					

Linux creator Linus Torvalds is skeptical of ZFS open source licensing and doesn't recommend using it on Linux.




					itsfoss.com
				




Thorvalds refers explicit to Oracle. Which is his right. 
But even without oracle zfs will continue to live. Because it contains some good ideas.


----------



## mer (Oct 19, 2022)

My opinions, take for what it's worth.

The historical "Linux" stance on ZFS has revolved around the license and quite a bit of "not invented here".
The license aspect means it will never be a "proper" filesystem in the kernel on Linux, so ZFS has to use the publicly available module interfaces.
OpenZFS is can't be claimed by Oracle (I'm not a lawyer so don't take this as gospel), so any arguments about Oracle don't really hold weight.

From a technical standpoint, OpenZFS on linux is the same as what FreeBSD 13.x on up uses.  I have not had any issues in daily use with any version of ZFS on FreeBSD.

In your specific use case there's not enough to say which filesystem would be better.


----------



## Erichans (Oct 19, 2022)

Additionally, when reading about ZFS and Linux, please be aware of the historic development and where ZFS—as part of the OpenZFS initiative—is now: OpenZFS & its code base history. Perhaps somewhat confusing ZoL (as an abbreviation for ZFS on Linux) is no more. As of FreeBSD 13.0 (I'm unfamiliar with any specific Linux distribution) ZFS means OpenZFS. Linux and FreeBSD indeed now rely on the same unified OpenZFS code repository and within the OpenZFS initiative both are supported as equal citizens.


----------



## Crivens (Oct 19, 2022)

achix said:


> this is about an installation on vessels (ships) traditionally aimed to run at about 10 years. Now we are moving to proxmox virtualization and those guys fully support and endorse ZFS.


One argument I have is that keeping a system running, updated, patched for that time is critical. Boot environments are a selling point here. Snapshots as well. Minimal downtime for botched updates. I ean we all should have heard the "success" the US navy had with NT, yes?


----------



## _martin (Oct 19, 2022)

My 2c: this is way too specific question and requires background information about the $job and what task it's going to do. Is it actually being shipped (pun intended) on ships at sea? If so I'd say simpler the better. I have no idea what is the upgrade strategy on those devices. From personal experience many specific devices are usually set and forget.

Are those only VMs or physical HW? I'd stick to ext4 in either case probably. Given you have robust recovery solution at place (such as rear,etc.).


----------



## Eric A. Borisch (Oct 19, 2022)

ZFS works well on linux; sometimes the packaging (getting kmods updated for the latest kernel) trips up smooth upgrades, but that’s the worst I’ve encountered in many years of use.

If you need to run Linux, and you’re comfortable with (and would like the benefits of) ZFS, there’s no reason to avoid it.


----------



## Alain De Vos (Oct 19, 2022)

On should make a clear distinction between boot-on-zfs & root-on-zfs.
I don't consider linux stable enough for boot-on-zfs, i.e. the kernel itself on zfs.
I prefer ext4 or ext2 for /boot
For root, old jfs,ext4,xfs,zfs are good options.

That is for linux, but even on this freebsd-desktop on which i write this message the bootloader is on ufs &  root filesystem is on zfs. Having the bootloader on a simple filesystem (ufs) makes it more robust I think. This is my personal take but mileage may vary.


----------



## achix (Oct 19, 2022)

mer said:


> My opinions, take for what it's worth.
> 
> The historical "Linux" stance on ZFS has revolved around the license and quite a bit of "not invented here".
> The license aspect means it will never be a "proper" filesystem in the kernel on Linux, so ZFS has to use the publicly available module interfaces.
> ...



ok, given data :
a) Linus said in 2020 ZFS is bad dont use, the average linux person will do as advised
b) Proxmox officially provide/support ONLY one FS : ZFS
c) We will have lousy net connectivity to the server

so, do we placate our linux ppl and try a custom installation on EXT4 ? or do we placate the proxmox ppl and try the default ZFS ?


----------



## achix (Oct 19, 2022)

Crivens said:


> One argument I have is that keeping a system running, updated, patched for that time is critical. Boot environments are a selling point here. Snapshots as well. Minimal downtime for botched updates. I ean we all should have heard the "success" the US navy had with NT, yes?


our current images run since 2005... unpatched , postgresql 8.4, I dont even remember debian versions or kernels. Those are not connected to the net, so the no1 aim is to just work. No corruption. Recovery is not so cheap in terms of time. Anyways, all those aspects are subject to redesign. We are in the middle of this huge upgrade.


----------



## Alain De Vos (Oct 19, 2022)

Personally I would place boot&kernel on ext2 or ext4.
And place data on zfs, because of the easy of backup & snapshot
For /,root i would be open to any option, ext4,zfs,xfs,jfs,...
Note, a bad internet connection never give a good "quality of service". That looks like a bad choice.
[My limited take on this problem]


----------



## achix (Oct 19, 2022)

_martin said:


> My 2c: this is way too specific question and requires background information about the $job and what task it's going to do. Is it actually being shipped (pun intended) on ships at sea? If so I'd say simpler the better. I have no idea what is the upgrade strategy on those devices. From personal experience many specific devices are usually set and forget.
> 
> Are those only VMs or physical HW? I'd stick to ext4 in either case probably. Given you have robust recovery solution at place (such as rear,etc.).


proxomox as you may have read, does not officially support EXT4, mdamd, etc. Only ZFS. This is an important point.

Now, yep the new hardware will be shipped to the ships by a smaller ship containing maybe some sheep hahaha kidding. Yep. New hardware running proxmox.
The original setup was based on bare metal. Then came some virtualization solution of which I was kept ... outside any decision (dangerous to leave the DBA out of all this), now we are thinking of dumping this in favor of our own virtualization solution based on proxmox.

As I wrote EXT4 is fine for me, never failed me (in contrary to XFS), but proxmox says this is not supported. + there is no comparison between the two FS's really.


----------



## achix (Oct 19, 2022)

Eric A. Borisch said:


> ZFS works well on linux; sometimes the packaging (getting kmods updated for the latest kernel) trips up smooth upgrades, but that’s the worst I’ve encountered in many years of use.
> 
> If you need to run Linux, and you’re comfortable with (and would like the benefits of) ZFS, there’s no reason to avoid it.



Great point.


----------



## mer (Oct 19, 2022)

Recovery.  I think that is a key requirement.
Assuming everything is "created" correctly, ZFS and boot environments would make recovery very easy.


----------



## achix (Oct 19, 2022)

mer said:


> Recovery.  I think that is a key requirement.
> Assuming everything is "created" correctly, ZFS and boot environments would make recovery very easy.


great point, now I get what Boot Environment means.


----------



## Erichans (Oct 19, 2022)

achix said:


> our current images run since 2005... unpatched , postgresql 8.4, I dont even remember debian versions or kernels. Those are not connected to the net, so the no1 aim is to just work. No corruption. Recovery is not so cheap in terms of time. Anyways, all those aspects are subject to redesign. We are in the middle of this huge upgrade.





mer said:


> Recovery.  I think that is a key requirement.
> Assuming everything is "created" correctly, ZFS and boot environments would make recovery very easy.
> 
> 
> ...


Perhaps have a closer look at Boot Environments and especially their relation to recovery; for example Managing Boot Environments and Let’s Talk OpenZFS Snapshots; also have a look at vermaden's message. For a deeper ZFS dive I recommend the two ZFS books: FreeBSD Development: Books, Papers, Slides

Boot Environments (BEs) are a feature enabled by ZFS. From an implementation standpoint they are "almost free of charge"; ZFS does the heavy lifting. BEs are (very) useful for managing updates: patches, minor and major version updates of FreeBSD and software packages. Those updates are related to system environments and system changes, hence the name Boot Environments: if an update does not function as intended (extreme case: does not boot) you can easily go back to a previous Boot Environment: the system as it was before the update. BEs are not intended for recovery in case of corrupt (user) data. Currently there are no Boot Environment options for Linux. 

When there is data corruption to such an extent that ZFS cannot correct the errors anymore and you have lost a ZFS pool, there are basically no tools available to recover a failed pool. That means all data in that pool is lost and you'll have to resort to backups. This is a kind of a consequence of ZFS taking very good care of your data (depending on the chosen redundancy setup) but if, in the end, there is a non-recoverable error (even in just one file), you will have lost all data in the pool where that file belongs to.

As you have described your situation so far the environment is an all Linux environment, so please be aware of what properties and additional tools mentioned belong exclusively to a FreeBSD ZFS environment and consider SirDice's suggestion*.

___
* unless you are in a position to consider a switch from Linux to FreeBSD: no proxmox on FreeBSD but, we do have _jails_ as a lightweight OS-level virtualisation mechanism


----------



## Alain De Vos (Oct 19, 2022)

I don't put the linux-bootloader on zfs.
I don't put the linux-kernel on zfs.
It's just too dangerous.
This means i use multiple file-systems.


----------



## Jose (Oct 19, 2022)

achix said:


> Any opinion on the Linus Torvalds stmt back in 2020 : "do not use ZFS"?


Yes, Linus threw a tantrum 'cause the Openzfs guys wouldn't give him code under terms he likes so he decided to spread FUD about ZFS.

The statement "(ZFS) was always more of a buzzword than anything else..." is so patently and obviously absurd that I can't believe someone as smart as Linus uttered it in good faith. ZFS is more than a filesystem. It's a volume manager and software RAID layer, and it makes the Linux md and LVM crapola look primitive.

Here's a longer take on the matter:








						Linus Torvalds says “Don’t use ZFS”—but doesn’t seem to understand it
					

Linus should avoid authoritative statements about projects he's unfamiliar with.




					arstechnica.com


----------



## ct85711 (Oct 19, 2022)

For my linux machine, I've been running root on zfs for a while without any issues on Debian.  I originally tried to also boot from zfs, but grub's zfs support is sorely painful that I moved the /boot to an ext3 partition.


----------



## Alain De Vos (Oct 19, 2022)

It's not only grub that is the problem. My experience is zfs works fine once the linux-kernel is loaded.


----------



## achix (Oct 20, 2022)

Erichans said:


> Perhaps have a closer look at Boot Environments and especially their relation to recovery; for example Managing Boot Environments and Let’s Talk OpenZFS Snapshots; also have a look at vermaden's message. For a deeper ZFS dive I recommend the two ZFS books: FreeBSD Development: Books, Papers, Slides
> 
> Boot Environments (BEs) are a feature enabled by ZFS. From an implementation standpoint they are "almost free of charge"; ZFS does the heavy lifting. BEs are (very) useful for managing updates: patches, minor and major version updates of FreeBSD and software packages. Those updates are related to system environments and system changes, hence the name Boot Environments: if an update does not function as intended (extreme case: does not boot) you can easily go back to a previous Boot Environment: the system as is was before the update. BEs are not intended for recovery in case of corrupt (user) data. Currently there are no Boot Environment options for Linux.
> 
> ...



Again great insight and real no-BS help which justifies why I came here and not in a linux place. I have done some things with vm/bhyve and jails of course. However distribution of new images in jails are kinda hard/not a thing, + all our ship/vessel system is based on Linux and some archaic tools that we still have to run (uucp). Yep uucp, a tool that was obsoleted in my school when I was a freshman back in 1986, go figure.


----------



## Crivens (Oct 20, 2022)

For what it is worth, we use proxmox at $DAYJOB. I have not had much to do with it, even if we have several virtual servers running in it. Developemt servers need a snapshot/rollback from time to time. And I had not much contact with it because it simply works without a glitch up to now. Maybe it's only me, but they look like they know their stuff.


----------



## hardworkingnewbie (Oct 20, 2022)

achix said:


> Any opinion on the Linus Torvalds stmt back in 2020 : "do not use ZFS"?


Torvalds was speaking about the license issues this might cause. If it really comes to the quality of file systems he is not the guy to listen to, because file system development was never done by him. I would then have a look at Dave Chinner (XFS in the Linux kernel) or Theodore T'so (father of ext2/3/4). 

By the way the default setup which Proxmox uses is ext4+LVM (have a look at this: https://pve.proxmox.com/wiki/Installation), not ZFS. You can choose ZFS though. LVM snapshots are known to work, but to be slow as well, so if you really want snapshots obviously ZFS is the way to go.


----------



## Erichans (Oct 20, 2022)

Proxmox-Wiki-Installation:


> Proxmox VE can be installed on ZFS. As ZFS offers several software RAID levels, this is an option for systems that don’t have a hardware RAID controller. The target disks must be selected in the Options dialog. More ZFS specific settings can be changed under Advanced Options (see below).
> 
> [Warning] 	ZFS on top of any hardware RAID is not supported and can result in data loss.


ZFS offers functionality and data protection far beyond that of any hardware RAID. (ZFS on top of hardware RAID is a definite no.) 

An additional hardware RAID consideration to keep in mind when not relying on ZFS. Current hardware RAID controllers are likely not the hardware RAID controllers you'd want or expect.  Hardware RAID controllers of the past in their _server_ hardware RAID controller incarnation had real protection mechanisms against some data corruption but the landscape has changed: Hardware Raid is Dead and is a Bad Idea in 2022


----------



## Jose (Oct 20, 2022)

Erichans said:


> Hardware RAID controllers of the past in their _server_ hardware RAID controller incarnation had real protection mechanisms against some data corruption but the landscape has changed: Hardware Raid is Dead and is a Bad Idea in 2022


This is good to know. A good rule of thumb in the past was that if your hardware RAID controller cost lest than $1000, you probably shouldn't use it. Oh, you're feeling lucky? Data corruption is your door prize. Yup, all your backups for the past month are corrupt too. Congratulations!


----------



## Alain De Vos (Oct 20, 2022)

I think you can do everything but hardware-raid & zfs just dont' go good together.
I wonder which filesystems are good for hardware-raid.


----------



## achix (Oct 20, 2022)

hardworkingnewbie said:


> Torvalds was speaking about the license issues this might cause. If it really comes to the quality of file systems he is not the guy to listen to, because file system development was never done by him. I would then have a look at Dave Chinner (XFS in the Linux kernel) or Theodore T'so (father of ext2/3/4).
> 
> By the way the default setup which Proxmox uses is ext4+LVM (have a look at this: https://pve.proxmox.com/wiki/Installation), not ZFS. You can choose ZFS though. LVM snapshots are known to work, but to be slow as well, so if you really want snapshots obviously ZFS is the way to go.


thnx
no, they support only ZFS : https://forum.proxmox.com/threads/z...raid-stability-and-reliability-of-zfs.116871/


----------



## Erichans (Oct 28, 2022)

tweet on UBUNTU 21.10 about Ubuntu zfs-linux package--bug 1906476



> PSA: DO NOT UPGRADE TO UBUNTU 21.10 IF YOU USE ZFS. DO NOT create a new installation of Ubuntu 21.10 with ZFS either.
> 
> It *will* corrupt your filesystem irreparably. There’s a warning in the release notes – but only after 2800 words >_>





Erichans said:


> As you have described your situation so far the environment is an all Linux environment, [...]


This is now in the past (Ubuntu 21.10 has reached EoL); I hope and expect that Ubuntu has learned from this and that this is a once-in-a-lifetime thing. However, it shows the possibility of problems that can arise when the ZFS part cannot be developed in sync with the kernel of an OS and the responsibility of integrating/matching ZFS with a current kernel is shifted to the team that constructs a specific OS distribution.


----------



## hardworkingnewbie (Oct 28, 2022)

achix said:


> thnx
> no, they support only ZFS : https://forum.proxmox.com/threads/z...raid-stability-and-reliability-of-zfs.116871/


Wrong. As you can see here in the most recent iteration of their own documentation it says this below:


_After selecting *Install Proxmox VE* and accepting the EULA, the prompt to select the target hard disk(s) will appear. The Options button opens the dialog to select the target file system.

*The default file system is ext4.* The Logical Volume Manager (LVM) is used when ext4 or xfs is selected. Additional options to restrict LVM space can also be set (see below)._


----------



## Alain De Vos (Oct 28, 2022)

So you can put root-filesystem&kernel on zfs ; and bootloader on a "simple filesystem" ? This is ideal.


----------



## dnb (Oct 31, 2022)

Alain De Vos said:


> I dual boot freesd-on-zfs , gentoo-linux-on-ext4.
> When i boot gentoo-linux i mount the freebsd-zpools & this works fine and perfect.


Fantastic, I used to think that it's better not to do this just in case due to compatibility issues ) Let's say when upgrading the system, we sometimes have to do a zpool upgrade. How about this in this case?

Would that zpool be created on FreeBSD? And how: just *zpool create mypool* or with some special zpool properties used?


----------



## dnb (Oct 31, 2022)

Alain De Vos said:


> but even on this freebsd-desktop on which i write this message the bootloader is on ufs &  root filesystem is on zfs. *Having the bootloader on a simple filesystem (ufs) makes it more robust* I think. This is my personal take but mileage may vary.


What if I use zfsboot to root-on-zfs on FreeBSD? I just set up the system like this:

```
zpool create -m none mypool /dev/diskid/....
zfs create -o mountpoint=/mnt mypool/rootfs
cd /mnt
wget https://download.freebsd.org/ftp/releases/amd64/13.0-RELEASE/base.txz
wget https://download.freebsd.org/ftp/releases/amd64/13.0-RELEASE/kernel.txz
[...]
zpool set bootfs=mypool/rootfs zpool
[...]

# install zfsboot
dd if=/boot/zfsboot of=/dev/diskid/.... count=1
dd if=/boot/zfsboot of=/dev/diskid/.... iseek=1 oseek=1024

# and in /boot/loader.conf
vfs.root.mountfrom="zfs:mypool/rootfs"
zfs_load="YES"
```

So I don't have partitions at all. Why is this way worse? Less stable system? Need to use Legacy BIOS? (zfsboot - bootcode for ZFS on BIOS-based computers). And the result is slightly lower performance? Theoretically, I can also import this zpool when booting to Gentoo.

It's just that the zfsboot method allows me to put the entire disk under ZFS management (although this is not currently necessary). But there is another aspect that this method is simple and beautiful: we use only ZFS and it manages the entire disk, and /etc/fstab can be either completely empty or intended for various small file systems like tmpfs. What do you think about zfsboot?


----------



## Alain De Vos (Oct 31, 2022)

There is nothing wrong if it works good for you.


----------

