# FreeBSD 13.1-RELEASE i386 and ZFS



## Simba7 (Jun 3, 2022)

I have noticed that the ZFS service on FreeBSD 13.1-RELEASE crashes for some reason. When I did a fresh install on a system with ~4GB of RAM on a i386 platform (dual P3), the install went fine. But when I rebooted the system I noticed none of my other datasets would mount. After looking at the log, ZFS seems to crash early during the boot process. This has happened on two systems (a ML370 w/4GB RAM and a DL360 w/2GB of RAM). It works perfectly fine on FreeBSD 12.3-RELEASE.

After work, I plan on trying it again on a Pentium 4 w/2GB of RAM and see if the results are the same. Any idea what would be causing this?


----------



## Vull (Jun 4, 2022)

I'm thinking ZFS on low memory and/or i386 systems might be quite a bit of overkill. I tried it once on an 8 GB i386 and didn't get good results either. Had I done more careful advanced research, I probably would not have attempted it.



> 20.6.2.1. Memory​As a lower value, the total system memory should be at least one gigabyte. The amount of recommended RAM depends upon the size of the pool and which features ZFS uses. A general rule of thumb is 1 GB of RAM for every 1 TB of storage. If using the deduplication feature, a general rule of thumb is 5 GB of RAM per TB of storage to deduplicate. While some users use ZFS with less RAM, systems under heavy load may panic due to memory exhaustion. ZFS may require further tuning for systems with less than the recommended RAM requirements.
> 20.6.2.2. Kernel Configuration​Due to the address space limitations of the i386™ platform, ZFS users on the i386™ architecture must add this option to a custom kernel configuration file, rebuild the kernel, and reboot:
> options        KVA_PAGES=512











						Chapter 20. The Z File System (ZFS)
					

ZFS is an advanced file system designed to solve major problems found in previous storage subsystem software




					docs.freebsd.org
				












						Best ZFS tuning for FreeBSD 10.2 on an i386 system with 4GB of physical RAM
					

Hi! Last afternoon I was trying to install FreeBSD 10.2 RELEASE i386 on my PC, on a ZFS file system. I got the warning saying that recommended minimum kmem_size is 512MB, so I could expect unstable behavior (like it's shown at the picture below). Then, I've been reading that ZFS on i386 systems...




					forums.freebsd.org


----------



## astyle (Jun 4, 2022)

i386 is a 32-bit architecture, so it can't see more than 4 GB of RAM - theoretically. Realistically speaking, only 3.86 GB of RAM is even usable on i386. Oh, and a lot of toolchains these days are optimized for 64-bit architectures like amd64 and aarch64. So I'd really recommend to OP to upgrade the hardware to something more recent.


----------



## bakul (Jun 4, 2022)

I think you will have to set some loader.conf variables to adjust some limits. Something like


```
vm.kmem_size="2048M"
vfs.zfs.arc_min "512M"
vfs.zfs.arc_max=“512M”
```
 
But this is from my faulty memory. Numbers used just as an example. There may even be other tunables.
Websearch for discussions on running zfs on 32bit machines and add the above three variable names. The zfs tuning guide in the handbook seems complicated and possibly obsolete.


----------



## grahamperrin@ (Jun 4, 2022)

Simba7 said:


> … FreeBSD 13.1-RELEASE … looking at the log, ZFS seems to crash early during the boot process. … w/4GB RAM … fine on FreeBSD 12.3-RELEASE. …



Maybe a bug, although I can't find a match in Bugzilla.

Please, can you share the result of a probe? 

`pkg install sysutils/hw-probe sysutils/pciutils sysutils/usbutils`
`hw-probe -all -upload`


----------



## CyberCr33p (Jun 5, 2022)

Maybe this is related?






						263473 – ZFS drives fail to mount datasets when rebooting - 13.1-RC4
					






					bugs.freebsd.org


----------



## PMc (Jun 5, 2022)

I did run ZFS on a dual P3 for a long time, and it didn't make me happy. It is not that there isn't enough physmem - this can be treated, and today I have ZFS running on nodes with less than 1G physmem (but 64bit) and that works.
I am quite certan that there is either some cornercase mishandling of 32bit variables or problems with the limited vmem. I did find and had fixed one or two cases, but there are likely more and we don't know which of them crawled back in with the rel-13 version. This comes mainly from a lack of substantial testing. I finally figured that the memory wasn't really good either (although ECC, apparently temperature related) and gave up on the platform in 2019. I then used a consumer Ivybridge with 4G ram, with no problems at all - so the 4G in itself is not the issue.

Edit: if Your problem is just that it doesn't mount on boot, that is probably not a crash, but a design change. The mounting during boot works differently with rel-13, and maybe something with Your rc files is in a bit of disorder. Do some testing in single-user. (In my experience my pools now need the `-f` flag when I need to mount them in single-user.)


----------



## Simba7 (Jun 9, 2022)

It's not a memory problem, or I'd be running out of memory in 12.3-RELEASE as well. This only happens with 13.1-RELEASE as far as I can tell.
As for the ML370, it has a dual P3 @1GHz, 3GB of RAM, 6x300GB of storage (3x HW R5 w/SW stripe to bond them [limitation of controller]) and it's running a Synchronet BBS , so..


----------



## Simba7 (Jun 9, 2022)

CyberCr33p said:


> Maybe this is related?
> 
> 
> 
> ...


That.. looks familiar.. I'm going to keep an eye on that thread.


----------



## astyle (Jun 9, 2022)

Simba7 said:


> That.. looks familiar.. I'm going to keep an eye on that thread.


Your sig tells me you're no stranger to amd64... but a _router _with a Xeon and 64 GB of RAM??? Tell you what. Make the 32-bit i386 assume the role of router, and install 13.1-RELEASE on the Xeon, you'll get MUCH better results that way. Sometimes, you gotta realize that certain hardware specs just fit certain tasks better. There's a reason that Xeons are aimed at the server market.


----------



## Simba7 (Jun 10, 2022)

astyle said:


> Your sig tells me you're no stranger to amd64... but a _router _with a Xeon and 64 GB of RAM??? Tell you what. Make the 32-bit i386 assume the role of router, and install 13.1-RELEASE on the Xeon, you'll get MUCH better results that way. Sometimes, you gotta realize that certain hardware specs just fit certain tasks better. There's a reason that Xeons are aimed at the server market.


This router also has 4x10GbE ports along with 8x1GbE ports and is running OPNsense. I went with 64GB because I have a tendency to max things out. Overkill? Yes. Will it work for the next 10-15 years? Yes.

As for the Compaq ML370G1, this is operating a Synchronet BBS, so the Supermicro system would be severe overkill for this type of project.

As for FreeBSD 12.3-RELEASE, it's operating just fine on it with 3GB of RAM.


----------



## astyle (Jun 10, 2022)

Simba7 said:


> This router also has 4x10GbE ports along with 8x1GbE ports and is running OPNsense. I went with 64GB because I have a tendency to max things out. Overkill? Yes. Will it work for the next 10-15 years? Yes.
> 
> As for the Compaq ML370G1, this is operating a Synchronet BBS, so the Supermicro system would be severe overkill for this type of project.
> 
> As for FreeBSD 12.3-RELEASE, it's operating just fine on it with 3GB of RAM.


OK, I get the logic, but I still think that 3 GB of RAM is just too stingy if you also want ZFS. That kind of tradeoffs make sense in the embedded programming market (like phones and smartwatches or smart toasters that tell you to go buy milk and eggs), but not on  devices where using ZFS is even a consideration.

And y'know, even phones these days come with 6 GB of RAM and 64-bit SnapDragon processors...


----------

