# Microbenchmark collection



## rootbert (Apr 9, 2021)

I regularly do some kind of microbenchmarks of various kinds. I wanted to share the outcomings, and maybe some of you have some results laying around, too. You are happily invited to contribute. Be it performance benchmarks of filesystem settings, database/network settings, tools, FreeBSD vs <other_os> etc. Hopefully this leads to a thread with an interesting discussion and a nice collection of data.

I was interested in how much faster zfs send/recv is compared to rsync. I used a 3-disk raidz1 (atime=off, compression=lz4) with 3 disks Toshiba MG07ACA12TE connected internally via SATA-3 (600 MB/s) to copy our code repository (7,297,671 files, ~373 GB) to a 3-disk raidz1 (atime=off, compression=lz4) with (atime=off, compression=lz4) connected to external drives via USB 3.0 (400 MB/s). ARC was limited to 8GB and before I did warmup the cache via "find /data/code". System has 2xnvme as root, Ryzen 7 3700X CPU with 64GB ECC RAM.

 `rsync -a /data/code /ext/code-rsync1`: real 475m35.261s; user 5m57.723s; sys 28m35.892s
 `zfs send data/code@snap | zfs recv ext/code-zfssync1`: real 86m17.891s; user 0m0.000s; sys 7m54.837s
 `zfs send data/code@snap | mbuffer -s 128k -m 2G -o - | zfs recv ext/code-mbuffer2`: real 87m44.900s; user 0m15.359s; sys 11m7.539s
 `zfs send data/code@snap | mbuffer -s 128m -m 2G -o - | zfs recv ext/code-mbuffer3`: real 82m9.039s; user 0m10.456s; sys 10m22.156s


----------



## IT IGP (May 17, 2021)

just stumbled on this thread looking for ZFS checksum microbenchmark data.

i am running a TrueNAS 12.0 but i don't have /proc there so i cannot access /proc/spl/kstat/zfs  - maybe you see some numbers there?
(adding /proc to /etc/fstab allows mounting /proc, but i do not see  /proc/spl/kstat/zfs in there)
see also https://openzfs.github.io/openzfs-docs/Basic Concepts/Checksums.html

i was mainly interested in flechter4 vs sha-512 given the recent speedup of the latter


----------



## rootbert (Aug 12, 2021)

will try to benchmark this when I have time.

another rsync vs zfs send/recv benchmark:
~ 146 GB, 248118 files of jpgs with lower than 3.6 MP resolution; target is an encrypted zfs dataset.
rsync from one external 7.2k disk to the other: real    104m34.104s, user    1m33.669s, sys    5m52.857s
zfs send/recv: real    60m9.500s, user    0m0.000s, sys    1m11.733s


----------



## mer (Aug 12, 2021)

I think the time benefit of ZFS send/recv winds up in the incrementals.  An initial snapshot and send/recv is going to take the longest because it has the most data. 
From then on incremental snapshots and send/recv is only sending the changes.

Why is that important?  I believe rsync processes things differently, as in still at least walks the full source to compare against destination.


----------



## rootbert (Jul 22, 2022)

did another microbenchmark - mostly Linux but also Linux in bhyve. I am planning to migrate a mailserver, converting dovecots mdbox to sdbox mail format. for the readers not involved in mailserver administration with dovecot, simply put: mdbox is like a container format with a fixed size and an index - thus suboptimal when using zfs since when adding a 1k mail to a 31MB file the whole file will be rewritten. sdbox is like one compressed file per mail and an index per folder. The benchmark compares different scenarios when converting about 61500 mails (~6.5GB) from mdbox to sdbox. the command "time (doveadm backup -u testuser -1 sdbox:/var/lib/testuser/mail)" was used, the target always residing on the same disks as the source.


original server, bare metal Intel i7-2600 CPU @ 3.40GHz with 32GB RAM and 2x3TB sata HDDs WD Red. Storage layer: LVM -> LUKS -> ext4 (source and target): *113m24.062s*
original server, but with storage layers: LVM -> LUKS -> zfs (version, 2.1.5, source and target): *32m11.332s* (Linus Torvalds has no idea of ZFS's performance ;-)
new server, Intel Xeon CPU E3-1275 v5 @ 3.60GHz with 64GB ECC RAM, 2x4TB Seagate Enterprise HDD, running FreeBSD 13.1 with bhyve. The VM resides on a geli-encrypted zpool and uses virtio, inside the VM: zfs (source and target).* 134m59.116s*
new server, but with 2xNVMe,; inside the VM: zfs: *16m15.386s*

benchmarks on native FreeBSD to follow


----------



## hardworkingnewbie (Jul 22, 2022)

rootbert said:


> did another microbenchmark - mostly Linux but also Linux in bhyve. I am planning to migrate a mailserver, converting dovecots mdbox to sdbox mail format. for the readers not involved in mailserver administration with dovecot, simply put: mdbox is like a container format with a fixed size and an index - thus suboptimal when using zfs since when adding a 1k mail to a 31MB file the whole file will be rewritten. sdbox is like one compressed file per mail and an index per folder. The benchmark compares different scenarios when converting about 61500 mails (~6.5GB) from mdbox to sdbox. the command "time (doveadm backup -u testuser -1 sdbox:/var/lib/testuser/mail)" was used, the target always residing on the same disks as the source.


Mdbox is about high performance. When just running your own little IMAP it does not matter much, but its advantages really do shine if you need to scale your installation a lot up. 

It's difference to generate backups let's say for 10.000 active accounts using Maildir every night, or using Mdbox. Since Mdbox uses less files, backup generation becomes much quicker due to having so much less files to walk through the file system.


----------



## rootbert (Jul 24, 2022)

mdbox is nice on conventional filesystems, however, my performance tests on servers with zfs showed massive performance improvements when choosing sdbox - for both usage (which was my primary concern) and backup. I have only done tests on servers with spinning rust disks. mdbox on zfs performed in fact so bad that after a few rounds I decided to cancel mdbox perf tests and instead focussed on benchmarking the compression algorithms of zfs and dovecot where I found out that the best performance/compression ratio for my data was with bot zfs AND dovecot storage set to zstd-9.


----------

