# Benchmarks: FreeBSD 13 vs. NetBSD 9.2 vs. OpenBSD 7 vs. DragonFlyBSD 6 vs. Linux - Phoronix



## grahamperrin@ (Dec 12, 2021)

February 2021 (FreeBSD 13 BETA):









						Phoronix: FreeBSD 13 BETA Benchmarks - Performance Is Much Better
					

Article:  https://www.phoronix.com/scan.php?page=article&item=freebsd-13-beta1  Straight to the benchmark graphs (scroll down):  https://openbenchmarking.org/result/2102103-HA-FREEBSD0515&imw=1&sgm=1&imw=1




					forums.freebsd.org
				




December 2021 (FreeBSD 13.0-RELEASE):






						Benchmarks: FreeBSD 13 vs. NetBSD 9.2 vs. OpenBSD 7 vs. DragonFlyBSD 6 vs. Linux - Phoronix
					






					www.phoronix.com
				




FreeBSD 13.0 only <https://openbenchmarking.org/result/2112039-TJ-BSDLINUX675&sgm=1&sor&&sgm=1&rmm=CentOS+Linux+8,Clear+Linux+35320,DragonFlyBSD+6.0.1,NetBSD+9.2,OpenBSD+7.0,Ubuntu+20.04.3+LTS,Ubuntu+21.10&ppt=D>
Discussions

<https://www.phoronix.com/forums/forum/software/bsd-mac-os-x-hurd-others/1295700>
<https://lists.freebsd.org/archives/freebsd-current/2021-December/001171.html>
<https://old.reddit.com/r/freebsd/duplicates/reedt0/-/>
<https://old.reddit.com/r/freebsd/comments/rdwcmu/-/ho719ul/>
<https://news.ycombinator.com/item?id=29510211>
Zstandard compression

FreeBSD 13.0 (from the article): <https://openbenchmarking.org/result/2112039-TJ-BSDLINUX675&sgm=1&sor&&sgm=1&rmm=CentOS+Linux+8,Clear+Linux+35320,DragonFlyBSD+6.0.1,NetBSD+9.2,OpenBSD+7.0,Ubuntu+20.04.3+LTS,Ubuntu+21.10&ppt=D#r-3384cfbd5b4a7dac9491a6763fa08dddaa48409f>

At least for compression level 8, superior results with vastly inferior hardware: <https://openbenchmarking.org/result/2112127-TJ-2112123TJ48>

Cross-reference <https://www.phoronix.com/forums/for...s-x-hurd-others/1295700?p=1296018#post1296018>


----------



## Alain De Vos (Dec 12, 2021)

What a pile of crap.

I always miss the following test :
-Perform high intensive database operations
-Pull the power cord
-Check how much data is LOST.

As second test , do a "speed test" on a system with 1GB ram.


----------



## grahamperrin@ (Dec 12, 2021)

Alain De Vos said:


> -Perform high intensive database operations
> -Pull the power cord
> -Check how much data is LOST.



I'm always interested in this type of thing, however it's probably unreasonably difficult to quantify/qualify losses – reproducibly – in a way that can be *meaningful for benchmark purposes*. 

An extreme example of consequences of untimely interruption during an intense operation (not database-focused): 



			https://forums.freebsd.org/attachments/1622975200009-png.10114/


----------



## Alain De Vos (Dec 12, 2021)

I remember after a power outage, the ufs fsck run,  and put the length of some files to zero.
Only the filename in directory listing was "conserved".


----------



## nakal (Dec 12, 2021)

The zstd native benchmark seems to have problems with measuring time in some cases. Maybe this is why Phoronix got low results for FreeBSD? See https://github.com/facebook/zstd/bl...c168f6e1581d72df983388/programs/timefn.c#L138


----------



## Alain De Vos (Dec 12, 2021)

Sorry, one just can't do a scientific analysis of a non-scientific benchmark.

Sometimes i think Phoronix test considers only computers with redundant power supply, redundant UPS-sytem, and unlimited ram.
Which are nowhere real conditions.
Sometimes Phoronix is measuring disk-acces-time (ie.syncing) , sometimes he is measuring ram-memory-access-time (ie caching) and this proves nothing about the underlying operating system and file-system.


----------



## ralphbsz (Dec 12, 2021)

Alain De Vos said:


> What a pile of crap.


Exactly. These benchmarks are disorganized, sloppy, probably unreproducible, and don't give anyone guidance for real world installations. Phoronix/Larabel is nothing but a bad journalist who generates his own news.

Anecdote: A friend and their spouse were for many years an independent reporter in a far away country that is very hot and often has disastrous wildfires. The joke they used to make was that one of them would be in the front of someone's house, interviewing the homeowner about how they feel that their house is about to burn down, while the other would be in the back of the house pouring gasoline on the walls.
[/QUOTE]

(About data loss tests)


grahamperrin said:


> I'm always interested in this type of thing, however it's probably unreasonably difficult to quantify/qualify losses – reproducibly – in a way that can be *meaningful for benchmark purposes*.


It is a bit difficult to measure. The problem is that with modern OSes and file systems, most of the time exactly nothing happens. So you need to try a lot of times. The standard technique is to set up a bank of systems (10 or 40 is reasonable, as it fits in a rack), equip them with remote-controllable power supplies, and then crash the regularly (every few minutes, barely long enough to reboot, check for data loss, and start a synthetic workload that stresses the storage devices and the write path). With such a system, you'll get several thousand samples per day. If the system survives a few weeks without any incidents, that is statistically quite meaningful.

There are two problems with this attempt. First, consumer-grade hardware will probably show problems in unrelated components, in particular power supplies and fans, and sometimes even motherboards (the DC/DC converters in the motherboard don't like the surges). This means that just babysitting and diagnosing problems turns into a full-time job. Second, the problem is more often than not the storage device itself. A friend in the industry was testing SSDs early on (in the 2000s, when SSDs were still hot and exciting), and found that just power-cycling SSDs during writes would cause completely bizarre behavior, such as timeouts, data-dependent behavior, and so on. That means that once a "data loss" event happens, diagnosing the root cause will take a lot of effort. You can't just say "the system failed", and even less can you blame the file system code, when in reality many failures are caused by disk hardware, CPUs, and so on.

Reminder: Memory and CPUs are far from perfect either, in particular if extensively power cycled.



Alain De Vos said:


> Sorry, one just can't do a scientific analysis of a non-scientific benchmark.


That's a really good summary of Phoronix. The GIGO principle applies here.


----------



## grahamperrin@ (Dec 13, 2021)

Why rant about _Phoronix_ when something can be patched in _FreeBSD_ to work around the issue that's documented by _Facebook_?


----------



## grahamperrin@ (Dec 13, 2021)

Alain De Vos said:


> I remember after a power outage, the ufs fsck run, …



There's the (extreme) case where fsck became impossible. 

I shouldn't recommend intentional interruptions without prior tuning, however the tuning is (way) off-topic.


----------



## Alain De Vos (Dec 13, 2021)

grahamperrin said:


> Why rant about _Phoronix_ when something can be patched in _FreeBSD_ to work around the issue that's documented by _Facebook_?


The point i wanted to make was to keep information & desinformation seperatly. When you put alot of links together one does the opposite.


----------



## Voltaire (Jul 27, 2022)

There used to be posts like this on Phoronix and people would do the same tests on their hardware and see that FreeBSD was usually faster than Linux in the same tests. I don't know how to explain it. Perhaps Phoronix uses super recent hardware that performs worse on FreeBSD. Maybe he's cherry-picking his results a bit to put Linux in a better light. I did some benchmarks today on FreeBSD and Clear Linux, on the following hardware:_ *i3-3240, 4GB RAM @ 1600Mhz, 850 EVO 500GB, Nvidia GTX 650*_*.* For FreeBSD I benchmarked the version 12.3 install which I have been using daily for almost 4 years usage. For Clear Linux I used an installation on a USB stick, so FreeBSD uses the proprietary Nvidia Linux driver, and Clear Linux uses nouveau. There is also a slight difference in Firefox version but I don't think these things had a big impact on the results. So these are my results:

*FreeBSD 12.3 + Firefox 102*
WebXPRT 4                     * 112*
Kraken 1.1                          *1774.6*
Speedometer 2.0            *62.5*
Octane 2.0                        *16317*
Basemark  Web 3.0       *141.44*
SilverBench                       *P3425*
JetStream 2                       *48368*
Gimp start time                *4.5s* the first time
LibreOffice start time     very slow, maybe because I installed thousands of fonts
Google ping latency       *13.4* ms
RAM XFCE and ZFS         810MB

*Clear Linux + Firefox 97*
WebXPRT 4                        *111*
Kraken 1.1                            *1799.9*
Speedometer 2.0              *74*
Octane 2.0                          *16625*
Basemark Web 3.0          *272*
SilverBench                        *P2216*
JetStream 2                        *62751*
Gimp start time                *7s* the first time
LibreOffice start time     3s or 4s the first time
Google ping latency        *13.55* ms
RAM Gnome                       910MB

FreeBSD opens the game 0 A.D. and the spreadsheet app Gnumeric both in exactly 1 second on my FreeBSD system. But I couldn't test the launch times in Clear Linux because I was using a Live USB stick with no persistence. Clear Linux apps start faster if you already opened them, I don't see this with FreeBSD.

1. FreeBSD is much slower in Basemark Web 3.0 and JetStream 2
2. Clear Linux is much slower in SilverBench

Clear Linux may indeed be faster on average, but not as much as you would expect by phoronix's benchmark and in not all things Clear Linux is effectively faster. Clear Linux is the fastest of all Linux systems according to Phoronix.

I think my benchmarks say that FreeBSD will usually perform similarly or slightly better than Ubuntu/Fedora/Arch Linux/Debian/Alpine Linux/NixOS.


----------



## Voltaire (Dec 19, 2022)

Yesterday I compared Clear Linux's performance against FreeBSD in WASM, JavaScript, HTML5 and CSS.

These were my findings for JavaScript, CSS, HTML5, WASM:

*JavaScript*





						Octane 2.0 JavaScript Benchmark
					

Octane 2.0 JavaScript Benchmark - the Javascript benchmark for the modern web




					chromium.github.io
				





			JetStream 2
		



			https://browserbench.org/ARES-6/
		

Conclusion: The difference between FreeBSD and Clear Linux in these benchmarks is less than 5% on average in Chromium.

*CSS*


			StyleBench 0.3
		






						CSS Layout Performance Test
					

IE9 Test Drive Demo




					testdrive-archive.azurewebsites.net
				



Conclusion: FreeBSD wins in Maze Solver in Chromium, Clear Linux wins in StyleBench in Firefox

*HTML5*





						Chalkboard HTML5 Benchmark
					

HTML5 and SVG Performance Benchmark




					testdrive-archive.azurewebsites.net
				



Conclusion: FreeBSD is 1% faster than Clear Linux in Firefox in this test.

*WASM*


			WebAssembly Benchmark by PSPDFKit
		

Conclusion: FreeBSD is 3% faster than Clear Linux in Chromium in this test.

My general idea would be that FreeBSD has similar CPU performance to Clear Linux.
Clear Linux doesn't have a decent ZFS implementation and also seems to have lower IOPS because many apps start slower.

Doesn't this result mean that FreeBSD has slightly higher CPU performance than most Linux distros and more IOPS than most Linux systems?
In my test Clear Linux is much faster than AlmaLinux in StyleBench and Speedometer in Firefox.


----------



## cracauer@ (Dec 19, 2022)

Voltaire said:


> Clear Linux doesn't have a decent ZFS implementation and also seems to have lower IOPS because many apps start slower.
> 
> Doesn't this result mean that FreeBSD has slightly higher CPU performance than most Linux distros and more IOPS than most Linux systems?
> In my test Clear Linux is much faster than AlmaLinux in StyleBench and Speedometer in Firefox.



Application start performance on identical hardware should be dominated by filesystem readahead. But more aggressive readahead costs memory elsewhere.

Speed of malloc(3) also plays a role. It it actually not difficult to beat the malloc in glibc. But you can fix it by using LD_PRELOAD to use Google's tcmalloc. Another factor is how many pre-zeroed VM pages are available. Always having many is also a tradeoff with memory usage.

Benchmarking this repeatedly is slightly annoying since FreeBSD doesn't have drop_caches for VM pages.


----------



## Voltaire (Dec 19, 2022)

cracauer@ said:


> Benchmarking this repeatedly is slightly annoying since FreeBSD doesn't have drop_caches for VM pages.


It is not with all apps that I notice it, but with GIMP, for example, which is installed on Clear Linux by default. GIMP opens almost twice faster on FreeBSD than on Clear Linux, you don't need an exact timer to observe it.

On the other hand, Clear Linux is also snappy on my old hardware and Wayland is nice to have, although I think Compton works well too.

I think Clear Linux uses a lot of Flatpak for apps. 0 A.D. boots as fast as in FreeBSD, it takes one second on a decent SSD. It could also be Flatpak causing some Clear Linux apps to start a little slower. It's hard to say for sure, but I found that Emacs also started faster on FreeBSD.


----------



## Voltaire (Dec 22, 2022)

I ran some other tests and these are my results:

*WebXPRT 3 on Firefox.*
Clear Linux: 179
GhostBSD: 180

*AnTuTu HTML5 Test on Chromium.*
Clear Linux: 41599
GhostBSD: 41763

*RoboHornet Pro on Chromium.*
Clear Linux: 8.055 seconds
GhostBSD: 7.459 seconds

My general conclusion would be that FreeBSD and Clear Linux are, on average, almost equally fast in Chromium and Firefox.
I've done quite a bit of general testing to say this with certainty.
Browsers are one of the most popular types of software out there.
I don't think I'll blindly trust Phoronix benchmarks from now on.


----------



## Alain De Vos (Dec 22, 2022)

The tests i performed speed of linux & freebsd are almost the same.
One did better in one test, another did better in another test.
So speed should not be determinend if you chose one or another.


----------



## Crivens (Dec 23, 2022)

One could factor in the time needed to keep the installation running, adapt the setup to new wasy of doing things, etc.
I found that I spend very little time doing this for my FreeBSD boxes.


----------



## Voltaire (Dec 23, 2022)

Crivens said:


> One could factor in the time needed to keep the installation running, adapt the setup to new wasy of doing things, etc.
> I found that I spend very little time doing this for my FreeBSD boxes.


A server must also have quite high performance. If you read the Phoronix article he makes it seem like Linux is on average 2x faster than FreeBSD. Although in reality FreeBSD is about exactly as fast as Clear Linux in most of the popular apps. Then you should also know that Clear Linux is a lot faster than, for example, AlmaLinux. Although Phoronix pretends that AlmaLinux is 'lightning fast' in its benchmarks. AlmaLinux isn't nearly as fast as Clear Linux, I'd guess it's *27% slower* than Clear Linux on average. Which means that overall, FreeBSD is going to be just _a little bit_ faster than most of the popular Linux distros. Think of Debian, OpenSUSE, Ubuntu and AlmaLinux. They will have slightly lower performance than FreeBSD in most apps.

What bothers me about Linux is its stability. The FreeBSD system has never completely crashed in over 4 years. If the desktop crashes, which I have hardly seen, I can always go to a console and shut down the PC from there. I've been testing Clear Linux for a few days now, and running FlightGear in Wayland completely crashed the system. By that I mean I had to cut power to the PC by holding down the power button. The system stopped responding at all. I can still understand that Wayland on nouveau does not work perfectly as it should. But then on xorg I also randomly saw two complete system crashes. I'm going to argue that _not_ every Linux is going to crash several times in a few days, but I've found that complete system crashes are a common phenomenon with Linux systems. I'm kind of a 'distrohopper' by nature, so I've used hundreds of Linux systems in the meantime. But with none of those systems have I ever felt as reliable in terms of stability as FreeBSD.

Another issue for me is the audio. I use some custom options on FreeBSD which makes the audio very good, although the default audio quality is very good as well. The audio quality of Clear Linux was nothing short of a horror. It literally sometimes makes popping noises that I never heard on FreeBSD. And I struggled to understand people in Zoom. Clear Linux's sound in audio also gives me the feeling that it has many times less detail and that the frequencies are not correct or are partially missing completely.


----------



## cracauer@ (Dec 23, 2022)

When I benchmark straight applications (as in one at a time) I rarely detect differences between Linux (Debian) and FreeBSD.

Things get interesting, however, if you run different benchmarks competing for different resources at the same time. For example, load the CPUs to the max and then also benchmark NFS serving. You can see different tradeoff decisions made in Linux and FreeBSD pretty clearly.


----------

