# Is this benchmark fair?



## Weinter (Nov 25, 2008)

A new benchmark have emerge testing FreeBSD vs OpenSolaris vs Ubuntu and FreeBSD doesn't seem to be pwning others (in the benchmark) any opinions?
See the benchmark here


----------



## kamikaze (Nov 25, 2008)

Sure it's fair. Default settings are not perfect for everything and RELENG_7 still has GIANT in the TTY and VFS layers.


----------



## Mel_Flynn (Nov 25, 2008)

That and most differences are marginal. The 2GB file encryption for instance, could be attributed to pth (threading library) being more optimized for linux threading under this specific workload.
It's finding out where the slowdowns are that is hard.


----------



## ArtemD (Nov 25, 2008)

Benchmarks were done using GENERIC kernel which as mentioned above is not optimized for speed. GENERIC kernel is optimized to run on as many hardware as possible and to be as stable as possible. This means it has no optimization tweaks and it's bloated with every possible driver that FreeBSD supports.

Another things that shines through this benchmarks is UFS2's poor performance compared to ZFS and Ext3. I think UFS2+gjournal created with "-o time" option would yield a lot better results.

Personally even if FreeBSD *was* incurably slower then Linux/Solaris I would still take stability and ports over speed any day.


----------



## Mel_Flynn (Nov 25, 2008)

ArtemD said:
			
		

> Another things that shines through this benchmarks is UFS2's poor performance compared to ZFS and Ext3. I think UFS2+gjournal created with "-o time" option would yield a lot better results.



Sort of. The sequential write test, ext3 seems to do very bad. However, it's hard to translate this to real workloads as the network stack will almost always be involved.

It is kind of weird, they chose to create UFS partitions and yes, I agree I would love to see those results with gjournal and ZFS.


----------



## trasz@ (Nov 25, 2008)

Were the tests performed several times?  Was the first run of each test discarded ('preheating')?  What was the standard deviation?  How can we tell if the measurement error is not bigger than the measured differences between systems? Without knowing that, the tests are not really relevant or useful.


----------



## Weinter (Nov 25, 2008)

ArtemD said:
			
		

> Benchmarks were done using GENERIC kernel which as mentioned above is not optimized for speed. GENERIC kernel is optimized to run on as many hardware as possible and to be as stable as possible. This means it has no optimization tweaks and it's bloated with every possible driver that FreeBSD supports.



I think they didn't optimize Linux kernel or Solaris kernel as well so that point should be invalid


----------



## Weinter (Nov 25, 2008)

ArtemD said:
			
		

> Another things that shines through this benchmarks is UFS2's poor performance compared to ZFS and Ext3. I think UFS2+gjournal created with *"-o time" option* would yield a lot better results.



Sorry i don't understand this part *"-o time" option*
Could you elaborate?


----------



## ArtemD (Nov 25, 2008)

Weinter said:
			
		

> Sorry i don't understand this part *"-o time" option*
> Could you elaborate?


From "man newfs":

```
-o optimization
             (space or time).  The file system can either be instructed to try
             to minimize the time spent allocating blocks, or to try to mini-
             mize the space fragmentation on the disk.  If the value of min-
             free (see above) is less than 8%, the default is to optimize for
             space; if the value of minfree is greater than or equal to 8%,
             the default is to optimize for time.  See tunefs(8) for more
             details on how to set this option.
```

I haven't done any benchmarks to test effect of this option in real life, but I assume it does help.


----------



## ArtemD (Nov 25, 2008)

Weinter said:
			
		

> I think they didn't optimize Linux kernel or Solaris kernel as well so that point should be invalid



That is correct. I do think thou that placing too much value on this makes no sense since I assume most of FreeBSD users run custom kernels.

P.S. I personally think FreeBSD is "fast enough" for everything I needed it for. I choose FreeBSD for stability, ports, good documentation etc.


----------



## Almindor (Nov 25, 2008)

The odd thing is, ext3 in stock options is defenetly slower than UFS on certain applications (see http://shaver.off.net/diary/2008/05/25/fsyncers-and-curveballs/).

For me ext3 is simply slower overall, and this is coming from someone who went from Ubuntu 8.1/AMD64 to FreeBSD 7.1-BETA2/AMD64 (still GENERIC too!)

Perhaps they didn't have softupdates on, otherwise I don't know. In my opinion FreeBSD 7.1 is much faster compared to Ubuntu 8.10 (just personal feeling).


----------



## tanked (Nov 25, 2008)

Does the version of FreeBSD they tested still have the dubug options turned on?


----------



## kamikaze (Nov 25, 2008)

Kernel debug does not slow the system down. The debug data is in separate files that only get loaded during debugging.


----------



## tanked (Nov 25, 2008)

Are you certain; whenever someone complains in the mailing list about slow performance on Beta / CURRENT versions the standard reply seems to be "turn the debugging off"... or am I thinking of something else?


----------



## kamikaze (Nov 25, 2008)

This is a relatively recent change. I think in RELENG_6 debugging is still a performance eater.


----------



## richardpl (Nov 25, 2008)

kamikaze said:
			
		

> Kernel debug does not slow the system down. The debug data is in separate files that only get loaded during debugging.



You really dont know what are you talking about.

kernel on CURRENT have many debug options turned on, and it is always slower than one without such options.


```
NOTE TO PEOPLE WHO THINK THAT FreeBSD 8.x IS SLOW:
        FreeBSD 8.x has many debugging features turned on, in
        both the kernel and userland.  These features attempt to detect
        incorrect use of system primitives, and encourage loud failure
        through extra sanity checking and fail stop semantics.  They
        also substantially impact system performance.  If you want to
        do performance measurement, benchmarking, and optimization,
        you'll want to turn them off.  This includes various WITNESS-
        related kernel options, INVARIANTS, malloc debugging flags
        in userland, and various verbose features in the kernel.  Many
        developers choose to disable these features on build machines
        to maximize performance.  (To disable malloc debugging, run
        ln -s aj /etc/malloc.conf.)
```


----------



## trasz@ (Nov 25, 2008)

kamikaze said:
			
		

> Kernel debug does not slow the system down. The debug data is in separate files that only get loaded during debugging.



Not true.  System with debugging enabled _is_ slower, because it runs various checks (debugging code) that system with debugging disabled does not run, because they get compiled out.  This affects both kernel (WITNESS, which makes locking slower by checking lock order, and INVARIANTS, which enables KASSERTs) and userland (malloc debugging).

Debug data you're talking about are so called "symbols".  That's true, symbols don't slow things down.  But other debugging options, that are enabled by default in -CURRENT (RELENG_8) definitely do.


----------



## tanked (Nov 25, 2008)

In the article it says they are using FreeBSD 7.1-BETA 2, so I assume all the debugging/WITNESS stuff is not turned on in this case.


----------



## MartijnAtLico (Nov 26, 2008)

Well the article didn't convince me to switch our Glassfish server to Solaris (which I _was_ considering), so I think FreeBSD came out pretty okay. Especially since there's some headroom for tuning.


----------



## Mel_Flynn (Nov 26, 2008)

trasz@ said:
			
		

> Were the tests performed several times?  Was the first run of each test discarded ('preheating')?  What was the standard deviation?  How can we tell if the measurement error is not bigger than the measured differences between systems? Without knowing that, the tests are not really relevant or useful.



Not to mention that:
- systems are not optimized for a single specific workload
- systems are tested one by one for a single specific workload
- No combined tests are done
- systems are evaluated on "nr. of 1st positions", difference between contestants isn't processed/weighed in ranking.

So, for this test to have any merit, you know only that for a specific workload os foo comes out best without it being optimized for it.
If you dedicate a machine for one specific workload (or a workload that is highly dominant for a specific worklad) you always optimize for it and it can't be assumed that "because without optimizing os foo comes out best, it will still be best once optimized".
All in all, it's good for stirring up discussions in forums, but not much else.


----------



## Eponasoft (Nov 27, 2008)

My thoughts are that these benchmark tests are complete bunk. You can use any test you feel like using to prove so-and-so system is better than so-and-so other system. Big corporations do it all the time.


----------



## kamikaze (Nov 28, 2008)

The only debugging options active by default in RELENG_7 are debugging symbols for the kernel. And they only use disk space.


----------



## tomh009 (Nov 28, 2008)

ArtemD said:
			
		

> That is correct. I do think thou that placing too much value on this makes no sense since I assume most of FreeBSD users run custom kernels.



I'm not sure that's true.  Currently all our FreeBSD servers (4.x, 5.x and 6.x) run GENERIC.  With 7.0, the only change I'm making is to choose SCHED_ULE.  In the distant past I did some more kernel tuning, but there simply hasn't been a need in recent years.

Given how well the GENERIC kernel works, how many users will actually go through the work of building a custom kernel?


----------



## Djn (Nov 28, 2008)

Well, there's always the subgroup of desktop users running CURRENT with the extra debugging disabled - though I suspect there's not that many of us.


----------



## marino (Nov 28, 2008)

tomh009 said:
			
		

> Given how well the GENERIC kernel works, how many users will actually go through the work of building a custom kernel?



I agree.  I actually had to reread the original quote because I thought it said, "I assume most of FreeBSD users *DO NOT* run custom kernels.", which I would have expected.


----------

