# First Look: FreeBSD 9.0 On Intel Sandy Bridge



## alie (Nov 18, 2011)

What do you guys think about this benchmark

http://www.phoronix.com/scan.php?page=article&item=intel_snb_freebsd&num=4


----------



## Bobbla (Nov 18, 2011)

I just read the first page and looked at the pictures on the rest...

I've got to ask, it seems like that graphics card is the ONLY hardware that is "equal" for both OS's. So how can this be a fair test if you are testing OS's on different hardware? Shouldn't the hardware be the same if you are testing the OS?

I'm confused... the author couldn't have just neglected this? right? :\


----------



## rusty (Nov 18, 2011)

ZFS is quite impressive in that benchmark when you consider the extra overhead from checksumming etc.


----------



## phoenix (Nov 18, 2011)

Bobbla said:
			
		

> I just read the first page and looked at the pictures on the rest...
> 
> I've got to ask, it seems like that graphics card is the ONLY hardware that is "equal" for both OS's. So how can this be a fair test if you are testing OS's on different hardware? Shouldn't the hardware be the same if you are testing the OS?
> 
> I'm confused... the author couldn't have just neglected this? right? :\



It's Phoronix.    They put the bare minimum amount of thought into benchmarking, and don't do anything remotely close to "scientific" or "mathematically sound" or even "statistically relevant" thought into it.

The only way to make this benchmark relevant would be to:

Install FreeBSD onto the laptop, without any GUI.  Do the benchmarks from single-user mode (or at last with as many services disabled as possible) several times.  Graph the results.
Install Ubuntu onto the laptop, without any GUI.  Do the benchmarks from single-user mode (or at last with as many services disabled as possible) several times.  Graph the results.

But, Phoronix isn't about statistically relevant benchmarks.  Just crazy headlines and pretty images that bring in page views.


----------



## alie (Nov 18, 2011)

Yes, but it gives bad impressions for FreeBSD, really unfair


----------



## aragon (Nov 19, 2011)

phoenix said:
			
		

> The only way to make this benchmark relevant would be to:
> 
> Install FreeBSD onto the laptop, without any GUI.  Do the benchmarks from single-user mode (or at last with as many services disabled as possible) several times.  Graph the results.
> Install Ubuntu onto the laptop, without any GUI.  Do the benchmarks from single-user mode (or at last with as many services disabled as possible) several times.  Graph the results.


I don't understand how benchmarking an operating system in a state that it would never normally be used is of any relevance.  Why do we care how operating systems perform in single user mode?


----------



## xibo (Nov 20, 2011)

aragon said:
			
		

> I don't understand how benchmarking an operating system in a state that it would never normally be used is of any relevance.  Why do we care how operating systems perform in single user mode?



We don't. However, if you want to benchmark ZFS vs EXT4, why have strigi and cron waste fs load? And more importantly, if strigi is slowing in Ubuntu, but not running in FreeBSD, what's that benchmark worth? Therefore we try to run the minimal required software while benchmarking, and this can be achieved by booting into single user mode.

It's interresting to see that in "theory" FreeBSD is running slower then Ubuntu, because to me FreeBSD "feels" faster. And I can add my music collection to vlc in a second with FFS2 while it takes nearly a minute with EXT4 (or ZFS).


----------



## phoenix (Nov 20, 2011)

aragon said:
			
		

> I don't understand how benchmarking an operating system in a state that it would never normally be used is of any relevance.  Why do we care how operating systems perform in single user mode?



When benchmarking two different systems, you have to eliminate as many variables as possible, leaving only the 1 or 2 variables that you are trying to test.  Everything else should be a constant (identical between systems).

For example, in the Phoronix benchmark, the following are different:

CPU speed
amount of RAM
graphics hardware
harddrive
OS
filesystem
version of Xorg
version of video driver
desktop environment
number and type of daemons running in the background
probably other things as well

One system is slower than the other.  Can anyone definitively say that the slowness is solely due to the OS?  Nope.  It could be from anything on that list.

Maybe single-user mode isn't the "best" way to benchmark two different OSes.  But it eliminates a lot of the differences between the two.

What's annoying about Phoronix is that they have the hardware, they have the software, and they have the time to run crappy benchmarks like the above, so why not spend a little more time, and do the benchmarks properly?

And it wouldn't be that hard to do:

Use the exact same piece of hardware for all tests (or two identically configured systems, but that adds another variable to the equation since two "identical" systems rarely ever are)
Configure both OSes with the exact same versions of all software and drivers
Run the tests multiple times with different filesystems (ext3, ext4, btrfs, zfs on Linux; UFS w/softupdates, UFS w/SU-J, zfs on FreeBSD) to try and eliminate that as a primary variable

Failure to do the above is nothing more than link-/flame-bait without any real value.

Unfortunately, Phoronix has no inclination or desire to do anything beyond link-/flame-bait.


----------



## alie (Dec 14, 2011)

Another bad benchmarking... i am speechless hahaha... they are comparing 2 different OS with different hardware...

http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA


----------



## aragon (Dec 14, 2011)

alie said:
			
		

> they are comparing 2 different OS with different hardware...


Different hardware?  Where?

The textual differences in the hardware table are most likely artifacts of their benchmark collecting hardware information from completely different kernels.


----------



## jalla (Dec 15, 2011)

alie said:
			
		

> Another bad benchmarking... i am speechless hahaha... they are comparing 2 different OS with different hardware...
> 
> http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA



The tests were performed on the same hardware. Here'e a clip from a posting to freebsd-stable today



> On 12/15/2011 04:41 AM, Michael Ross wrote:
> > Am 15.12.2011, 11:10 Uhr, schrieb Michael Larabel
> > <michael.larabel@phoronix.com>:
> >
> ...


----------



## frooyo (Dec 25, 2011)

@Everyone

Since the tests were in fact on the same hardware, as the updated article indicates, can someone please validate these test.

I ask because FreeBSD is *significantly *slower and this should be addressed.

Thanks in advance developers for all that you do.


----------



## wblock@ (Dec 25, 2011)

There has been a lot of discussion of this on the freebsd-current mailing list.  One of the big questions was why ZFS was used, when UFS was much more comparable to ext4.  The answer, as I saw it, was that FreeBSD had been installed on ZFS, so that's what they used.  That's not so much benchmarking as "hey, let's see what this does."  FreeBSD prerelease kernels have debug code that slows them down; did they build one without that?  It's not mentioned.  Useful benchmarks can be used to draw conclusions, but there are so many variables here that it's impossible to tell.

The author has been involved in those emails, so there's hope they will be more specific in future tests.


----------



## frooyo (Dec 25, 2011)

wblock@

Thanks for the comment.

Has there been any talk of someone reproducing the test results. 

Also, you mentioned this topic has been talked about a lot on the mail list. Where can I read that conversation on the web?


----------



## wblock@ (Dec 25, 2011)

frooyo said:
			
		

> Has there been any talk of someone reproducing the test results.



If so, I missed it.  I think the benchmark is downloadable, but unless you have the same hardware... or are planning on duplicating both their Linux and FreeBSD tests.  If you are planning on doing a useful benchmark, it would be good to ask for suggestions first.  It helps to be thick-skinned, as benchmarks are easy to do wrong, hard to do right, and generally a boring, painstaking, thankless chore.  Science, in other words.



> Also, you mentioned this topic has been talked about a lot on the mail list. Where can I read that conversation on the web?



Much of it is here: http://lists.freebsd.org/pipermail/freebsd-current/2011-December/thread.html.  Search for "benchmark".


----------



## frooyo (Dec 25, 2011)

wblock@ said:
			
		

> If so, I missed it.  I think the benchmark is downloadable, but unless you have the same hardware... or are planning on duplicating both their Linux and FreeBSD tests.  If you are planning on doing a useful benchmark, it would be good to ask for suggestions first.  It helps to be thick-skinned, as benchmarks are easy to do wrong, hard to do right, and generally a boring, painstaking, thankless chore.  Science, in other words.
> 
> Much of it is here: http://lists.freebsd.org/pipermail/freebsd-current/2011-December/thread.html.  Search for "benchmark".



Why do I need the us the same hardware they did?

Seems like I should just be able to fire up Virtualbox on my machine. Load Ubuntu in one instance, FreeBSD in the other instance.

Then run the tests and show which operating system is proportionally faster/slower.

(Obviously not run the images at the same time)


----------



## wblock@ (Dec 25, 2011)

See, that's the tricky part.  Different hardware might have vastly different reactions to the same code.  CPU and disk cache, bus bandwidth, there are thousands of things that would be changing.  Running in a VM is a big difference, too.  If you can remove all the variables except one--or in reality, a few--then you can get meaningful comparisons.

If you run the same benchmarks in a VM and FreeBSD is vastly faster... what could you point to and say "My test shows the opposite of theirs, and here's why mine was valid."


----------



## frooyo (Dec 25, 2011)

wblock@ said:
			
		

> See, that's the tricky part.  Different hardware might have vastly different reactions to the same code.  CPU and disk cache, bus bandwidth, there are thousands of things that would be changing.  Running in a VM is a big difference, too.  If you can remove all the variables except one--or in reality, a few--then you can get meaningful comparisons.
> 
> If you run the same benchmarks in a VM and FreeBSD is vastly faster... what could you point to and say "My test shows the opposite of theirs, and here's why mine was valid."



With that mindset, you're imply that benchmark test should never be performed because they are never useful.

Is that correct? 

(I hope not)


----------



## frooyo (Dec 25, 2011)

wblock

Since this topic has been obviously exhausted at length in the mail-list, I'll stop asking questions here.

Thanks again for everything you do for FreeBSD


----------



## wblock@ (Dec 25, 2011)

Good benchmarks are very useful.  The way they are done is to minimize variables so you can tell what is different between A and B.  So if you want to compare typical speed for a user-oriented release, compare an Ubuntu release with 9.0-RELEASE, not a prerelease version.  Or at least rebuild the kernel to disable the debugging code, as would be done for a real release.  Give them comparable filesystems, ext4 versus UFS.  Run the benchmarks on the exact same machine, multiple times, throwing away the first result to minimize the effect of filesystem buffering.  That would be a reasonable start.


----------

