# (Benchmarks) To clang or not to clang



## falkman (Jul 15, 2011)

Hello there,

I just got my new motherboard in the mail (the old one was shot and couldn't boot FreeBSD), and I put FreeBSD-9.0-CURRENT (201105-amd64-dvd1) on my box to try to figure out a lingering question I had... should I use clang or gcc? So, I did some tests!

First of all, I pulled FreeBSD head Rev 224044 for the tests, and built (worlds only, no kernels) one system with gcc (4.2.2-20070831), and the other with clang (3.0-20110502)and installed them in a chroot.

Note: All tests were done in RAM (the files to tar, the tarred file, and the flops program, but the system/world was on the HD). Also, the kernel used was compiled with gcc, I'll release benchmarks with a clang kernel later. Given that, these results are still very valid

System specs:

Intel i7-970 (6-core with HT [12 threads] @ 3.20GHz)
24GB ram (DDR3-1600 [Damnit, I have it set as 1067MHz in the BIOS])
50GB SSD HD (200MB/s write, 250MB/s read)

*Test #1: tar.bz2ing the FreeBSD head (cleaned) that I used to build the systems*
_Note: I tarred the files first, then tarred again with timing to clear up having caching changing performance a lot_

`tar cjf test.tar.bz2 tarme/`

`time` on clang:
209.75 real 203.81 user 5.93 sys

`time` on gcc:
215.62 real 207.54 user 8.07 sys

*Test #2: md5ing the generated tarball 100 times*

`time` on clang:
79.95 real 46.29 user 33.65 sys

`time` on gcc:
84.11 real 50.77 user 33.33 sys

*Test #3: sha1ing the generated tarball 100 times*

`time` on clang:
112.50 real 78.53 user 34.04 sys

`time` on gcc:
102.64 real 69.62 user 33.02 sys

*Test #4: sha256ing the generated tarball 100 times*

`time` on clang:
196.92 real 163.53 user 33.38 sys

`time` on gcc:
204.38 real 171.47 user 32.90 sys

*Test #5: MFLOPS from the benchmarks/flops port*

flops.c built as:
`clang -O2 -DUNIX flops.c`
`gcc -O2 -DUNIX flops.c`

_NOTE: THERE ARE 2 OUTPUTS FOR EACH SYSTEM FOR REDUNDANCY_

clang output #1:


```
FLOPS C Program (Double Precision), V2.0 18 Dec 1992

   Module     Error        RunTime      MFLOPS
                            (usec)
     1      4.0146e-13      0.0080   1753.5920
     2     -1.4166e-13      0.0065   1072.0893
     3      4.7184e-14      0.0062   2756.7597
     4     -1.2557e-13      0.0061   2453.0056
     5     -1.3800e-13      0.0116   2506.2720
     6      3.2380e-13      0.0272   1065.8150
     7     -8.4583e-11      0.0257    466.0625
     8      3.4867e-13      0.0120   2505.2298

   Iterations      =  512000000
   NullTime (usec) =     0.0000
   MFLOPS(1)       =  1339.7506
   MFLOPS(2)       =   938.0637
   MFLOPS(3)       =  1508.7634
   MFLOPS(4)       =  1768.1651
```

clang output #2:


```
FLOPS C Program (Double Precision), V2.0 18 Dec 1992

   Module     Error        RunTime      MFLOPS
                            (usec)
     1      4.0146e-13      0.0080   1754.0005
     2     -1.4166e-13      0.0065   1071.8402
     3      4.7184e-14      0.0062   2755.8083
     4     -1.2557e-13      0.0061   2450.9765
     5     -1.3800e-13      0.0116   2505.0859
     6      3.2380e-13      0.0272   1065.7021
     7     -8.4583e-11      0.0257    466.5673
     8      3.4867e-13      0.0120   2505.7709

   Iterations      =  512000000
   NullTime (usec) =     0.0000
   MFLOPS(1)       =  1339.4153
   MFLOPS(2)       =   938.6299
   MFLOPS(3)       =  1509.0247
   MFLOPS(4)       =  1767.9079
```

gcc output #1:

```
FLOPS C Program (Double Precision), V2.0 18 Dec 1992

   Module     Error        RunTime      MFLOPS
                            (usec)
     1      4.0146e-13      0.0080   1754.2786
     2     -1.4166e-13      0.0091    771.8205
     3      4.7184e-14      0.0066   2566.9269
     4     -1.2557e-13      0.0063   2396.7763
     5     -1.3800e-13      0.0229   1268.5356
     6      3.2380e-13      0.0111   2619.0838
     7     -8.4583e-11      0.0240    500.9846
     8      3.4867e-13      0.0232   1292.1545

   Iterations      =  512000000
   NullTime (usec) =     0.0000
   MFLOPS(1)       =  1000.5765
   MFLOPS(2)       =  1009.2525
   MFLOPS(3)       =  1431.8632
   MFLOPS(4)       =  1929.1627
```

gcc output #2:


```
FLOPS C Program (Double Precision), V2.0 18 Dec 1992

   Module     Error        RunTime      MFLOPS
                            (usec)
     1      4.0146e-13      0.0080   1753.9284
     2     -1.4166e-13      0.0091    771.5915
     3      4.7184e-14      0.0066   2566.8247
     4     -1.2557e-13      0.0063   2395.5376
     5     -1.3800e-13      0.0229   1268.4892
     6      3.2380e-13      0.0111   2619.6577
     7     -8.4583e-11      0.0239    501.2391
     8      3.4867e-13      0.0232   1292.0066

   Iterations      =  512000000
   NullTime (usec) =     0.0000
   MFLOPS(1)       =  1000.3124
   MFLOPS(2)       =  1009.5551
   MFLOPS(3)       =  1431.9474
   MFLOPS(4)       =  1929.0101
```

*Average MFLOPS for clang:*

1388.715075

*Average MFLOPS for gcc:*

1342.7099875

All in all, it seems that clang outperformed gcc by a decent bit when it came to tar, md5, sha256, and MFLOPS. The ONLY place where gcc seemed to outperform, was in sha1, which seems to be a fluke (bit I still double checked and it isn't).

When it comes to md5, sha256, and tar, there is a quite considerable difference, and in MFLOPS it does manage to beat gcc, but not by a very noticeable amount.

Now, you could go out and build a modern version of gcc which would probably outperform clang. But a modern gcc is not supported at all on FreeBSD and I'd prefer the BSD license over the GPL anyways.

Quick summary:

- If you care about performance, would sacrifice support, and don't care about licensing, use a modern day GCC.

- If you care about performance, want support (but can sacrifice a tiny amount of it), and care about licensing, use clang.

- If you don't care about performance, want the best support, and don't care about licensing, use the bundled 'old' gcc.

To me, the license and support pull me to clang, and as a dev I'm always looking for new projects to contribute to, and this one looks perfect for my tastes.

If you have any questions about my build environment, or want me to do other benchmarks (I probably wont do any more after tomorrow (Friday) as I'll be reinstalling a clean build of FreeBSD with clang).

Cheers,
Brandon Falk


----------



## UNIXgod (Jul 15, 2011)

Thank you for the benchmark.

Just out of curiosity; aside from gcc is there anything else in BSD base that is still GNU licensed?


----------



## Zare (Jul 15, 2011)

```
[zare@zpb /usr/src/gnu]$ ls -l lib/
total 22
-rw-r--r--  1 root  wheel  412 Aug  3  2009 Makefile
-rw-r--r--  1 root  wheel  111 Aug  3  2009 Makefile.inc
drwxr-xr-x  2 root  wheel    3 Jul 11 10:01 csu
drwxr-xr-x  3 root  wheel   34 Jul 11 10:01 libdialog
drwxr-xr-x  2 root  wheel    6 Jul 11 10:01 libg2c
drwxr-xr-x  2 root  wheel    3 Jul 11 10:01 libgcc
drwxr-xr-x  2 root  wheel    3 Jul 11 10:01 libgcov
drwxr-xr-x  2 root  wheel    4 Jul 11 10:01 libgomp
drwxr-xr-x  2 root  wheel    4 Jul 11 10:01 libiberty
drwxr-xr-x  2 root  wheel    3 Jul 11 10:01 libobjc
drwxr-xr-x  4 root  wheel    7 Jul 11 10:01 libreadline
drwxr-xr-x  4 root  wheel   14 Jul 11 10:01 libregex
drwxr-xr-x  3 root  wheel    5 Jul 11 10:01 libssp
drwxr-xr-x  3 root  wheel    5 Jul 11 10:01 libstdc++
drwxr-xr-x  2 root  wheel    3 Jul 11 10:01 libsupc++
[zare@zpb /usr/src/gnu]$ ls -l usr.bin/
total 32
-rw-r--r--   1 root  wheel  813 May 25  2010 Makefile
-rw-r--r--   1 root  wheel  134 Aug  3  2009 Makefile.inc
drwxr-xr-x   2 root  wheel    4 Jul 11 10:01 bc
drwxr-xr-x  22 root  wheel   26 Jul 11 10:01 binutils
drwxr-xr-x  22 root  wheel   27 Jul 11 10:01 cc
drwxr-xr-x   8 root  wheel   10 Jul 11 10:01 cvs
drwxr-xr-x   3 root  wheel    4 Jul 11 10:01 dc
drwxr-xr-x   3 root  wheel    9 Jul 11 10:01 dialog
drwxr-xr-x   3 root  wheel    6 Jul 11 10:01 diff
drwxr-xr-x   2 root  wheel    4 Jul 11 10:01 diff3
drwxr-xr-x   9 root  wheel   11 Jul 11 10:01 gdb
drwxr-xr-x   3 root  wheel    5 Jul 11 10:01 gperf
drwxr-xr-x   4 root  wheel   44 Jul 11 10:01 grep
drwxr-xr-x   8 root  wheel   10 Jul 11 10:01 groff
drwxr-xr-x   6 root  wheel   11 Jul 11 10:01 man
drwxr-xr-x   2 root  wheel   20 Jul 11 10:01 patch
drwxr-xr-x  14 root  wheel   20 Jul 11 10:01 rcs
drwxr-xr-x   2 root  wheel    4 Jul 11 10:01 sdiff
drwxr-xr-x   3 root  wheel   11 Jul 11 10:01 send-pr
drwxr-xr-x   2 root  wheel    7 Jul 11 10:01 sort
drwxr-xr-x   9 root  wheel   11 Jul 11 10:01 texinfo
[zare@zpb /usr/src/gnu]$
```


----------

