# [HOWTO] buildworld using clang + ccache



## Beeblebrox (Nov 23, 2012)

clang is now the default compiler for FreeBSD (instead of gcc-42), and things supposedly run quicker under clang. You can do the buildworld with clang, but including ccache in the process needs some small adjustments:

1. Edit your /etc/src.conf as below. clang is enabled by default anyway, but you may want to add extras to it. Originally this post advised WITHOUT_GCC for a quicker build, but this is ill-advised as explained below.

```
WITH_CLANG= yes
WITH_CLANG_EXTRAS= yes # Build additional clang and llvm tools, such as bugpoint.
WITH_CLANG_IS_CC= yes  # Useful for some buildworld errors
# WITHOUT_GCC= yes
```

2. In your /usr/local/etc/buildflags.conf set:

```
/usr/src | /usr/src/*{
        CC=     clang
        CPP=    clang-cpp
        CXX=    clang++
        USE_CCACHE
        USE_CCACHE_CPP2
}
```

You are now set to buildworld with clang + ccache.  CCACHE_CPP2 must be enabled because clang dislikes ccache trying extra flags.

If you do not use sysutils/bsdadminscripts, and therefore do not have buildflags.conf, you can set those parameters from command line :
[CMD="/usr/src> #"]make -DUSE_CCACHE -DUSE_CCACHE_CPP2 buildworld[/CMD]
In this situation you of course need these in your /etc/make.conf.

```
CC=     clang
CPP=    clang-cpp
CXX=    clang++
```

Thanks to Volodymyr Kostyrko for helping me solve this.

*EDIT:* Apparently as of the date of this post, setting WITHOUT_GCC= yes does not work yet and breaks buildworld. It must be left in /etc/src.conf for now.


----------



## wblock@ (Nov 23, 2012)

What ccache environment variables or /etc/make.conf settings are used with this?

Although it will increase compile time, it might be a good idea to leave gcc enabled in src.conf just to have it available.


----------



## Beeblebrox (Nov 23, 2012)

No additional ccache environemnt variables / settings required than when using gcc-4* (CCACHE_PATH, CCACHE_DIR) and setting the size (# ccache -M <size>)

The config file for building ccache now has the options for

```
CLANGLINK
LLVMLINK
```
These, when selected will automagically create FILE]ccache[/FILE] shortcuts for clang to ccache.

As a Third way of doing things, specifying these in /etc/make.conf should work:

```
CC=/usr/local/libexec/ccache/world/clang
CXX=/usr/local/libexec/ccache/world/clang++
CPP=/usr/local/libexec/ccache/world/clang-cpp
```

I use lang/gcc46 for compiling my ports. Users who do not have a separate gcc should not disable gcc42 because a good number of ports still require gnu-gcc to compile. Invoke WITHOUT_GCC= yes only if you plan to use a higher version of gcc for your ports.


----------



## xeube (Nov 23, 2012)

Did I miss something? I tought that clang was to become the default compiler starting with 10.0.


----------



## kpa (Nov 23, 2012)

It will become the default compiler in CURRENT but it's not yet done I believe.

It's a good idea to leave GCC in base because of some potential hickups in buildworld. Just recently it was impossible to compile parts of the boot code with clang(1) for a while and if you don't have GCC as the fallback compiler you can't finish the build until the problem is fixed in SVN.


----------



## Beeblebrox (Nov 23, 2012)

@xeube: Yes, clang is default for HEAD only. Its announcement as default is recent. Before that, it was only targeted / planned as default.

@kpa: The ride is certainly far from smooth...


----------



## ColdfireMC (Nov 24, 2012)

does CLANG have advantages compared with base gcc?


----------



## wblock@ (Nov 24, 2012)

Supposed to compile faster and have better error messages.  Also a BSD-compatible license.


----------



## Beeblebrox (Nov 24, 2012)

*UPDATE:* I just finished buildworld with settings:

```
WITHOUT_GCC= yes
WITH_CLANG= yes
WITH_CLANG_EXTRAS= yes
WITH_CLANG_IS_CC= yes
```
However, ccache does not always play well with clang and needs to be disabled if build stops.
This means, start with ccache enabled, but if build stops, disable ccache and <threads> then re-start, OK?

@ ColdfireMC:  Olivier Smedts informed me of below - your call: 





> buildworld is ~= 10% faster with a clang-compiled kernel + world than with a stock (old 4.2) gcc-compiled kernel+world. I'm not speaking of the compiler speed but really the speed of the same build when run under a clang- or a gcc-compiled base system.


----------



## ColdfireMC (Nov 24, 2012)

wblock@ said:
			
		

> *Supposed* to compile faster and have better error messages.  Also a BSD-compatible license.



but...? are there problems about that?

I recently read that binaries are slightly faster, because of newer and better cpu support than gcc4.2 but is near the same versus newer(or not-so-newer) gcc versions

is that true?

well, stop depending on a GNU (or another license with compatiblity issues with BSD license and philosophy) is a huge step towards gaining community independence, keep it up .


----------



## kpa (Nov 24, 2012)

GCC has been around lot longer than clang and it may perform slightly better code optimizations in some cases even on an old version like 4.2.1.


----------



## Beeblebrox (Nov 25, 2012)

UPDATE: I built my clang-only world and kernel with:

```
WITH_CLANG= yes
WITH_CLANG_EXTRAS= yes
WITH_CLANG_IS_CC= yes
WITHOUT_GCC= yes
```
CLANG_IS_CC is definitely needed if you decide to apply WITHOUT_GCC. I was however, forced to disable ccache at some point so as to continue the build from the point that buildworld broke (using -DNO_CLEAN). Then the build completed without any other problems. Running my clang-only world now


----------



## jozze (Jun 7, 2013)

Sorry for asking, but does clang natively include ccache, or should one install it separately from port devel/ccache?


----------



## kpa (Jun 7, 2013)

No, it's a separate tool and supports other compilers as well. You have to install the port devel/ccache


----------



## jozze (Jun 7, 2013)

Cool, thanks for the info .


----------



## wblock@ (Jun 7, 2013)

Last time I tested, ccache really did not make much of an improvement to clang buildworld times.  If anyone wants to benchmark, clear the cache (`ccache -C`) and run two timed buildworlds.  The first should take about as long as without ccache, the second should show the improvement in speed.

Incidentally, setting CCACHE_COMPRESS=yes seems to have no speed penalties and fits more in the cache.


----------



## jozze (Jun 7, 2013)

Where are all these variables defined? I have been trying to find them in /usr/src, but they're not there.


----------



## kpa (Jun 7, 2013)

In the devel/ccache documentation. It doesn't make any sense to make the system sources and the build tools dependent on an external tool, that's why you won't see any mention of devel/ccache in the system sources or the build(7) manual page.


----------



## jozze (Jun 7, 2013)

Pardon my ignorance, but `$ man ccache` and files in /usr/share/doc/ccache folder describe no such variables as USE_CCACHE or USE_CCACHE_CPP2, which is why I am asking. There is CCACHE_CPP2, however. I am not sure, if the approach described in this how-to gives the expected results, because the /usr/local/share/doc/ccache/ccache-howto-freebsd.txt suggests a different way, to use devel/ccache:


```
#
# $FreeBSD: head/devel/ccache/files/ccache-howto-freebsd.txt.in 300896 2012-07-14 13:54:48Z beat $
#

To use ccache add the following to /etc/make.conf.
You can replace cc and c++ with the compilers of your choice.
(remember that only GCC and Clang can build world and kernel)

.if (!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*))
.if !defined(NOCCACHE)
CC:=${CC:C,^cc,/usr/local/libexec/ccache/world/cc,1}
CXX:=${CXX:C,^c\+\+,/usr/local/libexec/ccache/world/c++,1}
.endif
.endif

For Korn/Bourne shells Add the following to /etc/profile:
 export PATH=/usr/local/libexec/ccache:$PATH
 export CCACHE_PATH=/usr/bin:/usr/local/bin

For csh/tcsh Add the following to /etc/csh.cshrc:
 setenv PATH /usr/local/libexec/ccache:$PATH
 setenv CCACHE_PATH /usr/bin:/usr/local/bin

For icc users:
 Add /usr/local/intel_cc_80/bin to CCACHE_PATH

--

To use distcc:

For Korn/Bourne shells Add the following to /etc/profile:
 export CCACHE_PREFIX=distcc
 export DISTCC_HOSTS="localhost host1 host2"

For csh/tcsh Add the following to /etc/csh.cshrc:
 setenv CCACHE_PREFIX distcc
 setenv DISTCC_HOSTS "localhost host1 host2"

--

If you have a problem building world
define NOCCACHE and try again.

If you have a problem building a port
reset PATH=$CCACHE_PATH and try again.
...
```

Or am I missing something here?


----------



## jozze (Jun 7, 2013)

wblock@ said:
			
		

> Last time I tested, ccache really did not make much of an improvement to clang buildworld times.  If anyone wants to benchmark, clear the cache (`ccache -C`) and run two timed buildworlds.  The first should take about as long as without ccache, the second should show the improvement in speed.
> 
> Incidentally, setting CCACHE_COMPRESS=yes seems to have no speed penalties and fits more in the cache.



Yes, that's what I thought, because I didn't see any difference. So now, I made two consecutive buildworlds, but the impact was really great! I don't have sysutils/bsdadminscripts installed, I didn't set the USE_CCACHE or USE_CCACHE_CPP2 variables. What I did is described below.

I installed devel/ccache with these config flags enabled:


```
# make -C /usr/ports/devel/ccache pretty-print-config
+CLANGLINK +DOCS +LLVMLINK -STATIC
```

My make.conf and my src.conf. Notice that I don't have CCACHE_CPP2 enabled. After that I added these two variables as the /usr/local/share/ccache/ccache-freebsd.txt suggested:


```
# echo "setenv CCACHE_PATH /usr/bin:/usr/local/bin" >> ~/.tcshrc
# echo "setenv CCACHE_LOGFILE ~/.ccache.log" >> ~/.tcshrc
# echo "setenv CCACHE_COMPRESS 1" >> ~/.tcshrc
```

I decided to keep a log, if something goes wrong, and to confirm that I am really using ccache to do the job, and to make compression (it really paid off -- from ~600 MB it reduced cache size to 80 KB).

After that I ran

```
# cd /usr/src
# time make -j16 buildworld
# time make -j16 buildworld
```

The first time it took me ~40 min to build (from 22:24 to 22:04). The second time it took considerably less: ~13 minutes (from 22:14 to 22:27).


----------



## wblock@ (Jun 7, 2013)

I use a slightly different entry in make.conf:

```
.if (!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*)) && !defined(NOCCACHE)
  CC:=${CC:C,^cc,/usr/local/libexec/ccache/world/cc,1}
  CXX:=${CXX:C,^c\+\+,/usr/local/libexec/ccache/world/c++,1}
.endif
```

The idea being to avoid the overhead of the extra "if" by short-circuiting.  I have not tested whether that's actually an improvement.


----------



## jozze (Jun 7, 2013)

Thank you, that definitely makes it more readable, but I don't think it has an impact on performance, since I think this file is read only once on the beginning (by accident I once changed the compiler in it during compilation and nothing changed until the compilation ended).

UPDATE:
I noticed, after playing with devel/ccache that world can easily be built without CCACHE_CPP2 option, but the kernel will fail (fun fact).

But what does the devel/ccache actually do? I read through the manual but I don't quite understand how does saving time or binary content speed up the compiling process. Does it reuse old binaries?


----------



## wblock@ (Jun 25, 2013)

From http://ccache.samba.org/:


> ccache is a compiler cache. It speeds up recompilation by caching previous compilations and detecting when the same compilation is being done again.


----------



## kpa (Jun 25, 2013)

"Compilation" in this context means only the pre-processor macro definitions that are used abundantly in C. Caching them provides quite significant savings in compilation times because the definitions can be read from a hash table in nearly constant time.


----------



## jozze (Jun 25, 2013)

Oh, thanks @kpa! That's what was puzzling me, I thought that it was saving compilation results.


----------



## wblock@ (Jun 25, 2013)

I see where it's using preprocessor information for the hash, but not that those are the only things cached: http://ccache.samba.org/manual.html#_the_direct_mode.  It is still early in the morning, though.


----------



## ShelLuser (Oct 3, 2013)

Very cool thread. Up until now I never paid much attention to clang and its history, but since I'm in the process of upgrading to FreeBSD 9.2-RELEASE using the source tree I figured that I might as well look into the compiler as well. I like doing things "the way they're intended" 

I'm happy to say (though I'm probably stating the obvious) that it is perfectly doable to build the world and kernel of FreeBSD-9.2. The whole upgrade was pretty straightforward and without issues. I'm currently even using clang to rebuild some of my ports (just in case) and things work flawlessly.

Thanks for the great tips, much appreciated.

Edit: null-edit & respect the Clang! :h


----------



## pboehmer (Oct 8, 2013)

jozze said:
			
		

> After that I ran
> 
> ```
> # cd /usr/src
> ...



Thanks for the info.  

One caveat, I think you'll find your times less impressive if you `rm -rf /usr/obj/usr` and possibly `make clean` between your `time make -j16 buildworld` commands.  Since you are running them consecutively, I'm willing to bet that all you are actually building is just the world build tool-chain.  Since the rest of world was built in the first occurrence, your next build just checks to see if the existing objects in /usr/obj/usr are complete, right?


----------

