# LLVM build times



## freebuser (Nov 5, 2021)

Is this normal? still hasn't finished.






Under virtualbox client with host as below:


----------



## a6h (Nov 5, 2021)

Yes. It is normal.

P.S.



freebuser said:


> Under virtualbox client with host as below:


The host specs are irrelevant. How much resource have you allocated for the guest? i.e. CPU/core, RAM, and HDD or SSD?

Some tips and tricks to boost performance, a little bit:

*VMM's guest setup:*
* Increase the number of processors.
* Allocate more memory.
* Use an SSD if it's possible.
* Don't use "Dynamically allocated" for Hard disk. Select the "fixed size".

*VMM's host setting (Windows)*

* Power management:
1. Run `powercfg.cpl`
2. Select one of the power profiles -- preferably the High performance, but it doesn't matter.
3. Select "change advanced power setting".

4. In the "process power management" section:
Set "minimum" to 100%.
Set "cooling policy" to "active".
Set "maximum" to 100%.

* Disable the swaping:
1. Run the `SYSDM.CPL`
2. Advanced tab | Virtual memory | Change
3. Set them all to "no paging file".

* Disable the "Virus & threat protection" if it is possible.

* To reduce background tasks -- some of them are heavy on the HDD:
1. Don't run windows programs when running VMM, and
2. Disable ComputerRestore on your VMM drive, e.g. C:
`Disable-ComputerRestore -Drive "C:\"`

* [NOT SURE] vmware runs as a service. But I'm not sure about vbox. I don't have it installed right now.
Do your own research, and if you find out that it is run as service then change these settings too:

1. Run the `SystemPropertiesAdvanced.exe`
2. Advanced | Performance | Settings | Processor scheduler
3. Set it to "Background service".

*Guest OS (FreeBSD):*
* Set the `dev.cpu.0.freq` to the max or turbo.
* Change the `kern.hz` in the /boot/loader.conf:

```
kern.hz=100
```


----------



## freebuser (Nov 5, 2021)

vigole said:


> Yes. It is normal.
> 
> P.S.
> 
> ...........


Thanks for the comprehensive response.

Majority of them are already set properly, I will change any remaining setting too and see if there's any improvement.

I also noticed this thread and change the config too.
https://forums.freebsd.org/threads/bare-minimum-build-of-llvm.77367/#post-481483


----------



## diizzy (Nov 5, 2021)

Since you're running Windows 10 Pro I can highly recommend Hyper-V (included) over Virtualbox.


----------



## a6h (Nov 5, 2021)

diizzy said:


> Since you're running Windows 10 Pro I can highly recommend Hyper-V (included) over Virtualbox.


Hyper-V is the best option, esp. for Windows/NT guests. But I had a problem using FreeBSD on the Hyper-V.
emulators/open-vm-tools didn't work. The port is necessary for changing the resolution (GUI). I'm not aware of current status.

When I was comparing vmware and vbox, I noticed that vmware was using less resources -- both RAM and CPU.
It wasn't a precise comparison though. Personally, I prefer to use vmware player. But it (the free version) doesn't have snapshot feature.


----------



## freebuser (Nov 5, 2021)

vigole said:


> Hyper-V is the best option, esp. for Windows/NT guests. But I had a problem using FreeBSD on the Hyper-V.
> emulators/open-vm-tools didn't work. The port is necessary for changing the resolution (GUI). I'm not aware of current status.
> 
> When I was comparing vmware and vbox, I noticed that vmware was using less resources -- both RAM and CPU.
> It wasn't a precise comparison though. Personally, I prefer to use vmware player. But it (the free version) doesn't have snapshot feature.



Is that to run in desktop mode? I only ssh to the Freebsd guest, rarely work directly on the guest.


----------



## a6h (Nov 5, 2021)

freebuser said:


> Is that to run in desktop mode? I only ssh to the Freebsd guest, rarely work directly on the guest.


Yes. If you don't want to use GUI/Desktop, Hyper-V is better -- for SSH-ing, etc.


----------



## a6h (Nov 5, 2021)

By the way, enabling Hyper-V conflicts with Android AVD. If by any chance you are using Android AVD, then you have to setup a dual-boot system.
One for the Hyper-V, i.e. `hypervisorlaunchtype on` and one for non hyper-V usage, i.e. `hypervisorlaunchtype off`.


----------



## diizzy (Nov 5, 2021)

freebuser said:


> Is that to run in desktop mode? I only ssh to the Freebsd guest, rarely work directly on the guest.


It works great for CLI


----------



## SirDice (Nov 5, 2021)

You also want to change this setting in poudriere.conf:

```
# List of packages that will always be allowed to use MAKE_JOBS
# regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
# which holdup the rest of the queue to build more quickly.
ALLOW_MAKE_JOBS_PACKAGES="pkg llvm* gcc* rust firefox node* mame"
```
That way the LLVM builds will use more than one core which will greatly reduce their build times. I've added a couple of other ports too, as they were regularly holding up the queue.


----------



## diizzy (Nov 5, 2021)

SirDice said:


> You also want to change this setting in poudriere.conf:
> 
> ```
> # List of packages that will always be allowed to use MAKE_JOBS
> ...


Unless you're really memory constrained you can just set like -j2 or -j3 something globally


----------



## SirDice (Nov 5, 2021)

diizzy said:


> Unless you're really memory constrained you can just set like -j2 or -j3 something globally


You could set this:

```
# By default MAKE_JOBS is disabled to allow only one process per cpu
# Use the following to allow it anyway
# ALLOW_MAKE_JOBS=yes
```
But it's not always that beneficial, it depends on your builds and the system itself. Sometimes it's better to have 4 jobs all compiling on a single core (on a 4 core machine) and finish at the same time than having 4 jobs running each using all 4 cores (and thus steal each others resources) and taking a lot longer to complete.


----------



## freebuser (Nov 5, 2021)

SirDice said:


> You also want to change this setting in poudriere.conf:
> 
> ```
> # List of packages that will always be allowed to use MAKE_JOBS
> ...



I did that and still only one builder is compiling. Is there any other setting I need to change for this to work?


----------



## SirDice (Nov 5, 2021)

freebuser said:


> I did that and still only one builder is compiling.


How many cores does the machine have? By default it will spawn a number of jobs depending on the core count.


```
# parallel build support.
#
# By default poudriere uses hw.ncpu to determine the number of builders.
# You can override this default by changing PARALLEL_JOBS here, or
# by specifying the -J flag to bulk/testport.
#
# Example to define PARALLEL_JOBS to one single job
# PARALLEL_JOBS=1

# How many jobs should be used for preparing the build? These tend to
# be more IO bound and may be worth tweaking. Default: PARALLEL_JOBS * 1.25
# PREPARE_PARALLEL_JOBS=1
```

Another thing that can happen is that this port is a dependency for a bunch of other ports. Those will all need to wait until that dependency is built. If you regularly have a port that's holding up the queue you probably want to add it to ALLOW_MAKE_JOBS_PACKAGES so it gets built as fast as possible.


----------



## freebuser (Nov 5, 2021)

SirDice said:


> How many cores does the machine have? By default it will spawn a number of jobs depending on the core count.



6 cores (12 threads). Just checked the resource monitor it seems it is using 6 threads (I specified this using J6 in poudriere.)
So it seems working but does not reflect in the web monitoring.


----------



## Alain De Vos (Nov 5, 2021)

I was confused at first myself, but then i realised not every port can be build parallel for simple reasons upstream.


----------



## angry_vincent (Nov 5, 2021)

poudriere also has ability to use precompiled binaries, look for `poudriere bulk -b`
I use it specifically for devel/llvm and lang/rust so that notebook is not stressed by compiling both from source.


----------



## SirDice (Nov 5, 2021)

angry_vincent said:


> poudriere also has ability to use precompiled binaries, look for `poudriere bulk -b`


Only available on ports-mgmt/poudriere-devel at the moment. Not available yet on ports-mgmt/poudriere. Unless this has recently been updated, it wasn't a while ago.


----------



## angry_vincent (Nov 5, 2021)

Yes, i was referring to poudriere-devel only.


----------



## a6h (Nov 5, 2021)

Setting ccache(1) (CCACHE_DIR) may help too.


----------



## freebuser (Nov 5, 2021)

SirDice said:


> How many cores does the machine have? By default it will spawn a number of jobs depending on the core count.
> 
> 
> ```
> ...





vigole said:


> Setting ccache(1) (CCACHE_DIR) may help too.



Setting -J6 and changing the options to minimal and setting ccache I was able to compile the LLVM11 in just over an hour, this is an improvement from nearly 16hrs of compiling of LLVM11 before.


----------



## a6h (Nov 6, 2021)

freebuser said:


> compile the LLVM11 in just over an hour


That's about 12 times faster than my machine! congratulation.


----------



## astyle (Nov 6, 2021)

One thing I'm noticing from OP's screenshot is that he's got LLVM 11 and LLVM 12 compiling at the same time.  I'd say that's problematic. On my profile page, one of the comments there (Date July 3, 2021) mentions that the `-jN` flag only works if ALL deps have been satisfied for a given port. Another comment from the same thread also mentions that the `-jN` flag (where N=4) for LLVM 10 cuts the compile time down from 2.5 hours to 50 minutes, a 3-fold improvement. My processor is a Ryzen 5 1400 3.4 Ghz. Would be nice if I could link to specific profile posts...


----------



## Alain De Vos (Nov 6, 2021)

Your post explains my experiences.  I have "ALLOW_MAKE_JOBS=yes" & "PARALLEL_JOBS=numberofcores+1". 
llvm takes 16 hours for me. But that's ok , because on the other cores other packages are being compiled simultaneously.


----------



## argwings (Nov 6, 2021)

After posting earlier today about my port upgrading practices I decided to take a look at poudriere. llvm12 seems to also only use one job for me:

```
[130i386-default] [2021-11-05_18h19m44s] [parallel_build:] Queued: 108 Built: 103 Failed: 0   Skipped: 0   Ignored: 0   Tobuild: 5    Time: 05:52:41
        [04]: devel/llvm12              | llvm12-12.0.1_6           build           (05:04:48 / 05:17:12)
[05:53:15] Logs: /usr/local/poudriere/data/logs/bulk/130i386-default/2021-11-05_18h19m44s
```
(Me trying to copy top output, LOL)

```
CPU: 25.4% user,  0.0% nice,  2.4% system,  0.7% interrupt, 71.5% idle
```

At one point it was building gcc10 simultaneously but GCC finished hours before, so I dunno.


----------



## freebuser (Nov 6, 2021)

Alain De Vos said:


> Your post explains my experiences.  I have "ALLOW_MAKE_JOBS=yes" & "PARALLEL_JOBS=numberofcores+1".
> llvm takes 16 hours for me. But that's ok , because on the other cores other packages are being compiled simultaneously.



But in my case other ports complete compilation way before LLVM finish, so even if I take parallel jobs still LLVM will take at least 12 to 14 hours more.

I think both setting minimal options for the port config and using multiple threads brought down the compiling time.


----------



## freebuser (Nov 6, 2021)

argwings said:


> After posting earlier today about my port upgrading practices I decided to take a look at poudriere. llvm12 seems to also only use one job for me:
> `[130i386-default] [2021-11-05_18h19m44s] [parallel_build:] Queued: 108 Built: 103 Failed: 0   Skipped: 0   Ignored: 0   Tobuild: 5    Time: 05:52:41
> [04]: devel/llvm12              | llvm12-12.0.1_6           build           (05:04:48 / 05:17:12)
> [05:53:15] Logs: /usr/local/poudriere/data/logs/bulk/130i386-default/2021-11-05_18h19m44s`
> ...



Did you use the -J flag in the command line?


----------



## argwings (Nov 6, 2021)

Looking at the Makefile, it could be this: https://reviews.llvm.org/D72402


----------



## argwings (Nov 6, 2021)

freebuser said:


> Did you use the -J flag in the command line?


Hmmm, no, I skimmed poudriere.conf, and saw that it defaulted to hw.ncpu builders, which I thought would equate to -j4 for me but I guess builders aren't the same as jobs.


----------



## astyle (Nov 6, 2021)

I'd rather build ports sequentially, one after another. Of course, deps need to be taken care of first. Unfortunately, make is not smart enough to balance (dependency resolving) with the idea that (several ports can be compiled at the same time on different CPU cores). And at this time, I haven't studied parallel computing enough to use terms like "parallellism" or "concurrency" with any kind of confidence that the terms are used correctly. But even with that level of knowledge, I'm seeing a conflation of what can be expected from a batch compilation, and the resulting confusion when looking at available numbers.


----------



## Alain De Vos (Nov 6, 2021)

I don't know anything about parallelized compilation.

But an experiment is the following.
In make.conf

```
MAKE_JOBS_NUMBER=555
```
When you do ps you will see no 555 c++ processes. 
Can anyone explain.

Ooh when you put
in poudriere.conf

```
PARALLEL_JOBS=555
```
Poudriere will spawn 555 builders. ... And each builder might contain a c++ proces


----------



## astyle (Nov 6, 2021)

Alain De Vos said:


> In make.conf
> 
> ```
> MAKE_JOBS_NUMBER=555
> ...


your i7 only has 8 cores. So the `-j` flag can only be set to the max value of 8, not 555.




Alain De Vos said:


> Ooh when you put
> in poudriere.conf
> 
> ```
> ...


`PARALLEL_JOBS=555` means processes will be spawned, but they will still be competing for one of your 8 cores. NOT the same thing as the `-j` flag.

This is why I keep saying, `make` is not that smart.


----------



## VladiBG (Nov 10, 2022)

Here's some statistic of building LLVM13 with and without ccache on synth

LLVM on parallel build with rust ~39min with j4 without ccache (16GB ram / 12 vCPU 13th gen) (hyper-v guest)







standalone build of llvm13 j8 with clean ccache (24GB RAM / 24vCPU i7-13700k @ 5.4) ~50% load 8 threads (hyper-v guest)





Second build of llvm13 with ccache. (24GB RAM (5GB ccache) / 24vCPU i7-13700k @ 5.4) ~50% load 8 threads (hyper-v guest)





And here's some interesting reading for AWS






						Building LLVM in 90 seconds using Amazon Lambda - Made of Bugs
					






					blog.nelhage.com


----------



## Alain De Vos (Nov 10, 2022)

Using poudriere for me build times are around 14-hours.
[Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz]


----------



## VladiBG (Nov 10, 2022)

On my old i7-2600k @ 3.4GHz /12GB RAM the build time was ~6hours

14 hours on i7-3770 is too much maybe you have another bottleneck. I'm building on top of regular HDD in raid5 4disk/500GB WD Gold so they are not performing fast as the cheapest SSD and also slow down the build process especially if you are using poudriere with zfs.


----------



## SirDice (Nov 11, 2022)

VladiBG said:


> On my old i7-2600k @ 3.4GHz /12GB RAM the build time was ~6hours


Similar numbers here. 06:55:55 for LLVM13 on my lowly i5-3470.


----------

