# Poudriere failures



## garyh (May 8, 2021)

Hi,

Since reinstalling the OS to 13 I have not been able to build llvm90 llvm10 or llvm11
I continually get messages about a runaway process. I have also noticed it taking a very long time to build any packages.

Has anyone else seen this or have a resolution for it?

Regards,

Gary


----------



## SirDice (May 8, 2021)

Those have been building fine for me. 

You might want to add `llvm*` to ALLOW_MAKE_JOBS_PACKAGES in /usr/local/etc/poudriere.conf to speed up the builds (depends on your system of course).


----------



## Denis Shaposhnikov (May 8, 2021)

garyh said:


> Since reinstalling the OS to 13 I have not been able to build llvm90 llvm10 or llvm11


Before the upgrade, did you use ccache?


----------



## Lamia (May 9, 2021)

Those are two of the best practices (ccache, ALLOW_MAKE_JOBS_PACKAGES)  for using Poudrière. 
There are several other recommendations. Are llvm* in your priority builds in the conf file? Contents of your make.conf - src.conf may also affect it regardless of them working flawlessly before until now. Perhaps, there is a blocker in them.
Runaway process may also need you extend the allocated time to build a PKG before failure. You may also need increase resources.


----------



## garyh (May 9, 2021)

To be clear, the OS has been installed from scratch, I have not used ccache and have a pretty much empty make.conf and src.conf, there is no priority set at all. When on 12 llvm10 took just over 6 hours to build, now on 13 and it is not complete after 36 hours.


----------



## SirDice (May 9, 2021)

Are you using ZFS with poudriere? Although it doesn't matter much for a single port it does matter when building a lot of ports. 

When poudriere is building one of the LLVMs have a look at top(1), is there only one core building? Or is the build spread over multiple cores? Did you set ALLOW_MAKE_JOBS_PACKAGES already?


----------



## garyh (May 9, 2021)

The whole system is using zfs but poudriere is not, as in no zfs options in the conf file, did not seem needed to me. I have set the ALLOW_MAKE_JOBS_PACKAGES to allow for all llvm ports.


----------



## Alain De Vos (May 9, 2021)

Playing Youtube music is currently better , non-interrupted, during poudriere builds.
My poudriere.conf file for a cpu with 8cores and 8G memory,

```
TIMESTAMP_LOGS=yes
HTML_TRACK_REMAINING=yes
USE_TMPFS=""
PARALLEL_JOBS=12
PREPARE_PARALLEL_JOBS=24
ALLOW_MAKE_JOBS=YES
ALLOW_MAKE_JOBS_PACKAGES="llvm10 rust qt5-webengine"
```
The CPU usage remains low ...


----------



## SirDice (May 9, 2021)

garyh said:


> poudriere is not, as in no zfs options in the conf file, did not seem needed to me.


It is. It uses ZFS to clone the build jails. So you really need to configure ZFS in the configuration file. Otherwise it's going to use cpio(1) to clone the base jails and this is quite an I/O intensive and slow process.


----------



## garyh (May 9, 2021)

Ok understood, but why would this behaviour only display in 13 and not 12? That's what I don't understand, the base jail has been installed from src cleanly after a src upgrade of the os, all config files are as they were on 12.


----------



## Alain De Vos (May 9, 2021)

"messages about a runaway process." , can you inform what is the exact message ?


----------



## garyh (May 10, 2021)

I'll let you know after it's completed, have recreated the jail to use ZFS and set it going again.


----------



## garyh (May 11, 2021)

I'm going to go out on a limb here and say this is not just limited to Poudriere, `make buildworld` is taking a long time now too, has been going for near on 36 hours, when on FBSD 12 it would take maybe 4 hours, starting to think there is something wrong with this install. Admittedly this is building on an Intel Core i3 2 core, 2 thread processor but this is very slow even for that processor.


----------



## Alain De Vos (May 11, 2021)

On a 8-core processor compiling chromium-browser on FreeBSD-12 takes a full day. So it depends...


----------



## SirDice (May 11, 2021)

garyh said:


> `make buildworld` is taking a long time now too, has been going for near on 36 hours, when on FreeBSD 12 it would take maybe 4 hours, starting to think there is something wrong with this install.


Yes, something seems off. No hard figures, but building world; 13-STABLE takes around the same time as building 12-STABLE on my ancient Core i5. Certainly not that much longer, it may even be a little quicker. 


garyh said:


> Admittedly this is building on an Intel Core i3 2 core, 2 thread processor but this is very slow even for that processor.


Even my Raspberry Pi 3 is able to build world in 36 hours and that's a really, really slow system.

How's your I/O doing? Compiling takes some CPU horsepower of course but it has to read and write a lot of intermediate files too. So slow I/O can have a really big impact.


----------



## Lamia (May 12, 2021)

It is expected that the first build - be it kernel/world or ports with Poudrière - would take a LONG time. 

Notwithstanding, something can be wrong. It is suggested to start on a clean slate for a new build particularly in a major release - new/update Poudrière, delete cache/*, delete /usr/obj/* etc. I would think that you have done all that.


----------



## garyh (May 12, 2021)

Not sure why it is expected for the first buildworld to take longer, even if that were true I have never seen a buildworld take longer than 8 hours on this system. It is indeed a fresh install of FreeBSD 13. I/O looks good, all disks are solid state, the layout is standard from install with a separate pool for /usr/home and /var/db/mysql which are on mirrored zpools each, poudriere is set up in /usr/home/poudriere as it's own dataset. Network performance looks good. IMAP access over the internet seems very responsive. I am at a loss.

Thank you all for your responses and help.


----------



## garyh (May 13, 2021)

Right so I have rebuilt world and kernel and installed both, recreated the build jail and things look a lot better, currently building llvm11 and at 40% after 3 hours 4minutes. So whatever the issue was it seems better now.

Thanks all.


----------



## Lamia (May 13, 2021)

garyh said:


> Right so I have rebuilt world and kernel and installed both, recreated the build jail and things look a lot better, currently building llvm11 and at 40% after 3 hours 4minutes. So whatever the issue was it seems better now.
> 
> Thanks all.


It is always irritating at first build.


----------



## Lamia (May 13, 2021)

You can achieve a better performance with those recommendations above - ccache, priority builds, etc.


----------



## garyh (May 13, 2021)

Yeah I have set ccache and priority builds but had already done that on the run before I blew away the poudriere data and jail directories and re set them up.


----------



## garyh (May 14, 2021)

All of my poudriere builds have finished in a time I would have expected now. I have a feeling there was some odd debugging issue in my original OS install. Seems to be cured after that src upgrade I performed, either that or the jails world and kernel were out of sync. 

Thanks to all that helped me resolve this.


----------

