# Troubleshooting a hung Poudriere



## astyle (Nov 25, 2021)

I'm trying to use Poudriere to bulk-build a successful base install on host A. Basically, I have an install with a very bare-bones KDE desktop (no Konsole, or anything like that, but it starts Wayland successfully). So, I'm trying to use Poudriere to bulk-build the packages, populate a repo that I prepared, so that I can replicate the install from host A just by pointing host B's pkg to the repo on A.

So far, so good, but - Poudriere process got hung 3 packages short of completing the build. So, with help from awk and grep, I managed to figure out that those 3 packages are: 

www/qt5-webengine
www/qt5-webview
x11-toolkits/plasma5-kdeplasma-addons
I tried rebooting my build rig and restarting the Poudriere build. Fortunately, Poudriere was smart enough to pick up just those 3 ports that still need to be built. But it's taking so long, that I'm beginning to ponder some 'cheat moves'. The rationale behind them is:

```
# poudriere ports -l
default   null   2021-11-23 17:10:39 /usr/ports/poudriere_tree
```
And thing is, I already have all those ports built in the regular /usr/ports tree. I have my reasons for the redundancy here. So, my thinking goes - since I already have the stuff built in the regular ports tree, maybe I can copy the ports directory (for the missing ports) from regular ports tree into Poudriere's 'default' tree. I don't run make clean in the ports directory, so all compiling work is intact 

Any pitfalls to watch out for?
I know that www/qt5-webengine takes forever to compile even on a decent CPU. I think on mine (Quad-core Ryzen 5 1400 3.4 GHz) it took at least 3 hours.


----------



## Alain De Vos (Nov 25, 2021)

Compiling gcc,llvm, some qt packages can take a long time.  Poudriere only uses one core for each. Compiling llvm on my pc takes 18hours


----------



## SirDice (Nov 25, 2021)

astyle said:


> I don't run make clean in the ports directory, so all compiling work is intact


Poudriere uses a different WRKDIRPREFIX to build ports.

```
---Begin make.conf---
USE_PACKAGE_DEPENDS=yes
BATCH=yes
WRKDIRPREFIX=/wrkdirs
PORTSDIR=/usr/ports
PACKAGES=/packages
DISTDIR=/distfiles
FORCE_PACKAGE=yes
PACKAGE_BUILDING=yes
PACKAGE_BUILDING_FLAVORS=yes
```


----------



## SirDice (Nov 25, 2021)

astyle said:


> I know that www/qt5-webengine takes forever to compile even on a decent CPU. I think on mine (Quad-core Ryzen 5 1400 3.4 GHz) it took at least 3 hours.


Try adding it to `ALLOW_MAKE_JOBS_PACKAGES` in poudriere.conf. That might help speed up the build.


----------



## astyle (Nov 25, 2021)

Thanks for your replies, everyone! This is really helpful.
--
What I did this time around is just reboot and try to let Poudriere finish on another overnight run. The 'cheat moves' I was considering - that was in case that reboot didn't work. However, it looks like it did work - according to logs, after another nearly 5 hours.

Total for my processor: 28 hours, 48 minutes, and 26 seconds.... Good thing this is done. Update sets (I only plan to update plasma5 and KF5) shouldn't take nearly this long. including the entire QT5 set - that was an upfront investment of processor effort, because until the KDE project switches to QT6.2, they're building against that same QT 5.15.2 set.


----------



## SirDice (Nov 25, 2021)

astyle said:


> What I did this time around is just reboot and try to let Poudriere finish on another overnight run. The 'cheat moves' I was considering - that was in case that reboot didn't work. However, it looks like it did work - according to logs, after another nearly 5 hours.


If you break off a build (for whatever reason) it's going to continue where it left off when you start it again. Thankfully, it doesn't start completely over from scratch. You can force a bulk build from scratch, that's sometimes necessary if you've been mucking around with dependency changes a lot. That's what `-c` is for. See poudriere-bulk(8):

```
-C       Clean only the packages specified on the command line or in the
              file given by -f file.  Implies -c for -a.

     -c       Clean all previously built packages and logs.
```


----------



## astyle (Nov 25, 2021)

Yeah, that's useful if I change my mind in the middle of a build. poudriere.conf does offer something nice that I plan to make use of: `KEEP_OLD_PACKAGES=yes`, and the default is up to 5 versions back. 

Right now, my next step is figuring out how to manage those versions to make it easier on pkg on other machines. Once I point pkg to my repo, I'd want to specify a version to pass to `pkg`, like v.0 for baseline, v1. for upgrades, etc.


----------



## SirDice (Nov 25, 2021)

The initial build is always the most time consuming. It has to build _everything_ it needs, all build, run and library dependencies for everything on your package wish list. Bigger, more cores, more memory, faster storage will all help speed up the process. Your next update, after you updated the ports tree, it will only (re)build what is needed. It will use the packages from the previous builds as much as possible.



astyle said:


> Once I point pkg to my repo, I'd want to specify a version to pass to `pkg`, like v.0 for baseline, v1. for upgrades, etc.


You can create symlinks in the /usr/local/poudriere/data/packages directory:

```
root@molly:/usr/local/poudriere/data/packages # ll
total 112
drwxr-xr-x  7 root  wheel  18 Oct  9 19:17 122-release-server/
drwxr-xr-x  5 root  wheel  16 Nov 22 14:12 13-stable-desktop/
drwxr-xr-x  7 root  wheel  18 Nov 21 23:42 13-stable-server/
drwxr-xr-x  7 root  wheel  18 Nov 22 09:25 130-release-desktop/
drwxr-xr-x  7 root  wheel  18 Nov 21 23:10 130-release-server/
lrwxr-xr-x  1 root  wheel  18 Nov  7  2020 FreeBSD:12:amd64@ -> 122-release-server
lrwxr-xr-x  1 root  wheel  18 Feb 15  2021 FreeBSD:13:amd64@ -> 130-release-server
```

On the same host I also have nginx running. There's a sample configuration for poudriere in /usr/local/share/examples/poudriere/. I added a /packages location to it pointing at /usr/local/poudriere/data/packages. 

I added two symlinks so URLs in my repository URL can use `${ABI}`, that was useful when I upgraded my machines from 12.2 to 13.0. The other repositories have predictable names due to the way I used different ports trees and jail names. You can probably figure out some structure that'll work for you.


----------



## grahamperrin@ (Dec 2, 2021)

SirDice said:


> See poudriere-bulk(8):



NB that's the page for ports-mgmt/poudriere*-devel*, not for ports-mgmt/poudriere – not that it matters for option `-c`, but generally, with poudriere, it's good practice to refer to the manual pages at <https://github.com/freebsd/poudriere/wiki>.


----------

