# Poudriere bulk failure



## Brian546 (May 1, 2019)

I have two separate servers, one of them with an intel i3 processor and one with an i7.

Poudriere is set up exactly the same way on both. Host OS is 12.0-RELEASE. UFS only. The poudriere.conf file is set up the same on both. Jails on each were created to build arm64 ports, downloading releng/12.0 via SVN. Same exact packages to be built on both. Options on those packages have been set the same. The ONLY difference in the bulk build is that on the i3 system it actually builds. On the i7 system running a bulk build immediately crashes with a bunch of "UNAME_r and OSVERSION (1200086) do not agree on major version number." messages and an "Error: Error looking up pre-build ports vars" message. I cannot for the life of me figure it out. I've deleted and rebuilt the jail from scratch. Same thing. I've deleted the Poudriere ports tree and re-created it. Reconfigured port options. Same result. Every single thing I've done on the i3 system I've done a carbon copy of on the i7.

The only difference I see between the two systems other than the i3/i7 processors is the hard drives. On the i3 system it is all on one hard drive. On the i7 system there is a secondary hard drive where /usr/local is mounted, and a swap partition as well. I don't think that should make a difference though.

I've done some searching on these error messages. Anything I've found about them says it's been fixed, but apparently not. Anyone run into this before?


----------



## hukadan (May 2, 2019)

Could you post :

the content of the default settings in the /etc/login.conf file of the poudriere jail, in particular the *setenv* part;
the value of *__FreeBSD_version* in the /usr/include/sys/param.h  file of your host.


----------



## SirDice (May 2, 2019)

Brian546 said:


> On the i7 system running a bulk build immediately crashes with a bunch of "UNAME_r and OSVERSION (1200086) do not agree on major version number." messages and an "Error: Error looking up pre-build ports vars" message. I cannot for the life of me figure it out. I've deleted and rebuilt the jail from scratch.


I think the issue might be the host itself. Make sure it's up to date and has the same or newer version of FreeBSD than your jails. Also make sure poudriere itself is updated.


----------



## Brian546 (May 2, 2019)

Thank you for your replies!

Hukadan I'll post the content of those files when I get to work.

SirDice, the hosts are both running 12.0-release. I just find it strange that it crashes on one machine but not the other. I've tried building the jails with both release/12.0.0 and releng/12.0. On the i3 system either jail works fine. On the i7 system they both get the same error messages. But I'll gladly download an updated /usr/src on the i7 host and build.

Edit: Yes also poudriere and all dependencies are up to date on both host systems.


----------



## Brian546 (May 2, 2019)

The setenv part of /etc/login.conf from the jail:
 :setenv=MAIL=/var/mail/$,BLOCKSIZE=K,UNAME_r=12.0-RELEASE-p3,UNAME_v=FreeBSD 12.0-RELEASE-p3 1200086,OSVERSION=1200086,ABI_FILE=/usr/lib/crt1.o,UNAME_m=arm64,UNAME_p=aarch64:\
#       :setenv=MAIL=/var/mail/$,BLOCKSIZE=K,UNAME_r=12.0-RELEASE-p3,UNAME_v=FreeBSD 12.0-RELEASE-p3 1200086,OSVERSION=1200086,ABI_FILE=/usr/lib/crt1.o,UNAME_m=arm64,UNAME_p=aarch64:\

The value of __FreeBSD_version from /usr/include/sys/param.h on the host itself:
#undef __FreeBSD_version
#define __FreeBSD_version 1200086       /* Master, propagated to newvers */

I ran diff on those files from both servers, and there was no difference.


----------



## hukadan (May 2, 2019)

Could you show the output of `freebsd-version -kru` on the host ?


----------



## Brian546 (May 2, 2019)

Ran the command on both hosts and the output was the same:

12.0-RELEASE
12.0-RELEASE
12.0-RELEASE

It's not really that important but we would like to obviously start building the ports on the i7 so we won't need the i3 anymore. Just for the heck of it though, I pulled the hard drives out of the i7 machine. Then installed a single hard drive, installed FreeBSD 12-Release again, but this time it's just on the single drive. Just finished building poudriere and now building the poudriere jail for the arm ports. We'll see if the results are different when I start the bulk build.


----------



## hukadan (May 2, 2019)

I hope someone more knowledgeable than me can help you. I would be curious to know what the problem (and the solution) is.


----------



## Brian546 (May 2, 2019)

Just tried to run the bulk build on the i7 and the same error messages happened. Sigh.. for some reason FreeBSD doesn't seem to like the machine. FYI it's a Dell Precision T1650.


----------



## Vull (May 2, 2019)

When I run `freebsd-version -kru` on my 12.0-RELEASE install, I get:
	
	



```
12.0-RELEASE-p3
12.0-RELEASE-p3
12.0-RELEASE-p3
```
Would it help to run `freebsd-update fetch` and `freebsd-update install`? Might be worth a try.


----------



## hukadan (May 2, 2019)

Someone was right from the start...


SirDice said:


> Make sure it's up to date and has the same or newer version of FreeBSD than your jails.


It is just poudriere(8) complaining that the host is lagging behind the jail. This is wrong, see post below.


----------



## Brian546 (May 2, 2019)

The jail I compiled after the rebuild was just 12.0-RELEASE so it matches the version of the host, still had the errors.

Also the system with the i3 running 12.0-RELEASE builds the packages just fine whether or not the jail is 12.0-RELEASE or 12.0-RELEASE-p3

Poudriere might have a problem if the host is a major version behind though. 

Still, I'll give it a try.. I've never run freebsd-update. Normally I download the sources and build on the host so I'll do that with releng/12.0. I'll let you know how it goes, but for now time to go home and relax. Thanks for your help everybody!


----------



## hukadan (May 3, 2019)

Brian546 said:


> Poudriere might have a problem if the host is a major version behind though.


I checked out of curiosity. It warns you but lets you do it


> [00:00:03] Warning: !!! Jail is newer than host. (Jail: 1300021, Host: 1200508) !!!
> [00:00:03] Warning: This is not supported.
> [00:00:03] Warning: Host kernel must be same or newer than jail.
> [00:00:03] Warning: Expect build failures.


----------



## Brian546 (May 3, 2019)

Ahhhh ok thanks. Well still, I'm upgrading the host and we'll see what happens. Building the world now......


----------



## SirDice (May 3, 2019)

Brian546 said:


> Poudriere might have a problem if the host is a major version behind though.


Poudriere uses jail(8). A jail can only have the same or a lower version than the host. A higher version jail than the host is NOT supported and will cause problems.


----------



## Brian546 (May 3, 2019)

Finished building/installing world and the kernel, rebooted, mergemaster.. etc. 
uname -a shows my custom kernel 12.0-RELEASE-P3

ran poudriere bulk on the 12.0-RELEASE jail and it failed with the same error messages. Sigh, I have other work to do though. I'll come back to it next week. 

Again thanks for all your help everybody!


----------



## acheron (May 4, 2019)

Are you building for aarch64? If so, copy /usr/local/bin/qemu-aarch64-static from your working host to your non working host (and don't upgrade qemu on your working host)

see also https://forums.freebsd.org/threads/...-but-it-does-not-right-now.70634/#post-425852


----------



## acheron (May 4, 2019)

I put a fix in https://github.com/seanbruno/qemu-bsd-user/issues/74#issuecomment-489339733


----------



## Brian546 (May 5, 2019)

Hello Acheron, it's interesting though that qemu would work properly on one host but not the other. Poudriere and all dependencies along with qemu are up to date on both hosts.

However I will be glad to try what you suggested on Monday. Thank you for your help and I'll let you know if it works!


----------



## acheron (May 6, 2019)

Fixed in https://svnweb.freebsd.org/ports?view=revision&revision=500819. Rebuild glib20 and qemu.


----------



## Brian546 (May 6, 2019)

Hi Acheron,

So I updated the ports tree of the host. 
Rebuilt glib20 and qemu.
Updated the default ports tree of poudriere
Updated any unset port options
Then tried to rebuild the ports but it resulted in the same error messages. Do I need to rebuild the jail as well?


```
root@port-compiler:/usr/local/etc/poudriere.d # poudriere bulk -j aarch64 -f portslist
[00:00:00] Cross-building ports for arm64.aarch64 on amd64 requires QEMU
[00:00:00] Creating the reference jail... done
[00:00:11] Mounting system devices for aarch64-default
[00:00:11] Mounting ports/packages/distfiles
[00:00:11] Using packages from previously failed build: /usr/local/poudriere/data/packages/aarch64-default/.building
[00:00:11] Mounting packages from: /usr/local/poudriere/data/packages/aarch64-default
[00:00:11] Copying /var/db/ports from: /usr/local/etc/poudriere.d/aarch64-options
[00:00:11] Setting up native-xtools environment in jail... done
[00:00:11] Raising MAX_EXECUTION_TIME and NOHANG_TIME for QEMU from QEMU_ values
[00:00:11] Copying latest version of the emulator from: /usr/local/bin/qemu-aarch64-static
/etc/resolv.conf -> /usr/local/poudriere/data/.m/aarch64-default/ref/etc/resolv.conf
[00:00:11] Starting jail aarch64-default
Abort trap (core dumped)
Abort trap (core dumped)
Abort trap (core dumped)
Abort trap (core dumped)
Abort trap (core dumped)
make: "/usr/ports/Mk/bsd.port.mk" line 1154: warning: "/usr/bin/uname -s" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1159: warning: "/usr/bin/uname -r" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1203: UNAME_r () and OSVERSION (1200086) do not agree on major version number.
make: "/usr/ports/Mk/bsd.port.mk" line 1154: warning: "/usr/bin/uname -s" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1159: warning: "/usr/bin/uname -r" returned non-zero status
make: "/usr/ports/Mk/bsd.port.mk" line 1203: UNAME_r () and OSVERSION (1200086) do not agree on major version number.
Abort trap (core dumped)
Abort trap (core dumped)
[00:02:07] Logs: /usr/local/poudriere/data/logs/bulk/aarch64-default/2019-05-06_09h20m57s
[00:02:07] Loading MOVED for /usr/local/poudriere/data/.m/aarch64-default/ref/usr/ports
make: "/usr/ports/Mk/bsd.port.mk" line 1154: warning: "/usr/bin/uname -s" exited on a signal
make: "/usr/ports/Mk/bsd.port.mk" line 1159: warning: "/usr/bin/uname -r" exited on a signal
make: "/usr/ports/Mk/bsd.port.mk" line 1203: UNAME_r () and OSVERSION (1200086) do not agree on major version number.
[00:02:40] Error: Error looking up pre-build ports vars
[00:02:40] Cleaning up
[00:02:40] Unmounting file systems
```


----------



## acheron (May 6, 2019)

No need to update the jails.
Abort trap (core dumped), are you sure that glib20 and qemu-user-static were rebuilt and reinstalled?


----------



## Brian546 (May 6, 2019)

Yes after updating the ports tree I went into both of those directories and ran make deinstall reinstall.

Then ran pkg version and all packages on the host show =


----------



## Brian546 (May 6, 2019)

Tried restarting the qemu_user_static service and ran the bulk build. I got the same error messages except this time no abort trap messages.


----------



## acheron (May 6, 2019)

If you trust me, use this binary. If it works, you need to make sure your ports tree is up to date and rebuild glib20/qemu.


----------



## Brian546 (May 6, 2019)

There we go, and the ports are building now. Thank you Acheron and everybody else for your help!


----------



## CrunchBerry (May 8, 2019)

just curious how things are going with your package building. working off current I had issues with my favorite ports gettext-runtime and libiconv . They hung poudriere during configure. would not get past "checking if // distinct from /" I'm updating my tree right now to update my bhyve and poudriere jails. to try again, but no luck with poudriere nor poudriere-devel for r346657 . I did notice pkg.freebsd.org does have packages for arm64 for 13 at least for libiconv, but not sure which tree revision they are building against or I'd use it. still not sure its just something I got wrong on my machine. plenty of other packages built fine for me, just those two hung up. also noticed the runaway process timeout setting in poudriere.conf seems to actually use twice the value at lease in poudriere, didn't let it time out in poudriere-devel. 

I've also considered dropping back to 12-Stable in at least poudriere jail, but i figure wifi drivers will hit current first, when they come out, and I don't mind experimenting.


----------



## acheron (May 8, 2019)

CrunchBerry said:


> just curious how things are going with your package building. working off current I had issues with my favorite ports gettext-runtime and libiconv . They hung poudriere during configure.


qemu is full of bugs, try this https://reviews.freebsd.org/D18835


----------



## Brian546 (May 8, 2019)

Just checked on the arm64 ports we built. Keep in mind we are really only building Xorg, Openbox, Chromium, and Fbpanel, and of course all their dependencies.

I didn't run into any of the checking // vs / messages you were receiving but the Chromium build did fail.

qemu: uncaught target signal 4 (Illegal instruction) - core dumped
transport_security_state_generator failed with exit code -4
ninja: build stopped: subcommand failed.
*** error code 1


----------



## CrunchBerry (May 9, 2019)

Brian, If you could get Xorg built you made it much further then I could my first stab. but good to here things went reasonably well for you.

Acheron, once again you rock, libiconv built fine with that tip. Wish I found it, maybe have to go back to google over duckduckgo... Now I'll have to read up on poudriere hooks to make sure this stays set for me between ports updates on the off change config.site is changed upstream in the future. or as i look over the available triggers a simple sh script for my poudriere ports update to check for it being set after the update. I'm glad I didn't try to skip this check by finding out a way to force "cross_building=yes" which would make the test basically do the same thing according to the m4 macros. Course that method would likely have broken all sorts of other things in different ways.  That was the only thing I could come up with the other night, but it seemed like a bad idea all around.

Here's to giving Xorg another go...


----------



## acheron (May 9, 2019)

Brian546 said:


> Just checked on the arm64 ports we built. Keep in mind we are really only building Xorg, Openbox, Chromium, and Fbpanel, and of course all their dependencies.
> 
> I didn't run into any of the checking // vs / messages you were receiving but the Chromium build did fail.
> 
> ...


It's probably crashing when accessing the ID_AA64PFR0_EL1 register. There are 3 files to change:
`grep ID_AA64ISAR0_EL1 /usr/ports/www/chromium/files/*`
replace id_aa64isar0 = READ_SPECIALREG(ID_AA64ISAR0_EL1); with id_aa64isar0 = 0;


----------



## acheron (May 9, 2019)

CrunchBerry said:


> Brian, If you could get Xorg built you made it much further then I could my first stab. but good to here things went reasonably well for you.
> 
> Acheron, once again you rock, libiconv built fine with that tip. Wish I found it, maybe have to go back to google over duckduckgo... Now I'll have to read up on poudriere hooks to make sure this stays set for me between ports updates on the off change config.site is changed upstream in the future. or as i look over the available triggers a simple sh script for my poudriere ports update to check for it being set after the update. I'm glad I didn't try to skip this check by finding out a way to force "cross_building=yes" which would make the test basically do the same thing according to the m4 macros. Course that method would likely have broken all sorts of other things in different ways.  That was the only thing I could come up with the other night, but it seemed like a bad idea all around.
> 
> Here's to giving Xorg another go...


antoine has committed a workaround (for -current), you can backport the following changes:








						Add usr/bin/wc to HLINK_FILES to workaround casper emulation issues i… · freebsd/poudriere@2772844
					

…n qemu-user




					github.com
				








						[base] Revision 347334
					






					svnweb.freebsd.org


----------



## Brian546 (May 9, 2019)

Thank you Acheron,

I edited those 3 files like you said but now have a new compile error. I think we must be getting closer though:


```
RuntimeError: ../../third_party/node/freebsd/node-freebsd-x64/bin/node ../../third_party/node/node_modules/polymer-css-build/bin/polymer-css-build --polymer-version 2 --no-inline-includes -f /wrkdirs/usr/ports/www/chromium/work/chromium-73.0.3683.103/out/Release/gen/chrome/browser/resources/md_history/bundled/app.vulcanized.html /wrkdirs/usr/ports/www/chromium/work/chromium-73.0.3683.103/out/Release/gen/chrome/browser/resources/md_history/bundled/lazy_load.vulcanized.html -o /wrkdirs/usr/ports/www/chromium/work/chromium-73.0.3683.103/out/Release/gen/chrome/browser/resources/md_history/app.vulcanized.p2.html /wrkdirs/usr/ports/www/chromium/work/chromium-73.0.3683.103/out/Release/gen/chrome/browser/resources/md_history/lazy_load.vulcanized.p2.html failed: Fatal error 'thread 0x4003d85900 exits with resources held!' at line 320 in file /usr/local/poudriere/jails/aarch64/usr/src/lib/libthr/thread/thr_exit.c (errno = 60)

ninja: build stopped: subcommand failed.
*** Error code 1

Stop.
make: stopped in /usr/ports/www/chromium
=>> Cleaning up wrkdir
===>  Cleaning for chromium-73.0.3683.103_2
build of www/chromium | chromium-73.0.3683.103_2 ended at Thu May  9 11:42:57 EDT 2019
build time: 00:33:01
!!! build failure encountered !!!
```


----------



## acheron (May 9, 2019)

```
Fatal error 'thread 0x4003d85900 exits with resources held!'
```
It's a qemu bug, unfortunately I have no fix for that...


----------



## CrunchBerry (May 9, 2019)

acheron said:


> antoine has committed a workaround (for -current), you can backport the following changes:
> 
> 
> 
> ...



I'll keep that one in mind. For now I'm going to hold off on updating my src tree until I confirm all my package building will install on my rpi 3. I hit several other bugs. For instance kept hitting Error 127 on random things, I even had some x11 fonts fail, haven't had font issues in years.... Its kinda like compiling with bad hardware/memory  I noticed after my first pass qemu-user-static had been bumped so I rebuilt everything in my bhyve for good measure and tried again. The failed ports, like the fonts and python36 finished, but other random errors and hangs happened. I'm thinking the only logical thing to do would be to save work sources between builds and enable core dumping. I'm assuming they'd end up in the work directories anyway. Least then I can maybe try to help debug qemu. Not sure my skills are good enough, but it is certainly time to get comfortable with debuggers, even if my C skills are too lacking too be able to post patches. At least that may help discern what is a ports error, versus a qemu issue.

if this one works next go around I think I can get xorg built, that will give me something to play with. I am definitely finding the longer into the build process something runs, the higher likelyhood of hitting Error 127 which I believe is a missing command from my quick searches that or just a general runaway process being killed issue. Either way random and inconsistent when they appear. Can get around 200 packages built at a clip before enough time has passed to start getting interesting. Obviously without the cores, I can't see whats going on, but I've only recently started coding seriously due to work needs, and never knew what to do with them in the past.


```
=======================<phase: check-sanity   >============================
===>  License BSD3CLAUSE PSFL accepted by the user
===========================================================================
=======================<phase: pkg-depends    >============================
===>   py36-idna-2.8 depends on file: /usr/local/sbin/pkg - not found
===>   Installing existing package /packages/All/pkg-1.10.5_5.txz
*** Error code 127

Stop.
make: stopped in /usr/ports/dns/py-idna
=>> Cleaning up wrkdir
===>  Cleaning for py36-idna-2.8
build of dns/py-idna | py36-idna-2.8 ended at Thu May  9 00:24:43 EDT 2019
build time: 00:00:03
!!! build failure encountered !!!
```


----------



## CrunchBerry (May 9, 2019)

Brian546 said:


> Thank you Acheron,
> 
> I edited those 3 files like you said but now have a new compile error. I think we must be getting closer though:
> 
> ...



You may want to just give it another go. I've found the random errors come and go, so it is possible it will compile clean on another try. I've definitely been finding that. So far I haven't seen an error that appears to be attributed to a system library error though.


----------



## Brian546 (May 10, 2019)

Yeah I'm giving it another go. Updated the poudriere ports tree and running a bulk build again. I'll be away on vacation so won't get to see the results until the end of next week.


----------

