# Poudriere/ARM/FreeBSD 13.1 broken?



## stephen.wall (May 23, 2022)

I'm having a problem getting poudriere to cross compile ports when using a 13.1 jail.  13.0 works fine.  The error was "No working C compiler found.  Tried /nvb-bin/usr/bin/cc and gcc."
I create the jails with the following:


```
poudriere jails -c -j 130aarch64 -v 13.0-RELEASE -m http -x -a arm64.aarch64
poudriere jails -c -j 131aarch64 -v 13.1-RELEASE -m http -x -a arm64.aarch64
```


```
poudriere bulk -j 131aarch64 -p trunkaarch64 ports-mgmt/pkg
```
 works fine, 
	
	



```
poudriere bulk -j 131aarch64 -p trunkaarch64 ports-mgmt/pkg
```
 fails.  (trunkaarch64 is a 2022Q2 ports tree.)  I entered the 13.1 jail with `testport -i` (actually, I entered both in separate windows to compare them), used /nxb-bin/usr/bin/cc to compile a simple 1-line C program and got this:


```
ld: error: /tmp/c-5dc3bb.o is incompatible with /usr/lib/crt1.o
cc: error: linker command failed with exit code 1 (use -v to see invocation)
```

So I tried 
	
	



```
/nxb-bin/usr/bin/cc -c c.c
```
, and found that /nxbin/usr/bin/cc is producing x86_64 object files by default.  This seems less than optimal in an ARM jail.  Can anyone help me get past this, or at least tell me where to report a bug?

Much thanks.

- Steve


----------



## acheron (May 24, 2022)

I think it's a known issue and it's fixed on current but I can't find the relevant commit


----------



## stephen.wall (May 25, 2022)

Is it an issue with pouriere, clang, or FreeBSD 13.1?  I'm guessing poudriere...


----------



## acheron (May 25, 2022)

I think it's https://cgit.freebsd.org/src/commit/Makefile.inc1?id=e2e2c622c35f3e54dbae88e3de3e48c51dbb4ebd


----------



## astyle (May 25, 2022)

Off the top of my head, sometimes, compilers need a compile-time flag passed to them, like `--arch=arm64` or something.  But even then, double-check the relevant manpages for specific syntax.

And - Poudriere has LOTS of different components that need to be lined up. Yeah, there are some default values available if OP doesn't want to fuss with the setup, but if OP wants to change something along the way - that needs to be done very carefully, to avoid disrupting the whole toolchain. Even renaming the jail or the ZFS dataset or the ports tree in use - that has consequences down the road, when the next link in the toolchain is not getting the data it expects.


----------



## stephen.wall (Jun 2, 2022)

acheron, that patch got us going, thank you much.


----------



## kazuo (Aug 4, 2022)

Where do you apply this patch? Is this for the arm64 jail?


----------



## stephen.wall (Aug 8, 2022)

The commit Acheron mentioned is in the FreeBSD base code. /usr/src typically.


----------



## jandras (Nov 9, 2022)

I can't compile ports for Raspberry Pi. Could someone help me, please?

This host runs "FreeBSD 13.1-RELEASE-p3 GENERIC amd64".

qemu-user-static-3.1.0_13 is installed. /etc/rc.conf contains `qemu_user_static_enable="YES"`.

I patched /usr/src/Makefile.inc1 with e2e2c622c35f3e54dbae88e3de3e48c51dbb4ebd, then created a poudriere jail with `poudriere jail -cbxj 131release-arm64 -a arm64.aarch64 -m src=/usr/src` .

Executing `poudriere bulk -j 131release-arm64 ports-mgmt/pkg` produces this output:

```
[00:00:00] Cross-building ports for arm64.aarch64 on amd64 requires QEMU
[00:00:00] Creating the reference jail... done
[00:00:01] Mounting system devices for 131release-arm64-default
[00:00:01] Mounting ports/packages/distfiles
[00:00:01] Converting package repository to new format
[00:00:01] Stashing existing package repository
[00:00:01] Mounting packages from: /usr/local/poudriere/data/packages/131release-arm64-default
[00:00:01] Copying /var/db/ports from: /usr/local/etc/poudriere.d/options
[00:00:01] Setting up native-xtools environment in jail... done
[00:00:02] Raising MAX_EXECUTION_TIME and NOHANG_TIME for QEMU from QEMU_ values
[00:00:02] Copying latest version of the emulator from: /usr/local/bin/qemu-aarch64-static
/etc/resolv.conf -> /usr/local/poudriere/data/.m/131release-arm64-default/ref/etc/resolv.conf
[00:00:02] Starting jail 131release-arm64-default
[00:00:02] Will build as nobody: (65534:65534)
[00:00:03] Logs: /usr/local/poudriere/data/logs/bulk/131release-arm64-default/2022-11-09_15h34m37s
[00:00:03] WWW: http://....../poudriere/build.html?mastername=131release-arm64-default&build=2022-11-09_15h34m37s
[00:00:03] Loading MOVED for /usr/local/poudriere/data/.m/131release-arm64-default/ref/usr/ports
[00:00:06] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:06] Gathering ports metadata
[00:00:06] Calculating ports order and dependencies
[00:00:06] Sanity checking the repository
[00:00:06] Checking packages for incremental rebuild needs
[00:00:06] Deleting stale symlinks... done
[00:00:06] Deleting empty directories... done
[00:00:34] Cleaning the build queue
[00:00:34] Sanity checking build queue
[00:00:34] Processing PRIORITY_BOOST
[00:00:34] Balancing pool
[00:00:34] Recording filesystem state for prepkg... done
[00:00:36] Building 1 packages using 1 builders
[00:00:36] Starting/Cloning builders
[00:00:38] Hit CTRL+t at any time to see build progress and stats
[00:00:38] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.18.4
[00:00:39] [01] [00:00:01] Finished ports-mgmt/pkg | pkg-1.18.4: Failed: configure
[00:00:40] Stopping 1 builders
131release-arm64-default-job-01: removed
131release-arm64-default-job-01-n: removed
[00:00:41] No package built, but repository needs to be created
[00:00:41] Creating pkg repository
pkg-static: No package files have been found
Cannot create repository catalogue
[00:00:41] Cleaning up
131release-arm64-default: removed
131release-arm64-default-n: removed
[00:00:41] Unmounting file systems
```


/usr/local/poudriere/data/logs/bulk/131release-arm64-default/2022-11-09_15h34m37s/logs/pkg-1.18.4.log contains

```
===>  Configuring for pkg-1.18.4
No installed jimsh or tclsh, building local bootstrap jimsh0
No working C compiler found. Tried /nxb-bin/usr/bin/cc and gcc.
===>  Script "configure" failed unexpectedly.
```
`# /usr/local/poudriere/jails/131release-arm64/nxb-bin/usr/bin/cc -v` gives

```
FreeBSD clang version 13.0.0 ([email]git@github.com[/email]:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
Target: x86_64-unknown-freebsd13.1
Thread model: posix
InstalledDir: /usr/local/poudriere/jails/131release-arm64/nxb-bin/usr/bin
```
What should I do so that I could build ports on amd64 for Raspberry Pi?


----------



## stephen.wall (Nov 9, 2022)

I did `make buildkernel installkernel`, rebooted, and `make buildworld installworld` after applying the patch, before I created the jail.


----------



## SirDice (Nov 9, 2022)

stephen.wall said:


> I did `make buildkernel installkernel`, rebooted, and `make buildworld installworld` after applying the patch, before I created the jail.


Not a good order to do this. Do `make buildworld buildkernel` first, then `make installkernel` and `make installworld`. Also don't forget about etcupdate(8).


----------



## jandras (Nov 12, 2022)

stephen.wall said:


> I did `make buildkernel installkernel`, rebooted, and `make buildworld installworld` after applying the patch, before I created the jail.


Thanks, Stephen!
As the jail's world was built from the patched source tree during jail creation, I didn't expect any improvement from rebuilding the host's world. (But I tried it anyway.)
It turned out that I somehow messed up Makefile.inc1. After correcting it everything works as expected, and now I can build ports for Raspberry Pi.


----------

