# Upgrade from source, using r/o NFS shares and cross-compilation



## zirias@ (Mar 21, 2022)

A simple way to upgrade from source using builds done on a different machine is to just share /usr/src and /usr/obj with NFS and only execute the `installkernel` and `installworld` make targets on the target machine. This is really straight-forward, as long as both machines use the same (pre-upgrade) ABI, including the same target architecture.

Otherwise, you'll run into problems because some tools are built targeting the _build machine_; these are used during further building _and_ during installation. So for example, if you build on `amd64` targeting `i386` (super easy with `TARGET=i386`), the result won't be installable with the "NFS trick" mentioned above on an actual i386 machine because e.g. `make installworld` will try to execute tools there that were built for amd64.

My current workaround is building for i386 twice. Once as a cross-build, using the result to populate a jail for poudriere as well as another "i386 builder" jail, which I use to build for the second time (to a different /usr/obj without cross-building). From this second build, installation on an actual i386 machine works fine.

Well, my question is: is there a (hopefully simple) way to just tell the build-system to also cross-compile these temporary tools, so I don't need this workaround building everything twice?


----------



## mer (Mar 21, 2022)

Isn't /usr/obj the default output location for building?  If so, then I would expect it to collide if you are building to the same /usr/obj on different architectures.
The NFS trick is nice when everything is the same architecture, you simply build on one machine then use the NFS mounts to install from.
If the target machine does not have the physical space for a local /usr/obj, I would look for a knob that allows you to specify the output directory:
/usr/obj/amd64 or /usr/obj/i386 as the top level instead of /usr/obj


----------



## zirias@ (Mar 21, 2022)

mer said:


> Isn't /usr/obj the default output location for building? If so, then I would expect it to collide if you are building to the same /usr/obj on different architectures.


Uhm, the actual world and kernel are built to architecture-specific subdirs, so this isn't a problem. BUT these temporary build tools (like e.g. `make` itself) exist only once. That's why I wrote I use a completely separate /usr/obj in my i386 jail doing the second ("native") build.

What I have works fine, but always building twice is a waste of both CPU time and disk space, so I wonder whether there's some way to cross-build the temporary tools as well


----------



## mer (Mar 21, 2022)

Zirias said:


> Uhm, the actual world and kernel are built to architecture-specific subdirs,


Ahh, it's been a long time since I've built from source and couldn't remember.
As to the temporary tools, do they really only exist once?  I would have thought they exist until the source gets updated causing a build to regenerate them or do a "make clean".


----------



## zirias@ (Mar 21, 2022)

mer said:


> As to the temporary tools, do they really only exist once?


Uhm, I was wrong on that, they are indeed inside the architecture-specific subdir. BUT, they always target the _build_ system, not the _target_ system. See for example:

```
# file /usr/obj/usr/src/i386.i386/tmp/usr/bin/objcopy
/usr/obj/usr/src/i386.i386/tmp/usr/bin/objcopy: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 13.0 (1300139), FreeBSD-style, not stripped
```

So, I might get away with only one /usr/obj tree, but I'd still have to build twice. Or is there a target that _only_ builds these temp tools?


----------



## covacat (Mar 21, 2022)

what if you make install from the building machine to a exported location and then on the target machine just rsync


----------



## mer (Mar 21, 2022)

Zirias That is interesting.  On one hand it makes sense;  using a cross compiler on the host simply uses the tools to create output in the desired target, but some tools are indeed running under the host arch.
I don't know the answer to your question.


----------



## zirias@ (Mar 22, 2022)

covacat said:


> what if you make install from the building machine to a exported location and then on the target machine just rsync


I tried installing _onto_ an NFS share, this doesn't work because chflags(1) isn't (fully) supported. Sure, using rsync(1) might solve this, but I'd worry it might just update files it shouldn't (like everything below /etc which should be handled by etcupdate(8) instead).


----------



## SirDice (Mar 22, 2022)

Zirias said:


> Or is there a target that _only_ builds these temp tools?


`make toolchain` perhaps? I'm not sure what this target builds. I suspect you only have to build this for your target architecture. 


```
toolchain            Create the build toolchain needed to build the rest
                          of the system.  For cross-architecture builds, this
                          step creates a cross-toolchain.
```


----------



## zirias@ (Mar 22, 2022)

SirDice said:


> `make toolchain` perhaps? I'm not sure what this target builds. I suspect you only have to build this for your target architecture.


Definitely worth a test, will do so when I build an upgrade the next time! Although it _might_ just build compiler/linker and not the other temp tools, but I'll see


----------



## Alain De Vos (Mar 22, 2022)

You must differentiate between cross-compile and install.
A. The kernel.
B. The userland/world
C. The kernel-modules.
D. The ports


----------



## zirias@ (Mar 22, 2022)

Uhm, what? This is about the base system, and the source tree supports cross-compiling (which, of course, needs some tools targeting the build machine). "Base system" includes kernel, modules and userland.

Ports are out of scope here, they're build "natively" (no issue with i386 on an amd64 machine) in a jail. For "real" foreign archs, qemu can help, and there's an option to build and install "native xtools".


----------

