# Cross-Compiling to ARM64 & .IMG



## BSDreams (Jul 15, 2020)

I am having trouble creating a .img file for a. RPI3 on BSD version 13, source code I acquired from Github.

I have successfully compiled AMD64 while creating a .img for AMD64-based architectures, but seem to be having trouble with creating a .img for another system architecture type.


```
make TARGET_ARCH=aarch64 buildworld
make TARGET_ARCH=aarch64 builkernel
```

Both compiled successfully, now it's time to create an arm64 based .img and I am looking for recommendations on what the best approach might be as I have tried using the "make-memstick.sh" in arm64 directory, but failed.

Thank you for any advice.


----------



## SirDice (Jul 15, 2020)

BSDreams said:


> I am having trouble creating a .img file for a. RPI3


You may want  to  have a look  at crochet.


----------



## diizzy (Jul 15, 2020)

Please don't use crochet as it's considered deprecated
https://wiki.freebsd.org/arm/Build_image_using_release_building_infrastructure is probably what you want


----------



## SirDice (Jul 15, 2020)

diizzy said:


> Please don't use crochet as it's considered deprecated


Noted, I  didn't know that. The code could still provide some insights into the entire process though.


----------



## BSDreams (Jul 15, 2020)

When Running Crochet: I can't seem to get the pkg I need.

```
pkg: No packages available to install matching 'u-boot-rpi3' have been found in the repositories.
```


I used Github and compiled the latest code (version 13). https://wiki.freebsd.org/arm/Build_image_using_release_building_infrastructure looks like a great source, but I am not sure how i should edit the file based on the fact I am running on a AMD64, while trying to build a .img for code I already compiled for arm64.

Without doing additional research at the moment, I think I can use the configuration example provided in that link. Could I please get a better example for RPI3.conf?

I saw u-boot-rpi3 on Github, but wasn't sure where I should clone it, maybe in  "sysutils/"

```
EMBEDDEDPORTS="sysutils/u-boot-bananapi" --> EMBEDDEDPORTS="sysutils/u-boot-orangepi-pc"
```




Another Attempt, using another method using make-memstick.sh in the arm64 directory (/usr/src/release/arm64)

*sh make-memstick.sh /usr/src /usr/BSDpi*


```
set -e

PATH=/bin:/usr/bin:/sbin:/usr/sbin
export PATH

if [ $# -ne 2 ]; then
        echo "make-memstick.sh /path/to/directory /path/to/image/file"
        exit 1
fi

if [ ! -d ${1} ]; then
        echo "${1} must be a directory"
        exit 1
fi

if [ -e ${2} ]; then
        echo "won't overwrite ${2}"
        exit 1
fi

echo '/dev/ufs/FreeBSD_Install / ufs ro,noatime 1 1' > ${1}/etc/fstab
echo 'root_rw_mount="NO"' > ${1}/etc/rc.conf.local
makefs -B little -o label=FreeBSD_Install -o version=2 ${2}.part ${1}
rm ${1}/etc/fstab
rm ${1}/etc/rc.conf.local

mkimg -s gpt \
    -p efi:=${1}/boot/boot1.efifat \
    -p freebsd:=${2}.part \
    -o ${2}
rm ${2}.part
```

Failed:

```
:/usr/src/release/arm64 # sh make-memstick.sh /usr/src /usr/BSDpi
Calculated size of `/usr/BSDpi.part': 3465150464 bytes, 85496 inodes
Extent size set to 32768
/usr/BSDpi.part: 3304.6MB (6767872 sectors) block size 32768, fragment size 4096
        using 5 cylinder groups of 823.50MB, 26352 blks, 23424 inodes.
super-block backups (for fsck -b #) at:
     192, 1686720, 3373248, 5059776, 6746304,
Populating `/usr/BSDpi.part'
Image `/usr/BSDpi.part' complete
mkimg: partition 1: No such file or directory
```


----------



## diizzy (Jul 15, 2020)

You can't "inject" files using the release.sh script, it'll just create a vanilla image just like when you download a snapshot off FreeBSD's site and/or mirrors.
Rasberry Pi 3 already has a config so you don't need to create one. https://svnweb.freebsd.org/base/head/release/arm64/RPI3.conf
Just edit it if you want to change anything before compiling...

`sh release.sh -c arm64/RPI3.conf`


----------



## BSDreams (Jul 15, 2020)

I have tried that already, and this is what I get:

`/usr/src/release # sh release.sh -c arm64/RPI3.conf`


```
Illegal instruction (core dumped)
umount: /scratch/dev: statfs: No such file or directory
umount: /scratch/dev: unknown file system
```

I did not modify anything in RPI3.conf, mayby I need to?


----------



## diizzy (Jul 15, 2020)

Uhm... Illegal instruction usually occurs if you have a binary that uses instructions not supported by your CPU.
One slightly annoying issue with the release script is that it doesn't exit cleanly if it fails so you need to remove /scratch manually and possibly active mountpoints. I usually just reboot because of the latter otherwise it'll fail after a while..


----------



## BSDreams (Jul 16, 2020)

I was hoping you weren't going to say that, but that's what I thought too after a little research.  That script makes more sense to me now after reviewing the material you shared. I'll report back the VPS results later.


***UPDATE***
`/usr/src/release`
`make memstick`

I now have a AMD64 *memstick.img* that I made on *MY *server after recomiling everything for AMD64. I am still waiting for the VPS to finish cross-compiling to aarch64 and then I will run `/usr/src/release # sh release.sh -c arm64/RPI3.conf`



***VPS RESULTS**
 make TARGET_ARCH=aarch64 buildkernel*


```
--- modules-all ---
--- allwinner/sun50i-a64-nanopi-a64.dtb ---
Generating allwinner/sun50i-a64-nanopi-a64.dtb from /usr/src/sys/gnu/dts/arm64/allwinner/sun50i-a64-nanopi-a64.dts
sh: /usr/src/sys/tools/fdt/make_dtb.sh: Permission denied
*** [allwinner/sun50i-a64-nanopi-a64.dtb] Error code 126

make[4]: stopped in /usr/src/sys/modules/dtb/allwinner
1 error

make[4]: stopped in /usr/src/sys/modules/dtb/allwinner
*** [all_subdir_dtb/allwinner] Error code 2

make[3]: stopped in /usr/src/sys/modules
--- vfs_cluster.o ---
ctfconvert -L VERSION -g vfs_cluster.o
--- vfs_cache.o ---
ctfconvert -L VERSION -g vfs_cache.o
--- vfs_bio.o ---
ctfconvert -L VERSION -g vfs_bio.o
--- modules-all ---
--- all_subdir_dc ---
ctfconvert -L VERSION -g if_dc.o
A failure has been detected in another branch of the parallel make

make[4]: stopped in /usr/src/sys/modules/dc
*** [all_subdir_dc] Error code 2

make[3]: stopped in /usr/src/sys/modules
--- all_subdir_cxgbe ---
ctfconvert -L VERSION -g t4_hw.o
A failure has been detected in another branch of the parallel make

make[5]: stopped in /usr/src/sys/modules/cxgbe/if_cxgbe
*** [all_subdir_cxgbe/if_cxgbe] Error code 2

make[4]: stopped in /usr/src/sys/modules/cxgbe
1 error

make[4]: stopped in /usr/src/sys/modules/cxgbe
*** [all_subdir_cxgbe] Error code 2

make[3]: stopped in /usr/src/sys/modules
3 errors

make[3]: stopped in /usr/src/sys/modules
*** [modules-all] Error code 2

make[2]: stopped in /usr/obj/usr/src/arm64.aarch64/sys/GENERIC
1 error

make[2]: stopped in /usr/obj/usr/src/arm64.aarch64/sys/GENERIC
*** [buildkernel] Error code 2

make[1]: stopped in /usr/src
1 error

make[1]: stopped in /usr/src
*** [buildkernel] Error code 2

make: stopped in /usr/src
1 error

make: stopped in /usr/src
```

I was hoping for better results, but after buildworld I am failing at building the kernel at the moment, even though the same source compiled fine on my local server.


----------



## acheron (Jul 16, 2020)

BSDreams said:


> sh: /usr/src/sys/tools/fdt/make_dtb.sh: Permission denied


Do you have special mount option for /usr/src?


----------



## BSDreams (Jul 16, 2020)

Not that I am aware of, but the VPS OS image must have something preset, that my local server does not. I have all source files dumped into */usr/src *and buildworld executes/compiles fine, it's just the kernel compilation process that gets this permissions denied error.

I literally tried chmod 777, and touch in all directories, and this is being executed as root user, and this still gets denied. This compilation of the kernel would succeed if it were not for this permission error.

Any suggestions?


----------



## acheron (Jul 16, 2020)

`mount | grep /usr/src` ?


----------



## BSDreams (Jul 17, 2020)

I started up a new VPS running Freebsd 12.1 x64 ufs and I ran the following command and I return no results.



acheron said:


> mount | grep /usr/src ?



**UPDATE**
Upon deploying this OS image on the VPS the source files for compiling were already in /usr/src, and I avoided GitHub's code, which shouldn't matter. It's also a different system image than before as I am using another filesystem type, so my previous deployment had different preset settings/configurations. That's on the provider of the VPS, not me.

Anyways....I compiled the kernel and OS for arm64, and I am now running the shell script in release.
`sh release.sh -c arm64/RPI3.conf`

Waiting for it to finish at the moment, but so far all appears to be well.


----------



## BSDreams (Jul 17, 2020)

I received this error:

```
===> usr.sbin/wpa/ndis_events (installconfig)
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
[Creating objdir /usr/obj/usr/src/amd64.amd64/release...]
===>  Deinstalling for u-boot-rpi3
/bin/sh: /usr/local/sbin/pkg-static: not found
===>   u-boot-rpi3 not installed, skipping
===>  License GPLv2 accepted by the user
===>   u-boot-rpi3-2020.07 depends on file: /usr/local/sbin/pkg - not found
===>  License BSD2CLAUSE accepted by the user
=> freebsd-pkg-1.14.6_GH0.tar.gz doesn't seem to exist in /tmp/distfiles/.
=> Attempting to fetch https://codeload.github.com/freebsd/pkg/tar.gz/1.14.6?dummy=/freebsd-pkg-1.14.6_GH0.tar.gz
freebsd-pkg-1.14.6_GH0.tar.gz                 Bad system call (core dumped)
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/freebsd-pkg-1.14.6_GH0.tar.gz
freebsd-pkg-1.14.6_GH0.tar.gz                 Bad system call (core dumped)
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/ports-mgmt/pkg
*** Error code 1

Stop.
make: stopped in /usr/ports/sysutils/u-boot-rpi3
```
*I tried installing it, but it is not fetching it. I installed u-boot-rpi3 though and re-tried, and still nothing of course until I get that other package.*


----------



## JohnnySorocil (Jul 18, 2020)

diizzy said:


> Please don't use crochet as it's considered deprecated
> https://wiki.freebsd.org/arm/Build_image_using_release_building_infrastructure is probably what you want


Is it possible to use that (offline) with /usr/src and /usr/ports which are git repositories?


----------



## diizzy (Jul 18, 2020)

JohnnySorocil said:


> Is it possible to use that (offline) with /usr/src and /usr/ports which are git repositories?


Only SVN for now however FreeBSD is start to transition to GIT


----------



## diizzy (Jul 18, 2020)

BSDreams said:


> I received this error:
> 
> ```
> ===> usr.sbin/wpa/ndis_events (installconfig)
> ...


Are you possibly trying to build -HEAD on a box that runs 12.1/-STABLE?





						Re: /usr/src/release/release.sh -> ports -> fetch pkg -> Bad system call (core dumped)
					






					www.mail-archive.com


----------



## BSDreams (Jul 24, 2020)

I found this in release.sh

```
# The default svn checkout server, and svn branches for src/, doc/,
        # and ports/.
        SVNROOT="svn://svn.FreeBSD.org/"
        SRCBRANCH="src/head@rHEAD"
        DOCBRANCH="doc/head@rHEAD"
        PORTBRANCH="ports/head@rHEAD"
```

I also made a copy of release.conf.sample, now since I am running this `sh release.sh -c arm64/RPI3.conf`, wouldn't I just edit the release.sh file, instead of release.conf or both?

release.conf

```
## Set the directory within which the release will be built.
CHROOTDIR="/scratch"

## Set the svn host.
SVNROOT="svn://svn.FreeBSD.org/"

## Set the src/, ports/, and doc/ branches or tags.
SRCBRANCH="base/stable/12@rHEAD"
DOCBRANCH="doc/head@rHEAD"
PORTBRANCH="ports/head@rHEAD"

## Run svn co --force for src checkout.
#SRC_FORCE_CHECKOUT=yes
```

UPDATE:
This resource was helpful, but I am still wondering what I need to change the default sources above to? In the beginning I just went to /usr/src and completed a buildworld for arm64, that was it.


----------



## BSDreams (Jul 26, 2020)

Any help or advice on this would be appreciated.


----------



## mark_j (Jul 26, 2020)

diizzy said:


> Please don't use crochet as it's considered deprecated
> https://wiki.freebsd.org/arm/Build_image_using_release_building_infrastructure is probably what you want


Deprecated or not it works for all rpis upto 4.


----------



## BSDreams (Jul 26, 2020)

I added 
	
	



```
SRCBRANCH="base/head@r357335 "
```
 to RPI3.conf, and I am trying this again...


----------



## mark_j (Jul 26, 2020)

Jeez just use crochet and you would have been finished days ago.


----------



## BSDreams (Jul 26, 2020)

That attempt last night failed with the same error, so i am now using Crochet


```
/usr/crochet # sh crochet.sh -b RaspberryPi3
Starting at Sun Jul 26 14:09:29 UTC 2020
Board: RaspberryPi3
Source version is: unknown
Building FreeBSD version: 12.1
Image name is:
    /usr/crochet/work/FreeBSD-aarch64-12-GENERIC-RaspberryPi3.img
Building FreeBSD version: 12.1
Object files are at: /usr/crochet/work/obj/usr/src
Found suitable FreeBSD source tree in:
    /usr/src
Found U-Boot port in:
    /usr/local/share/u-boot/u-boot-rpi3
Found firmware port in:
    /usr/local/share/rpi-firmware
Building FreeBSD aarch64 world at Sun Jul 26 14:09:29 UTC 2020
    (Logging to /usr/crochet/work/_.buildworld.aarch64.log)
```

It's rebuilding everything, hopefully it works...


----------



## BSDreams (Jul 26, 2020)

```
DONE.
Completed disk image is in: /usr/crochet/work/FreeBSD-aarch64-12-GENERIC-RaspberryPi3.img

Copy to a suitable memory card using a command such as:
dd if=/usr/crochet/work/FreeBSD-aarch64-12-GENERIC-RaspberryPi3.img of=/dev/da0 bs=1m
(Replace /dev/da0 with the appropriate path for your card reader.)
```

That was easy, but I am just wondering why I could not get my other method to work?


----------



## diizzy (Jul 26, 2020)

```
SRCBRANCH="base/releng/12.1@r363030"
```
If you want the latest release of 12.1


----------



## BSDreams (Jul 27, 2020)

```
SRCBRANCH="base/releng/12.1@r363030"
```

The added the line above to "RPI3.conf", then ran the release.sh command.

The RPI3 image was created in the /scratch directory, but after I flash it to the RPI3, it boots up fine until I get to the login screen, then everything becomes unresponsive. I am trying a different revision of 12.1


----------



## BSDreams (Jul 27, 2020)

Both build complete, but I think GrowthFS is causing this problem on both images created, how can I stop GrowthFS from initiating?


----------



## diizzy (Jul 27, 2020)

I doubt that's the real issue, anyhow...
Mount the root partition of the SD card and remove the growfs from /etc/rc.conf


----------



## BSDreams (Jul 27, 2020)

I need a USB SDcard adapter for this, and it might work. It would also be easier to boot the mounted SD Card in the VM I am now running on my machine too.


----------



## kennethcf (Nov 14, 2020)

I too have been fighting a cross compile (host is amd64, target is arm64) 12.2 RELEASE PI3 build image of my own.

Source
URL: https://svn.freebsd.org/base/releng/12.2
Revision: 367493

*Method 1* - I made a command line buildworld (GENERIC) with arm64 according to the manual and lots of blogs and Installed to the ufs partition of the disk image.  I copied /etc and /boot conf files to the ufs partition of my pi3 image from the 12.2 release pi3 image. I copied the msdos partion from the 12.2 release pi3 image to my pi3 image. 
*Method* *2* - I used crochet with my own u-boot-rpi3-2020.10 and rpi-firmware-120200723 built in poudriere.  I had to replace the uboot.bin & EFI/BOOT/bootaa64.efi files of the crochet build with the files from the 12.2 release pi3 image.

Results - no network plugged in at boot up
*make Method 1* - closest I came was dmesg output up to where the keyboard driver was being configured, and the next line would have been the loop back "lo0: link state changed to up" message, but it just hung.

*crochet Method 2* - I got a working system, but the beetsie boot menu displayed scrambled characters (maybe because it was configured with no color??).


----------



## Phishfry (Nov 15, 2020)

kennethcf said:


> beatsie boot menu displayed scrambled characters


Are you using a Serial Console or HDMI ?
Beastie boot menu being scrambled is a symptom of wrong baud for your serial console.


----------



## JohnnySorocil (Nov 16, 2020)

On my armv7 machine /boot/loader.conf contains:

```
# Disable the beastie menu and color
beastie_disable="YES"
loader_color="NO"
```


----------



## kennethcf (Nov 9, 2021)

I am coming up on a full year since trying.  I am focusing on using release.sh assuming that will be supported more readily.  I modified the release.sh to checkout a specific src commit and to use pkg if the ports were not enabled.  I managed to build an image file from release.sh 13.0-Release p4, src git commit 940681634ee17, and the msdosfs files from pkg u-boot-rpi-arm64-2021.07 and rpi-firmware-1.20210303.g20210303.  However, my image would not boot unless I replaced u-boot.bin with a copy from the aarch64-RPI SD card image. All other msdosfs files matched between my release.sh build and the SD card image.

Does anyone know why the u-boot.bin file from u-boot-rpi-arm64-2021.07 does not work? Remember, all other msdosfs files from pkg on my build match the msdosfs files from the SD card image.
I next tried allowing release.sh to build u-boot and get necessary msdosfs files from ports.  My ports build image did not work, even after replacing the ports built u-boot.bin with the SD card image.  I did not do a comparison of the remaining msdosfs files.

Does anyone know if the ports u-boot-rpi-arm64 or rpi-firmware require configuring before building or why it may not be working for me?

Has anyone been successful at building a working u-boot.bin from ports or used release.sh to produce a 100% working SD image for sh release.sh -c arm64/RPI?

The runtime error at SD image boot time on my RPI3B+ SD card, is a once "Missing environment variable pxeuuid" followed by a repeated "Missing environment variable bootfile".  Booting eventually gives up and drops to the u-boot prompt.


----------



## kennethcf (Nov 12, 2021)

Update - Good news!  I was successful at building my own u-boot.bin from ports on the base system, outside of the release build directory.  I used the ports all-depends-list target to generate a list of dependencies (removing the u-boot related packages), and installed the dependencies from packages.  Returning to ports sysutils/u-boot-rpi-arm64, I issued a build and install, which put u-boot-rpi-arm64-2020.10 into /usr/local/share/u-boot/u-boot-rpi-arm64/u-boot.bin.  I then replaced the non-working u-boot.bin from my release.sh build with my ports build in the release.sh built .img file (mount the img file with mdconfig, then mount the msdos partition, you can replace the u-boot.bin file at that point).  I burned the SD card and it worked in my Pi3B+.  Now, to get the release.sh process to create a working u-boot.bin, and to get familiar enough to not have to use release.sh (rebuilds everything from scratch, one thing nice about Crochet is that it would re-use, I just need to figure out how to do the same with release).


----------



## kennethcf (Nov 13, 2021)

I noticed in the dependencies list for u-boot-rpi-arm64, perl5.32.  In the packages, perl5 cannot be referenced in this way.  It must be referenced as perl5-5.32 or as perl5.  My pkg listing also had perl5.30 and perl5.34.  If u-boot mus have perl5.32, then the dependency needs to be changed to perl5 or perl5-5.32.


----------

