# Can not Install FreeBSD 12 on PCengines APU 2D system



## Chris236 (Sep 30, 2019)

Hello,

we have substantial trouble installing more recent FreeBSD versions on PCEngines APU2D4 systems. It worked with 10.x,  with 11.x it was a hit/miss game depending on the Memdstick we used, and on 12 its a total disaster. 

The APU is a headless low power amd64 compatible sysytem.  The problem is that boot stops at the point where it needs to mount root from the stick.  Timeout. I tried with modified timing values, (both 
vfs.mountroot.timeout and kern.cam.boot_delay), but did not find any combination of those that got the thing working. 

For 11.3 we helped ourselves by creating an 11.3 installation stick that runs on a 10.3 kernel. (Once installed, BSD runs well enough on those boxes) 

But I am still looking for an angle to get 12 working natively. Any ideas, experiences etc?


----------



## Phishfry (Sep 30, 2019)

I have nothing but praise for the APU2. I use one as my daily Wireless Access Point.
The only thing I can offer is there were some issues for some with different firmwares.


			APU2, legacy firmware 4.0.22, FreeBSD 12.0 hangs in boot
		


Have you tried the pfSense trick?








						apu2-documentation/pfSense-install-guide.md at master · pcengines/apu2-documentation
					

Documentation and scripts for building and adjusting PC Engines APU2 firmware - apu2-documentation/pfSense-install-guide.md at master · pcengines/apu2-documentation




					github.com
				




I am still using FreeBSD 11.2 and a custom NanoBSD build I built for it.

```
root@APU2:~ # uname -a
FreeBSD APU2 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r336118: Wed Jul 18 05:35:43 EDT 2018     root@gigabyte:/usr/obj/nanobsd.apu2ap/usr/src/sys/GENERIC  amd64

root@APU2:~ # uptime
 9:17PM  up 24 days,  8:12, 1 user, load averages: 0.00, 0.00, 0.00
```

If I were you I would try some different firmwares. These boxes are amazing and cheap.


			PC Engines - github pages


----------



## Chris236 (Oct 4, 2019)

Phishfry said:


> I have nothing but praise for the APU2. I use one as my daily Wireless Access Point.
> The only thing I can offer is there were some issues for some with different firmwares.
> 
> 
> ...



Firmware: we have BIOS version v4.9.0.4, SeaBIOS (version rel-1.12.1.1-0-g55d345f)
Not sure why we should use older BIOSes, and we can't easily switch to a new
one, as we use a modified version that changes the serial speed to 9600
( so we can operate the boxes in datat centers, with terminal servers a few dozen
meters away.)

I tried the pfsense "trick", namely setting hint.ahci.0.msi="0" in loader.conf,
without seeing any result.

Here an excerpt from a typical the failed boot.

```
FreeBSD 12.1-BETA1 r352546 GENERIC amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.)
VT(vga): resolution 640x480
CPU: AMD GX-412TC SOC                                (998.15-MHz K8-class CPU
(.....)
uhub0: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub1: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub1: 4 ports with 4 removable, self powered
uhub0: 2 ports with 2 removable, self powered
ugen1.2: <vendor 0x0438 product 0x7900> at usbus1
uhub2 on uhub0
uhub2: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbu1
ugen0.2: <Kingston DT 101 II> at usbus0
umass0 on uhub1
umass0: <Kingston DT 101 II, class 0/0, rev 2.00/13.c6, addr 1> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0xc100
umass0:2:0: Attached to scbus2
uhub2: 4 ports with 4 removable, self powered
run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command, 3 more tries remain
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command, 2 more tries remain
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command, 1 more tries remain
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command, 0 more tries remain
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Error 5, Retries exhausted
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <TS128GMSA370 P1225CH1> ACS-2 ATA SATA 3.x device
ada0: Serial Number E856960516
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 122104MB (250069680 512 byte sectors)
Trying to mount root from ufs:/dev/ufs/FreeBSD_Install [ro,noatime]...
random: unblocking device.
arc4random: no preloaded entropy cache
arc4random: no preloaded entropy cache
Mounting from ufs:/dev/ufs/FreeBSD_Install failed with error 2; retrying for 10s
Mounting from ufs:/dev/ufs/FreeBSD_Install failed with error 2.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/ufs/FreeBSD_Install
  vfs.root.mountfrom.options=ro,noatime

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:zroot/ROOT/default
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>
```

I also tried the question mark at this prompt, but do not really understand its output:
	
	



```
mountroot> ?

List of GEOM managed disk devices:
  diskid/DISK-E856960516p6 diskid/DISK-E856960516p5 diskid/DISK-E856960516p4 diskid/DISK-E856960516p3 diskid/DISK-E856960516p2 diskid/DISK-E856960516p1 ufsid/58b630c090(rest cut off by terminal)

mountroot>
```


----------



## msplsh (Oct 4, 2019)

What's the process for creating the stick?


----------



## Chris236 (Oct 4, 2019)

msplsh said:


> What's the process for creating the stick?


Which one?

The one I am testing with right now?

- get the image and write it to a stick
- mount it on an existing  FreeBSD system (`mount /dev/da0s2a /mnt` )
- edit /mnt/boot/loader.conf
[update/correction - of course we edit the loader.conf on the stick  ]

put in:

```
# console speed that your BIOS uses as well (standard is 115200, ours is modified to 9600)
# comconsole_speed="9600"
comconsole_speed="115200"
console="comconsole"
boot_multicons="YES"
boot_serial="YES"
# parameters to toy with for timing
vfs.mountroot.timeout="200"
vfs.root_mount_always_wait=1
kern.cam.boot_delay="10000"
```

sync and umount:
`sync; sync; sync; umount /mnt`

remove stick, move to APU, try booting


----------



## Chris236 (Oct 4, 2019)

BTW - I got lucky on the test box.  After 4 of 5 warm reboots from the bootblock, suddenly the boot ran through and let me do an install.

Now I have a FreeBSD 12.1 Beta and it can not recognize the USB Sticks I give it (this used to work in 11.3)


```
Oct  4 21:49:27 dnsprobe1 kernel: ugen0.2: <6989 Intenso Alu Line> at usbus0
Oct  4 21:49:27 dnsprobe1 kernel: umass0 on uhub1
Oct  4 21:49:27 dnsprobe1 kernel: umass0: <6989 Intenso Alu Line, class 0/0, rev 2.00/2.00, addr 1> on usbus0
Oct  4 21:49:27 dnsprobe1 kernel: umass0:  SCSI over Bulk-Only; quirks = 0x0100
Oct  4 21:49:27 dnsprobe1 kernel: umass0:2:0: Attached to scbus2
Oct  4 21:50:30 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
Oct  4 21:50:30 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Oct  4 21:50:30 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): Retrying command, 3 more tries remain
Oct  4 21:50:31 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
Oct  4 21:50:31 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Oct  4 21:50:31 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): Retrying command, 2 more tries remain
Oct  4 21:50:32 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
Oct  4 21:50:32 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Oct  4 21:50:32 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): Retrying command, 1 more tries remain
Oct  4 21:50:33 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
Oct  4 21:50:33 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Oct  4 21:50:33 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): Retrying command, 0 more tries remain
Oct  4 21:50:34 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
Oct  4 21:50:34 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Oct  4 21:50:34 dnsprobe1 kernel: (probe0:umass-sim0:0:0:0): Error 5, Retries exhausted
```

The same effect with an older Kingston stick. Apparently 12.x has a serious problem with the USB on the APU...


----------



## D-FENS (Oct 4, 2019)

Chris236 said:


> Firmware: we have BIOS version v4.9.0.4, SeaBIOS (version rel-1.12.1.1-0-g55d345f)
> Not sure why we should use older BIOSes, and we can't easily switch to a new
> one, as we use a modified version that changes the serial speed to 9600
> ( so we can operate the boxes in datat centers, with terminal servers a few dozen
> ...


I have had a very similar error like yours, and it seems to me that it happened after an upgrade (11.2 or 12.0 I don't remember).
I think I pinned it down to some probable bug in the SATA III code and the workaround that helped me was to downgrade the disk to SATA II, which is still more than enough for a HDD.
Take a look at the thread here: https://forums.freebsd.org/threads/kernel-ahcich-timeout-in-slot.51868/page-3#post-423317

Edit: It's probably not the same. I see yours is related to the mass storage device.


----------



## Phishfry (Oct 4, 2019)

Somewhere in my APU reading I saw there is a bug with USB3 memsticks.
Have you tried with a USB2 stick?



> 2) - all apu boards have problems with USB 3.x stick detection in BIOS, system is working properly, but after reboot/warmboot/coldboot USB stick may not appear in BIOS boot menu (depends on the stick, well working sticks are mentioned in this document)





			PC Engines - github pages


----------



## msplsh (Oct 4, 2019)

You left out this part of creating the stick.  I'm just assuming that's what you did?
`# sudo dd if=FreeBSD-12.0-RELEASE-amd64-memstick.img of=/dev/disk2 bs=1m conv=sync`

I tried this on my APU1D4 after changing boot.conf as you described, and it worked.  Unfortunately, this is not the unit you have, so there must be some other esoteric problem.  Mine is running SeaBIOS (version ?-20140405_120742-frink), which is some kind of modified coreboot from 2014

`# uname -a
FreeBSD  12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC  amd64`

I used some piece of junk USB disk
`ugen1.2: <USB Flash Disk USB Product String123456> at usbus1
umass1 on uhub4
umass1: <USB Flash Disk USB Product String123456, class 0/0, rev 2.00/1.00, addr 2> on usbus1
umass1:  SCSI over Bulk-Only; quirks = 0xc100
umass1:7:1: Attached to scbus7`


----------



## Chris236 (Oct 4, 2019)

Phishfry said:


> Somewhere in my APU reading I saw there is a bug with USB3 memsticks.
> Have you tried with a USB2 stick?
> 
> 
> ...



BTW  - the same memsticks works on the same Hardware
- under an already installed 11.2-RELEASE
- as install stick with a 10.3 kernel

So this is quite likely not a hardware bug - it must be a FreeBSD issue

The Kingston (Data Traveler 101 G2) is pretty ancient and definitely USB2.
The Intenso is a current but cheap product,  and its data from usbconfig seems
mostly bogus (VendorID 0xFFFF - yeah!). But it claims Speed 480Mbps, and
thus I assume it's USB2 as well.


----------



## Chris236 (Oct 4, 2019)

msplsh said:


> You left out this part of creating the stick.  I'm just assuming that's what you did?
> `# sudo dd if=FreeBSD-12.0-RELEASE-amd64-memstick.img of=/dev/disk2 bs=1m conv=sync`



LOL.- I kinda mentioned it when I said: 





> - get the image and write it to a stick



Though I worked as root and didn't use the conv parameter. I think it does nothing in this situation.


----------



## msplsh (Oct 5, 2019)

Probably not.  I know "write it to a stick" could mean a lot of different things to different people, which is why I asked.  Sorry I don't have an APU2 to fiddle with to help more.


----------



## gigagoochelaar (Nov 2, 2019)

Chris236 said:
			
		

> Firmware: we have BIOS version v4.9.0.4, SeaBIOS (version rel-1.12.1.1-0-g55d345f) Not sure why we should use older BIOSes.


Because the PC Engines website explicitly states:


> For FreeBSD based OS like OPNSense and pfSense please use the legacy versions.


v4.9.0.4 is a mainline release. My APU2C4 had v4.0.17 (legacy) and I ran into the same mounting problem, after setting `hint.ahci.0.msi="0"` in /boot/device.hints the FreeBSD 12 installer launched succesfully. Once I updated the firmware to v4.0.29 everything installed as expected, even without disabling MSI.

EDIT: The PC Engines website got updated in the meanwhile.


----------

