# RockPro64 with PCIe slot



## Phishfry (Sep 27, 2022)

I just bought a RockPro64 with 4GB/64GB removable eMMC module and a nice case.
Has anybody used an NVMe on these yet?
I see lots of people using SATA controller in the x4 slot.

Will stock u-boot from ports boot off an NVMe on this board? Whats the speed look like?









						ROCKPro64 | PINE64
					

Hexa-Core Rockchip RK3399 Up to 4GB LPDDR4 RAM PCIe 4x open-ended slot Optional 802.11ac WiFi with Bluetooth 5.0 Gigabit Ethernet Bootable microSD or optional eMMC module Go to Store The ROCKPro64 is…




					www.pine64.org
				








						ROCKPro64
					






					wiki.pine64.org
				





			arm/RockChip - FreeBSD Wiki


----------



## diizzy (Sep 28, 2022)

I know some who does, speeds are as you expect (given the SoC) and I think you can boot off it you but need to change the uboot script
MicroSD and iSCSI works well too


----------



## JohnnySorocil (Sep 28, 2022)

I am using NVMe SSD on (non RockPro64) RK3399 SBC:

```
# uname -srm
FreeBSD 14.0-CURRENT arm64
# pciconf -lv
pcib1@pci0:0:0:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1d87 device=0x0100 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Rockchip Electronics Co., Ltd'
    device     = 'RK3399 PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
nvme0@pci0:1:0:0:       class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa804 subvendor=0x144d subdevice=0xa801
    vendor     = 'Samsung Electronics Co Ltd'
    device     = 'NVMe SSD Controller SM961/PM961/SM963'
    class      = mass storage
    subclass   = NVM
```

It is not possible to boot from NVMe directly, u-boot needs to be on SPI flash/eMMC/SD card (this is limitation of RK3399).
But everything else is on NVMe EFI partition (boot script, /boot/loader.efi, and devicetree included).
I am not using u-boot from ports (using vanilla u-boot with custom patches needed for something non-storage related) but I believe that version from ports will work.

"Speed test" NVMe (ZFS rootFS): around 800 MB/s for simple write, 1000 MB/s for simple read.

```
# dd if=/dev/zero of=/root/tmp bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 12.715344 secs (844445781 bytes/sec)

# dd if=/dev/zero of=/root/tmp bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 12.560426 secs (854861013 bytes/sec)

# dd if=/root/tmp of=/dev/zero bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 10.264598 secs (1046063214 bytes/sec)

# dd if=/root/tmp of=/dev/zero bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 10.244900 secs (1048074433 bytes/sec)
```


----------



## Phishfry (Oct 3, 2022)

JohnnySorocil said:


> I am using NVMe SSD on (non RockPro64) RK3399 SBC:


Can you please share your platform? I also have RockPi 3A  but unsuccessful so far.

I cannot get an NVMe to show up on RockPro64. nvme support is built in my FreeBSD u-boot.
Nothing will show. I tried PCIe to M.2 adapter for NVMe using older Toshiba XG3.
I also used a PCIe x4 SFF-8673 to U.2 NVMe PM983 Samsung with no luck.
Tried scanning for nvme in u-boot which fails to find any device.

But I have one small breakthrough. A record first PCIe Atheros Wifi card on ARM. Yipee.
Maybe finally an Arm Wireless AP (not using marginal usb wifi devices).


```
# pciconf -lv
pcib1@pci0:0:0:0:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1d87 device=0x0100 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Rockchip Electronics Co., Ltd'
    device     = 'RK3399 PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
none0@pci0:1:0:0:    class=0x028000 rev=0x01 hdr=0x00 vendor=0x168c device=0x0030 subvendor=0x144d subdevice=0x4107
    vendor     = 'Qualcomm Atheros'
    device     = 'AR93xx Wireless Network Adapter'
    class      = network
```


----------



## JohnnySorocil (Oct 4, 2022)

Phishfry said:


> Can you please share your platform? I also have RockPi 3A  but unsuccessful so far.
> 
> I cannot get an NVMe to show up on RockPro64. nvme support is built in my FreeBSD u-boot.
> Nothing will show. I tried PCIe to M.2 adapter for NVMe using older Toshiba XG3.
> ...



I am using FriendlyElec SoM-RK3399.
Can you try in u-boot:
`pci enum
nvme scan
nvme dev`
What dts file are you using?


----------



## Phishfry (Oct 4, 2022)

JohnnySorocil said:


> What dts file are you using?


rockpro64.dtb
Using FreeBSD 13.1-RELEASE and the u-boot looks to be from 07-2021 (built in May this year)

I had trouble as you see from pciconf. The Atheros was not showing up. SOICCREATE or some error.
Then I remembered the special sauce.
/boot/loader.conf
if_ath_load="YES"

Now it shows a node ath0 instead of none0:

```
root@rockpro64:~ # pciconf -lv
pcib1@pci0:0:0:0:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1d87 device=0x0100 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Rockchip Electronics Co., Ltd'
    device     = 'RK3399 PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
ath0@pci0:1:0:0:    class=0x028000 rev=0x01 hdr=0x00 vendor=0x168c device=0x0030 subvendor=0x144d subdevice=0x4107
    vendor     = 'Qualcomm Atheros'
    device     = 'AR93xx Wireless Network Adapter'
    class      = network
```

On AMD64/i386 this module is loaded automatically.


----------



## Phishfry (Oct 5, 2022)

I had previously tried the `nvme scan` feature from u-boot.

Frustrated I ran the command over and over 4 times and then the NVMe showed up.


```
pcib1@pci0:0:0:0:    class=0x060400 rev=0x00 hdr=0x01 vendor=0x1d87 device=0x0100 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Rockchip Electronics Co., Ltd'
    device     = 'RK3399 PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
nvme0@pci0:1:0:0:    class=0x010802 rev=0x01 hdr=0x00 vendor=0x1179 device=0x010f subvendor=0x1179 subdevice=0x0001
    vendor     = 'Toshiba Corporation'
    device     = 'NVMe Controller'
    class      = mass storage
    subclass   = NVM
```

Hooray.
Some quick numbers.

```
# diskinfo -wS /dev/nda0
/dev/nda0
    512             # sectorsize
    512110190592    # mediasize in bytes (477G)
    1000215216      # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    THNSN5512GPU7 TOSHIBA    # Disk descr.
    76QS10HKTTNV    # Disk ident.
    nvme0           # Attachment
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:   2149.7 usec/IO =      0.2 Mbytes/s
       1 kbytes:   2150.2 usec/IO =      0.5 Mbytes/s
       2 kbytes:   2149.8 usec/IO =      0.9 Mbytes/s
       4 kbytes:   2186.8 usec/IO =      1.8 Mbytes/s
       8 kbytes:   2273.1 usec/IO =      3.4 Mbytes/s
      16 kbytes:   2415.7 usec/IO =      6.5 Mbytes/s
      32 kbytes:   3929.9 usec/IO =      8.0 Mbytes/s
      64 kbytes:   2528.7 usec/IO =     24.7 Mbytes/s
     128 kbytes:   2707.9 usec/IO =     46.2 Mbytes/s
     256 kbytes:   6101.1 usec/IO =     41.0 Mbytes/s
     512 kbytes:   3532.7 usec/IO =    141.5 Mbytes/s
    1024 kbytes:   7341.3 usec/IO =    136.2 Mbytes/s
    2048 kbytes:   8814.4 usec/IO =    226.9 Mbytes/s
    4096 kbytes:  19092.7 usec/IO =    209.5 Mbytes/s
    8192 kbytes:  88914.8 usec/IO =     90.0 Mbytes/s
```


```
root@rockpro64:~ # diskinfo -t /dev/nda0
/dev/nda0
    512             # sectorsize
    512110190592    # mediasize in bytes (477G)
    1000215216      # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    THNSN5512GPU7 TOSHIBA    # Disk descr.
    76QS10HKTTNV    # Disk ident.
    nvme0           # Attachment
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Seek times:
    Full stroke:      250 iter in   0.035301 sec =    0.141 msec
    Half stroke:      250 iter in   0.035102 sec =    0.140 msec
    Quarter stroke:      500 iter in   0.069992 sec =    0.140 msec
    Short forward:      400 iter in   0.055919 sec =    0.140 msec
    Short backward:      400 iter in   0.055948 sec =    0.140 msec
    Seq outer:     2048 iter in   0.205185 sec =    0.100 msec
    Seq inner:     2048 iter in   0.204788 sec =    0.100 msec

Transfer rates:
    outside:       102400 kbytes in   0.232860 sec =   439749 kbytes/sec
    middle:        102400 kbytes in   0.223878 sec =   457392 kbytes/sec
    inside:        102400 kbytes in   0.230223 sec =   444786 kbytes/sec
```


----------



## Phishfry (Oct 5, 2022)

JohnnySorocil said:


> Can you try in u-boot:
> 
> ```
> pci enum
> ...


What I found is one M.2 to PCIe adapter I have has Gen1 training issues.
Have to `nvme scan` many times for showup.

But testing others M.2 adapters no such problems. I do have to run `pci enum` first.
So how do I automate the above? u-boot scr file?
Need to do `pci enum` then `nvme scan`. Then boot.


----------



## JohnnySorocil (Oct 5, 2022)

Phishfry said:


> So how do I automate the above? u-boot scr file?
> Need to do `pci enum` then `nvme scan`. Then boot.


You can automate it with boot.scr. But then file must be on some media accessible by u-boot which is not NVMe.
Other option is to put it in u-boot environment, in preboot var or some custom named var.

Something like:

```
=> printenv preboot
preboot=pci enum; nvme scan; nvme dev; pwm config 1 0 100000 7000; pwm enable 1 0

=> printenv freebsd
freebsd=pci enum; nvme scan; nvme dev; load nvme 0:1 $kernel_addr_r efi/freebsd.efi; load nvme 0:1 $fdt_addr_r $fdtfile; bootefi $kernel_addr_r $fdt_addr_r
=> run freebsd
```


----------



## Phishfry (Oct 5, 2022)

I bought another NVMe to test. Newer Toshiba 256GB. I wanted small so this is 2230 M.2 module.
Surprised at the descent write speeds.

```
# diskinfo -wS /dev/nda0
/dev/nda0
    512             # sectorsize
    256060514304    # mediasize in bytes (238G)
    500118192       # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    KBG40ZNS256G NVMe KIOXIA 256GB    # Disk descr.
    Y1FPH51XQXA3    # Disk ident.
    nvme0           # Attachment
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:  1485.5 usec/IO =      0.3 Mbytes/s
       1 kbytes:   1638.4 usec/IO =      0.6 Mbytes/s
       2 kbytes:   1563.8 usec/IO =      1.2 Mbytes/s
       4 kbytes:   1342.7 usec/IO =      2.9 Mbytes/s
       8 kbytes:   1445.1 usec/IO =      5.4 Mbytes/s
      16 kbytes:   1734.3 usec/IO =      9.0 Mbytes/s
      32 kbytes:   1676.4 usec/IO =     18.6 Mbytes/s
      64 kbytes:    944.8 usec/IO =     66.2 Mbytes/s
     128 kbytes:   1079.7 usec/IO =    115.8 Mbytes/s
     256 kbytes:   1307.0 usec/IO =    191.3 Mbytes/s
     512 kbytes:   1720.9 usec/IO =    290.6 Mbytes/s
    1024 kbytes:   2576.0 usec/IO =    388.2 Mbytes/s
    2048 kbytes:   4314.8 usec/IO =    463.5 Mbytes/s
    4096 kbytes:   7922.7 usec/IO =    504.9 Mbytes/s
    8192 kbytes:  14945.5 usec/IO =    535.3 Mbytes/s
```


----------



## Phishfry (Oct 5, 2022)

JohnnySorocil said:


> printenv preboot





JohnnySorocil said:


> => printenv freebsd


Thank You. I will investigate this avenue. I was already mucking with printenv. Making sure 'pcidisable' was not set.

I am wondering if there is anything on my SPI. I see it is found when booting.
Also noted that FreeBSD is using a new device node name for SPI.
/dev/flash/
I backed it up with dd in anticipation of flashing it.

I noticed that my PWM nodes are already showing.

```
# ls /dev/pwm
pwmc0.0    pwmc1.0    pwmc2.0
```
From the Pine forums it appears the Fan Connector is an easy testing spot for PWM LED's.
I look forward to testing PWM.

JohnnySorocil so you are controlling a fan with PWM?



JohnnySorocil said:


> pwm config 1 0 100000 7000; pwm enable 1 0
> [/QUOTE



I have failure on testing pin 40 of the Pi-Header. Not sure all is right with GPIO for this board.
It seems there are 5 GPIO busses and all 5 busses have the same pin pad names.
That is not what I am used to. Usually each SOC pin pad has a unique name. Spread across multiple GPIO busses.


```
# ls /dev/gpio*
/dev/gpioc0    /dev/gpioc1    /dev/gpioc2    /dev/gpioc3    /dev/gpioc4
```


----------



## Phishfry (Oct 5, 2022)

One thing I like with this board is that the buttons work correctly.
Press power and it starts. Run `shutdown -p now` and it shuts down correctly. 
Press power when running and it shuts down properly.
Its one of those things you take for granted on x86. Been rough on Arm but getting there.
HDMI output needs some help on Arm. Console is reliable.


----------



## JohnnySorocil (Oct 6, 2022)

Phishfry said:


> JohnnySorocil so you are controlling a fan with PWM?



Yes, I am controlling fan with PWM1. It was enabled in u-boot and later slowed down in /etc/rc.local. IIRC you can also control backlight with it but there is better way - `backlight`



Phishfry said:


> One thing I like with this board is that the buttons work correctly.
> Press power and it starts. Run `shutdown -p now` and it shuts down correctly.
> Press power when running and it shuts down properly.
> Its one of those things you take for granted on x86. Been rough on Arm but getting there.
> HDMI output needs some help on Arm. Console is reliable.


I needed to patch ATF (ARM Trusted Firmware, code which runs on MCU embedded in RK3399 and does power stuff) to enable shutdown (via cmd). Well, more like hack it but it is working. Look info PSCI standard if you want to know more.
Hardware power off/power on works with phy buttons. Power on button needs to be pressed for 4-5 seconds but that's probably limitation of PMIC chip.
Didn't manage to get power off button to nicely works from FreeBSD but it was possible from Linux (pressing power button results in the same action as running `shutdown`


----------



## Phishfry (Oct 16, 2022)

Thanks to JohnnySorocil I have root operating system on NVMe.

I still need to make a boot.scr but that is just automation.
I have it working manually.
Key was figuring out u-boot.
`pci enum` is needed to show up pci 1 bus. That is where nvme resides.
PCI bridge shows up as pci bus 0.

So after I enumerate PCI bus 1 then I do an `nvme scan` and all is good.

My steps.
Flash u-boot pkg onto 256MB microsd card and add GPT scheme and EFI partition with offset 32768.

Format msdos and copy all EFI partition files from RELEASE-image to microsd card

Add /efi/freebsd/loader.env along with directory.
currdev=disk1p2
rootdev=disk1p2

Then I flashed FreeBSD 13.1-RELEASE-ROCKPRO64 image to NVMe.
Yes I have two EFI partitions that way, but heck, it works..
One on microSD to kickstart it and one resides on NVMe.
Not sure it needs both but it was easy enough.
I was determined to not waste a 64GB eMMC module on u-boot and EFI.


----------



## Phishfry (Oct 17, 2022)

I have some RockPro64 GPIO pins working on FreeBSD.

Header Pin 3=   /dev/gpioc1-PIN20
Header Pin 5=   /dev/gpioc1-PIN21
Header Pin 7=   /dev/gpioc4-PIN24
Header Pin 8=   Reserved for UART2
Header Pin 10= Reserved for UART2
Header Pin 11= /dev/gpioc1-PIN22
Header Pin 12= Reserved for I2S0
Header Pin 13= /dev/gpioc1-PIN18
Header Pin 15= /dev/gpioc1-PIN1
Header Pin 16= /dev/gpioc1-PIN4
Header Pin 18= /dev/gpioc4-PIN21
Header Pin 19= Reserved for SPI
Header Pin 21= Reserved for SPI
Header Pin 22= /dev/gpioc4-PIN25
Header Pin 23= Reserved for SPI
Header Pin 24= Reserved for SPI
Header Pin 25= Reserved for SPI
Header Pin 26= /dev/gpioc1-PIN13
Header Pin 27= Reserved for I2C4
Header Pin 28= Reserved for I2C4
Header Pin 29= /dev/gpioc4-PIN27
Header Pin 31= /dev/gpioc4-PIN28
Header Pin 32= Reserved for I2S0
Header Pin 33= Reserved for I2S0
Header Pin 35= Reserved for I2S0
Header Pin 36= Reserved for I2S0
Header Pin 37= Reserved for I2S0
Header Pin 38= Reserved for I2S0
Header Pin 40= Reserved for I2S0










						gpio/rpro64.gpio.png at master · tuxd3v/gpio
					

A gpio Library for SBCs. Contribute to tuxd3v/gpio development by creating an account on GitHub.




					github.com


----------



## diizzy (Oct 17, 2022)

Phishfry
Do you know where options for buttons are set for the RockPro64?


----------



## Phishfry (Oct 17, 2022)

I do not. Even though I raved about the working buttons they no longer work while running.
Same exact FreeBSD version. Different root device. Head scratcher.

With u-boot script everything is working seamlessly. SSH Headless now. Was using UART2 serial console.
HDMI output stops anyway. Never worked to command prompt. Only early on and u-boot.

I dogged the NVMe with `portsnap auto` and `git clone /usr/src`.
No errors.

I will let you know if I find the buttons with GPIO search. I suspect they are there along with the boards LED's.

Really tricky naming scheme for the pin pads.
Even though Rock64 also calls their 40 Pin connector 'Pi-2 connector' both boards use different pin assignments.


----------



## Phishfry (Oct 17, 2022)

diizzy said:


> Do you know where options for buttons are set for the RockPro64?


I had to go to the schematic for that one.


			http://files.pine64.org/doc/rockpro64/rockpro64_v20-SCH.pdf
		

From Page 8 view U1C
PWR_KEY_ L = gpio0_A5

For gpioctl = /dev/gpioc0-PIN5


----------



## Phishfry (Oct 17, 2022)

From that same schematic:
Work LED= gpio0_B3

To toggle onboard white LED:
`gpioctl -f /dev/gpioc0 -t 11`


----------



## Phishfry (Oct 17, 2022)

diizzy said:


> where options for buttons are set


No I do not. I would be very wary of toggling that PWR_KEY pin I quoted.
It could send an immediate shutdown possibly corrupting the disk.
On the other hand it could just trigger the PMU to shutdown.

Experiments like that I would try on microSD, not my shiny new NVMe on Arm!!!

My bet is the dtb sets the options. Have you decompiled and studied? Looked at ofwdump? devinfo?
I saw i2c stuff when I peaked. Tempted to strip out SPI rom support.
That or go all in and try and u-boot off spi0 and ESP+root on nvme


----------



## Phishfry (Oct 21, 2022)

Does anybody have HDMI working on this board with FreeBSD 13.1-RELEASE? Any version?
I can get to u-boot and early parts of loader but I lose HDMI early on.
I tried loader menu switching to video only #5 but it doesn't help.


----------



## diizzy (Oct 21, 2022)

Works fine on my end


----------



## Phishfry (Oct 23, 2022)

Not mine I moved HDMI cable from HDMI Switch and went direct but same result.
Probably fails around EFI Framebuffer device loads. Single mode fails too.

Here is the riser I settled on for NVMe








						PCIe NVMe M.2 NGFF SSD to PCI-E PCI express 3.0 x4 x8 x16 adapter card new  | eBay
					

Find many great new & used options and get the best deals for PCIe NVMe M.2 NGFF SSD to PCI-E PCI express 3.0 x4 x8 x16 adapter card new at the best online prices at eBay! Free shipping for many products!



					www.ebay.com
				



Has nice dashed lines to cut off excess fingers. Plus I hacked the length to the 2242 Hole. Made it smallest possible.
Using Samsung M.2-2242 PM991 drive 512GB.

Ends up just as tall as my custom heatsink.
Unfortunately too tall for stock case. I am going to have to build a custom chassis for this one.

I did buy a flexible cable riser but it did not work for me in the case.

Hacked together a RTC battery from dupont cable ends and shrink wrapped bios battery with leads.

I bought a second RockPro64 for PCIe card testing. Wondering about a Wintv card on it...Arm media server...


----------



## Phishfry (Oct 23, 2022)

I went back to a FDTI TTL UART cable to examine system for HDMI errors.
Here is uboot until EFI boot

```
Connected

U-Boot TPL 2022.04 (Sep 04 2022 - 15:14:02)
Channel 0: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
Channel 1: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
256B stride
lpddr4_set_rate: change freq to 400000000 mhz 0, 1
lpddr4_set_rate: change freq to 800000000 mhz 1, 0
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2022.04 (Sep 04 2022 - 15:14:02 +0000)
Trying to boot from MMC2


U-Boot 2022.04 (Sep 04 2022 - 15:14:02 +0000)

SoC: Rockchip rk3399
Reset cause: POR
Model: Pine64 RockPro64 v2.1
DRAM:  3.9 GiB
PMIC:  RK808
Core:  292 devices, 29 uclasses, devicetree: separate
MMC:   mmc@fe310000: 3, mmc@fe320000: 1, mmc@fe330000: 0
Loading Environment from SPIFlash... SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

In:    serial
Out:   vidconsole
Err:   vidconsole
Model: Pine64 RockPro64 v2.1
Net:   eth0: ethernet@fe300000
starting USB...
Bus usb@fe380000: USB EHCI 1.00
Bus usb@fe3a0000: USB OHCI 1.0
Bus usb@fe3c0000: USB EHCI 1.00
Bus usb@fe3e0000: USB OHCI 1.0
Bus usb@fe800000: Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.10
Bus usb@fe900000: Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.10
scanning bus usb@fe380000 for devices... 1 USB Device(s) found
scanning bus usb@fe3a0000 for devices... 1 USB Device(s) found
scanning bus usb@fe3c0000 for devices... 1 USB Device(s) found
scanning bus usb@fe3e0000 for devices... 2 USB Device(s) found
scanning bus usb@fe800000 for devices... 1 USB Device(s) found
scanning bus usb@fe900000 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0
=> hdmi
Unknown command 'hdmi' - try 'help'
=> boot
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
Found U-Boot script /boot.scr
100 bytes read in 3 ms (32.2 KiB/s)
## Executing script at 00500000

IDE device 0: Vendor: 0x144d Rev: BL2QFXV7 Prod: S4UKNF2NA84495  
            Type: Hard Disk
            Capacity: 488386.3 MB = 476.9 GB (1000215216 x 512)
SCRIPT FAILED: continuing...
76400 bytes read in 10 ms (7.3 MiB/s)
Card did not respond to voltage select! : -110
Scanning disk mmc@fe310000.blk...
Disk mmc@fe310000.blk not ready
Scanning disk mmc@fe320000.blk...
Card did not respond to voltage select! : -110
Scanning disk mmc@fe330000.blk...
Disk mmc@fe330000.blk not ready
Scanning disk nvme#0.blk#1...
Found 5 disks
** Unable to read file ubootefi.var **
Failed to load EFI variables
BootOrder not defined
EFI boot manager: Cannot load any image
Found EFI removable media binary efi/boot/bootaa64.efi
1262604 bytes read in 62 ms (19.4 MiB/s)
Booting /efi\boot\bootaa64.efi
```

Then EFI to FreeBSD:


```
Consoles: EFI console
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p1:
FreeBSD/arm64 EFI loader, Revision 1.1

   Command line arguments: loader.efi
   Image base: 0xf0dcb000
   EFI version: 2.90
   EFI Firmware: Das U-Boot (rev 8226.1024)
   Console: efi (0x1000)
   Load Path: /efi\boot\bootaa64.efi
   Load Device: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/SD(1)/SD(1)/HD(1,GPT,1af53d74-4d7b-11ed-a4f1-d4bed91c1ed4,0x8000,0x19000)
    Setting currdev to configured rootdev disk1p2
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
```
After Loader Prompt HDMI freezes here:

```
Trying to mount root from ufs:/dev/nda0p2 [rw]...
nda0: nvme version 1.3 x4 (max x4) lanes PCIe Gen1 (max Gen3) link
nda0: 488386MB (1000215216 512 byte sectors)
Dual Console: Video Primary, Serial Secondary
uhub2: 1 port with 1 removable, self powered
ugen3.2: <vendor 0x099a USB Keyboard> at usbus3
ukbd0 on uhub3
ukbd0: <EP1 Interrupt> on usbus3
kbd1 at ukbd0
```
I doubt it is a Keyboard problem. I will have to compare.


----------



## Phishfry (Oct 23, 2022)

I see my PCIe bus is only running at GEN1 and GEN2 is possible. Anybody figure this out?




diizzy said:


> Do you know where options for buttons are set for the RockPro64?


Another fragment of Power Switch is showing in `ofwdump -a`

```
Node 0xd0b0: gpio-keys
    Node 0xd108: power
```
  Also here:

```
Node 0xc784: buttons
      Node 0xc790: pwrbtn
```


----------



## diizzy (Oct 23, 2022)

How old is your u-boot version? You need to fairly new one 2021.07 or newer if I recall correctly, there are also issues with 4k displays (I've only used 1080p and below). If you have (at least cheap adapters) you might also run into issues, https://wiki.freebsd.org/arm/RockChip#Known_issues

PCIe Gen2 training very sensitive from what I recall, it seems to work on my boards at least but I would imagine it's highly dependent on device being used.

```
igb0@pci0:1:0:0:        class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x1734 subdevice=0x11cf
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 10 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
                 max read 512
                 link x2(x4) speed 2.5(5.0) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 2 corrected
    ecap 0003[140] = Serial 1 001999fffffe8651
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 8 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0004
                     VF Device ID 0x1520
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 0018[1c0] = LTR 1
    ecap 000d[1d0] = ACS 1
```

I'd be happy to add any PCIe you test on your RockPro64 success or failure, just send me a PM of what you've tested etc.
You can look at the table to determine required information, https://wiki.freebsd.org/arm/RockChip#Tested_PCIe_devices_on_RockPro64


----------



## JohnnySorocil (Oct 23, 2022)

Phishfry said:


> I see my PCIe bus is only running at GEN1 and GEN2 is possible. Anybody figure this out?


Mine shows Gen2 for NVMe:

```
nda0: nvme version 1.2 x4 (max x4) lanes PCIe Gen2 (max Gen3) link
nda0: 244198MB (500118192 512 byte sectors)
```

Probably because of "max-link-speed". Don't know if it has real impact on performance.

```
grep -A5 pcie0 my.dts
&pcie0 {
        bus-scan-delay-ms = <1000>;
        ep-gpios = <&gpio2 RK_PA4 GPIO_ACTIVE_HIGH>;
        max-link-speed = <2>;
        num-lanes = <4>;
        pinctrl-names = "default";
```


----------



## Phishfry (Oct 23, 2022)

Yes I actually saw somebody using an overlay for GEN2 PCIe when researching.
That is why I asked.
Looking at U-boot I see nothing about PCIe link speeds in either pci or nvme .

Which brings me to my next question:
What is your source for DTS overlays on RockPro64. I see some for rockchip64 and rk3399?
I see sources for RockPi4 overlays from Raxda that might work.

I want to add some goodies via overlay starting with one-wire.


----------



## diizzy (Oct 23, 2022)

rk3399-rockpro64.dtsi « rockchip « arm64 « src « device-tree « contrib « sys - src - FreeBSD source tree
					






					cgit.freebsd.org
				



Seems to suggest that Gen2 is enabled (I use 13.1-RELEASE)


----------



## Phishfry (Oct 23, 2022)

From that I found my second LED.

```
diy_led: led-1 {
            label = "diy";
            default-state = "off";
            gpios = <&gpio0 RK_PA2 GPIO_ACTIVE_HIGH>;
```

So `gpioctl -f /dev/gpioc0 -t 2` or shorthand (because gpioc0 is default bus) `gpioctl -t 2`
It is already set to an OUTPUT and is ready to use. It controls the Red LED right next to the White LED.


----------



## Phishfry (Oct 23, 2022)

I am compiling a custom kernel. Stripped out anything but rk3399 from GENERIC arm64 kernconf.
Stupidly I forgot to use the -j flag.
How many cores do I have? Is it really 6 or 4+2? 4 rockchip cores plus 2 cortez cores?
What should I use for optimal compiling?


----------



## diizzy (Oct 23, 2022)

You have 6 cores at your disposal, kernel is fine using all 6 but be careful with memory hungry ports.
You also have https://wiki.freebsd.org/arm/RockChip#Optimization_for_SoCs


----------



## Phishfry (Oct 24, 2022)

I am getting down to the nitty gritty now decompiling the dtb
`dtc -I dtb -O dts rk3399-rockpro64.dtb > rk3399-rockpro64.dts`

Looks like kernel compiling caught all 6 cores too.

```
>>> Kernel build for ROCKPRO64 completed on Sun Oct 23 21:19:57 EDT 2022
--------------------------------------------------------------
>>> Kernel(s)  ROCKPRO64 built in 13058 seconds, ncpu: 6
--------------------------------------------------------------
```


----------



## Phishfry (Oct 24, 2022)

I have a starting point for OneWire overlay on Header Pin 15. GPIO1_A1

rockpro64-w1-gpio.dts

```
/dts-v1/;
/plugin/;

/ {
    compatible = "rockchip,rk3399";
    fragment@0 {
        target-path = "/";
        __overlay__ {
            w1: onewire@0 {
                compatible = "w1-gpio";
                pinctrl-names = "default";
                gpios = <&gpio1 1 0>;
                status = "okay";
            };
        };
    };
};
```

`dtc -O dtb -o /boot/dtb/overlays/rockpro64-w1-gpio.dtbo -@ rockpro64-w1-gpio.dts`

Add your loader settings:
/boot/loader.conf

```
ow_load="YES"
owc_load="YES"
ow_temp_load="YES"
fdt_overlays="rockpro64-w1-gpio.dtbo"
```

Reboot and check the output:


```
sysctl -a | grep ow_temp
ow_temp0: <Advanced One Wire Temperature> romid 28:ee:6c:0c:20:16:01:8a on ow0
dev.ow_temp.0.parasite: 0
dev.ow_temp.0.reading_interval: 10000
dev.ow_temp.0.badread: 0
dev.ow_temp.0.badcrc: 25
dev.ow_temp.0.temperature: 22.812C
dev.ow_temp.0.%parent: ow0
dev.ow_temp.0.%pnpinfo: romid=28:ee:6c:0c:20:16:01:8a
dev.ow_temp.0.%location:
dev.ow_temp.0.%driver: ow_temp
dev.ow_temp.0.%desc: Advanced One Wire Temperature
dev.ow_temp.%parent:
```


----------



## Phishfry (Oct 25, 2022)

I had to recompile kernel and it is much quicker using 6 cores.

```
--------------------------------------------------------------
>>> Kernel build for ROCKPRO64 completed on Mon Oct 24 19:02:44 EDT 2022
--------------------------------------------------------------
>>> Kernel(s)  ROCKPRO64 built in 2710 seconds, ncpu: 6, make -j6
--------------------------------------------------------------
```


----------



## Phishfry (Oct 27, 2022)

I have a bench-top adjustable DC Power Supply with meters.
My RockPro64 is using .3 amps at 12VDC at bootup and during kernel compile it spikes at .4A.
I was expecting much more with NVMe.

Rigging up Meanwell DRC-40A with 12V-3.4ah SLA battery and I wanted to get a grip on power consumption.


----------

