# Installing FreeBSD 12.2 ppc64 in KVM on POWER9



## bloudraak (Dec 7, 2021)

I'm trying to install FreeBSD 12.0 ppc64 in KVM running on POWER9 (Talos II) with Ubuntu 20.04 as the host. Currently, it's failing with the following:


```
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Kernel entry at 0x102390 ...
panic: Standard large pages (SLB[L] = 1, PTE[LP] = 0) not supported by this system. Please enable huge page backing if running under PowerKVM.
cpuid = 0
time = 1
KDB: stack backtrace:
#0 0xc000000000729964 at ??+0
#1 0xc0000000006c1254 at ??+0
#2 0xc0000000006c1320 at ??+0
#3 0xc000000000b1a764 at ??+0
#4 0xc000000000b0b2ac at ??+0
#5 0xc000000000b04a0c at ??+0
#6 0xc000000000102440 at ??+0
Uptime: 1s
```

I created the VM using the following command:


```
virt-install \
    --name freebsd12-ppc64-01 \
    --memory 2048 \
    --virt-type=kvm \
    --disk pool=default,size=64,format=qcow2 \
    --vcpus=1 \
    --memorybacking hugepages=yes \
    --network network=br0 \
    --os-variant freebsd12.2 \
    --graphics none \
    --cdrom /var/lib/libvirt/boot/FreeBSD-12.2-RELEASE-powerpc-powerpc64-dvd1.iso \
    --debug \
    --force
```

And that spits out the following:


```
[Tue, 07 Dec 2021 06:24:23 virt-install 44891] DEBUG (cli:208) Launched with command line: /usr/share/virt-manager/virt-install --name freebsd12-ppc64-01 --memory 2048 --virt-type=kvm --disk pool=default,size=64,format=qcow2 --vcpus=1 --memorybacking hugepages=yes --network network=br0 --os-variant freebsd12.2 --graphics none --cdrom /var/lib/libvirt/boot/FreeBSD-12.2-RELEASE-powerpc-powerpc64-dvd1.iso --debug --force
[Tue, 07 Dec 2021 06:24:23 virt-install 44891] DEBUG (virt-install:207) Distilled --network options: ['network=br0']
[Tue, 07 Dec 2021 06:24:23 virt-install 44891] DEBUG (virt-install:139) Distilled --disk options: ['pool=default,size=64,format=qcow2']
[Tue, 07 Dec 2021 06:24:23 virt-install 44891] DEBUG (cli:224) Requesting libvirt URI default
[Tue, 07 Dec 2021 06:24:23 virt-install 44891] DEBUG (cli:227) Received libvirt URI qemu:///system
[Tue, 07 Dec 2021 06:24:23 virt-install 44891] DEBUG (storage:139) Found default pool name=default target=/var/lib/libvirt/images
[Tue, 07 Dec 2021 06:24:23 virt-install 44891] DEBUG (storage:208) refreshing pool=default
[Tue, 07 Dec 2021 06:24:23 virt-install 44891] DEBUG (disk:224) Creating volume 'freebsd12-ppc64-01-4.qcow2' on pool 'default'
[Tue, 07 Dec 2021 06:24:23 virt-install 44891] DEBUG (disk:358) disk.set_vol_install: name=freebsd12-ppc64-01-4.qcow2 poolxml=
<pool type='dir'>
  <name>default</name>
  <uuid>fc40ce7b-fd1e-477d-807d-912394afc9dd</uuid>
  <capacity unit='bytes'>210304475136</capacity>
  <allocation unit='bytes'>116991582208</allocation>
  <available unit='bytes'>93312892928</available>
  <source>
  </source>
  <target>
    <path>/var/lib/libvirt/images</path>
    <permissions>
      <mode>0711</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

[Tue, 07 Dec 2021 06:24:24 virt-install 44891] DEBUG (guest:463) Setting Guest osinfo name <_OsVariant name=generic>
[Tue, 07 Dec 2021 06:24:24 virt-install 44891] DEBUG (installer:398) installer.detect_distro returned=None
[Tue, 07 Dec 2021 06:24:24 virt-install 44891] DEBUG (guest:463) Setting Guest osinfo name <_OsVariant name=freebsd12.2>
[Tue, 07 Dec 2021 06:24:24 virt-install 44891] WARNING (virt-install:332) CDROM media does not print to the text console by default, so you likely will not see text install output. You might want to use --location. See the man page for examples of using --location with CDROM media
[Tue, 07 Dec 2021 06:24:24 virt-install 44891] DEBUG (virt-install:647) Guest.has_install_phase: True
[Tue, 07 Dec 2021 06:24:24 virt-install 44891] DEBUG (cli:272)
Starting install...

Starting install...
[Tue, 07 Dec 2021 06:24:24 virt-install 44891] DEBUG (storage:642) Creating storage volume 'freebsd12-ppc64-01-4.qcow2' with xml:
<volume>
  <name>freebsd12-ppc64-01-4.qcow2</name>
  <capacity>68719476736</capacity>
  <allocation>0</allocation>
  <target>
    <format type="qcow2"/>
    <features>
      <lazy_refcounts/>
    </features>
  </target>
</volume>

[Tue, 07 Dec 2021 06:24:24 virt-install 44891] DEBUG (storage:681) Using vol create flags=1
Allocating 'freebsd12-ppc64-01-4.qcow2'                                                                       |  64 GB  00:00:09     
[Tue, 07 Dec 2021 06:24:33 virt-install 44891] DEBUG (storage:687) Storage volume 'freebsd12-ppc64-01-4.qcow2' install complete.
[Tue, 07 Dec 2021 06:24:33 virt-install 44891] DEBUG (installer:441) Generated install XML:
<domain type="kvm">
  <name>freebsd12-ppc64-01</name>
  <uuid>48de4e44-cc3d-41f4-b6d7-68c54acfb12c</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://freebsd.org/freebsd/12.2"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory>2097152</memory>
  <currentMemory>2097152</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu>1</vcpu>
  <os>
    <type arch="ppc64le" machine="pseries">hvm</type>
    <boot dev="cdrom"/>
    <boot dev="hd"/>
  </os>
  <clock offset="utc"/>
  <on_reboot>destroy</on_reboot>
  <devices>
    <emulator>/usr/bin/qemu-system-ppc64le</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/freebsd12-ppc64-01-4.qcow2"/>
      <target dev="vda" bus="virtio"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/var/lib/libvirt/boot/FreeBSD-12.2-RELEASE-powerpc-powerpc64-dvd1.iso"/>
      <target dev="sda" bus="scsi"/>
      <readonly/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15"/>
    <controller type="scsi" index="0" model="virtio-scsi"/>
    <interface type="network">
      <source network="br0"/>
      <mac address="52:54:00:f1:4f:a0"/>
      <model type="virtio"/>
    </interface>
    <console type="pty"/>
    <channel type="unix">
      <source mode="bind"/>
      <target type="virtio" name="org.qemu.guest_agent.0"/>
    </channel>
  </devices>
</domain>

[Tue, 07 Dec 2021 06:24:33 virt-install 44891] DEBUG (installer:443) Generated boot XML:
<domain type="kvm">
  <name>freebsd12-ppc64-01</name>
  <uuid>48de4e44-cc3d-41f4-b6d7-68c54acfb12c</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://freebsd.org/freebsd/12.2"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory>2097152</memory>
  <currentMemory>2097152</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu>1</vcpu>
  <os>
    <type arch="ppc64le" machine="pseries">hvm</type>
    <boot dev="hd"/>
  </os>
  <clock offset="utc"/>
  <devices>
    <emulator>/usr/bin/qemu-system-ppc64le</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/freebsd12-ppc64-01-4.qcow2"/>
      <target dev="vda" bus="virtio"/>
    </disk>
    <disk type="file" device="cdrom">
      <target dev="sda" bus="scsi"/>
      <readonly/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15"/>
    <controller type="scsi" index="0" model="virtio-scsi"/>
    <interface type="network">
      <source network="br0"/>
      <mac address="52:54:00:f1:4f:a0"/>
      <model type="virtio"/>
    </interface>
    <console type="pty"/>
    <channel type="unix">
      <source mode="bind"/>
      <target type="virtio" name="org.qemu.guest_agent.0"/>
    </channel>
  </devices>
</domain>

[Tue, 07 Dec 2021 06:24:36 virt-install 44891] DEBUG (installer:496) XML fetched from libvirt object:
<domain type='kvm' id='20'>
  <name>freebsd12-ppc64-01</name>
  <uuid>48de4e44-cc3d-41f4-b6d7-68c54acfb12c</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://freebsd.org/freebsd/12.2"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='ppc64le' machine='pseries-focal'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>POWER9</model>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-ppc64le</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/freebsd12-ppc64-01-4.qcow2' index='2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/boot/FreeBSD-12.2-RELEASE-powerpc-powerpc64-dvd1.iso' index='1'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <readonly/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <model name='spapr-pci-host-bridge'/>
      <target index='0'/>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f1:4f:a0'/>
      <source network='br0' portid='ab8126fa-bcec-450c-936e-e72dbc504ac5' bridge='br0'/>
      <target dev='vnet7'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/8'/>
      <target type='spapr-vio-serial' port='0'>
        <model name='spapr-vty'/>
      </target>
      <alias name='serial0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </serial>
    <console type='pty' tty='/dev/pts/8'>
      <source path='/dev/pts/8'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
      <address type='spapr-vio' reg='0x30000000'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-20-freebsd12-ppc64-01/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Kernel entry at 0x102390 ...
panic: Standard large pages (SLB[L] = 1, PTE[LP] = 0) not supported by this system. Please enable huge page backing if running under PowerKVM.
cpuid = 0
time = 1
KDB: stack backtrace:
#0 0xc000000000729964 at ??+0
#1 0xc0000000006c1254 at ??+0
#2 0xc0000000006c1320 at ??+0
#3 0xc000000000b1a764 at ??+0
#4 0xc000000000b0b2ac at ??+0
#5 0xc000000000b04a0c at ??+0
#6 0xc000000000102440 at ??+0
Uptime: 1s
```

I followed the instructions at the Ubuntu site, and ran the following command:


```
cat /proc/meminfo | grep Huge
```

It shows huge pages are enabled


```
AnonHugePages:   2904064 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:   16384
HugePages_Free:    15360
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:        33554432 kB
```

This is persistent, meaning I can reboot the host numerous times and the command produces similar output. 

My motivation for installing FreeBSD 12.2 is to have a lab with most operating systems I can find so I can research, develop and test applications on several platforms. I am also running FreeBSD 13.


----------



## covacat (Dec 7, 2021)

freebsd kernel reads available page sizes from openfirmware device tree @ibm,segment-page-sizes
if the key is present but it does not report 16MB pages it bombs


----------



## bloudraak (Dec 7, 2021)

I'm a bit lost in configuring Ubuntu 20.04 on POWER9 to use a 16MB page size. The default huge page size remains 2M no matter what I do, even if I change the kernel parameters.  I can't seem to register a pool with a 16M page size.  The Debian docs say 16M is supported.

Am I missing something simple?


----------



## covacat (Dec 8, 2021)

can you somehow set qemus ibm,segmen-page-sizes ? or is it determined from host's config?


----------

