# buildworld on CPU-A, installworld on CPU-B ends up with SIGILL



## Ruslan Garipov (Nov 21, 2019)

*Prehistory*: I want to build kernel and world on a fast virtual machine for other ones (all the virtual machines are hosted on VMware EXSi hypervisors, which have slightly different physical CPUs).  I had read ``23.6. Tracking for Multiple Machines'' chapter in the handbook, and shared /usr/src and /usr/obj on the build machine to be accessed from other virtual hosts, which mount the shares.

*Problem*: After `make -j16 buildworld` has finished successfully on the build machine, I get on that machine, for example, /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/install program having the shlxq instruction (one from the BMI2 instruction set extensions). This eventually causes `make installkernel` and `make installworld` to fail with SIGILL on a virtual machine which must consume built world and kernel, and which is hosted on another ESXi instance, with older physical CPU (_read_: with CPU not implementing shlxq).

I believe the problem isn't specific to virtualized CPUs or VMware ESXi hypervisor, because hardware failure is just an expected consequence.  Therefore, I posted this on FreeBSD forums, not in VMware forums.

*Details/steps to reproduce:*

Build machine

FreeBSD-13.0-CURRENT r354802 (custom kernel, amd64).

Both /etc/make.conf and /etc/src.conf don't exist.

CPU capabilities:


```
# cpucontrol -i 7 /dev/cpuctl0
cpuid level 0x7: 0x00000000 0xd19f6ffb 0x00000018 0xbc000000
```

Bit 08 in EBX is set -- the CPU supports BMI2.


```
# make clean && rm -r /usr/obj/*
# make -j16 buildworld
```

Build succeeded, check/view the instructions:


```
$ lldb /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/install
(lldb) disassemble -a 0x26152b
...
install[0x26152b] <+139>:  shlxq  %rdx, %rsi, %rsi
...
```

Client machine

FreeBSD-13.0-CURRENT r354802 (custom kernel, amd64).


```
# cpucontrol -i 7 /dev/cpuctl0
cpuid level 0x7: 0x00000000 0x00000002 0x00000000 0xbc000000
```

Bit 08 in EBX is cleared -- the CPU doesn’t support BMI2; therefore:


```
root@:/usr/src # /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/install
pid 880 (install), jid 0, uid 0: exited on signal 4
illegal instruction
```

I get that error from above during `make installkernel` and/or `make installworld`.


```
root@:/usr/src # lldb /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/install
Current executable set to '/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/install' (x86_64)
(lldb) r
Process 878 launching
Process 878 launched: '/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/install' (x86_64)
Process 878 stopped
* thread #1, name = 'install', stop reason = signal SIGILL: privileged instruction
install`base_block_alloc:
->  0x26152b <+139>: shlxq  %rdx, %rsi, %rsi
    0x261530 <+144>: movq   $-0x1000, %rdx    ; imm = 0xF000
```

Moreover, the generated install program also contains AVX512F's instruction vpbroadcastq (supported by build machine's CPU):


```
$ lldb install
(lldb) disassemble -a 0x280821
...
install[0x280821] <+225>: vpbroadcastq %rcx, %zmm0
...
```

And that causes FreeBSD-13.0-CURRENT running on physical Intel Core i3-8300T (not supporting AVX512) to fail on that program too.

*Question*: How to disable using those instruction set extensions on my build machine? Or may be I do something wrong, and to be able to share built world and kernel between several machines I must do something else?


----------



## SirDice (Nov 22, 2019)

Topics about unsupported FreeBSD versions


----------

