# Kernel compile error 11.2 -> 11.3 upgrade



## cds1 (Feb 21, 2020)

Hello,

I get the error below compiling the GENERIC kernel from both 11.3-RelEng and 11-Stable:


```
cc -c -x assembler-with-cpp -DLOCORE -O2 -pipe -march=native -fno-strict-aliasing -g -nostdinc  -I. -I../../.. -I../../../contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD  -MF.depend.support.o -MTsupport.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999  -Werror ../../../amd64/amd64/support.S
../../../amd64/amd64/support.S:1112:2: error: instruction requires: AVX-512 ISA
 vmovdqa64 %zmm0, %gs:0x3c0
 ^
../../../amd64/amd64/support.S:1113:2: error: instruction requires: AVX-512 ISA
 vpxor %zmm0, %zmm0, %zmm0
 ^
../../../amd64/amd64/support.S:1116:2: error: instruction requires: AVX-512 DQ ISA
 vorpd (%rdx), %zmm0, %zmm0
 ^
../../../amd64/amd64/support.S:1117:2: error: instruction requires: AVX-512 DQ ISA
 vorpd (%rdx), %zmm0, %zmm0
 ^
../../../amd64/amd64/support.S:1129:2: error: instruction requires: AVX-512 ISA
 vmovdqa64 %gs:0x3c0, %zmm0
 ^
*** Error code 1

Stop.
make: stopped in /usr/src/11.3/sys/amd64/compile/GENERIC
```

Here is what I can find for CPU info from dmesg:


```
CPU: Intel(R) Core(TM) i3 CPU         530  @ 2.93GHz (2942.53-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x20652  Family=0x6  Model=0x25  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x98e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
```


From what I can tell, the CPU is missing the AVX-512 instruction set. I can't find anything in the Errata regarding this error. Is the CPU just too old now?

TIA


----------



## SirDice (Feb 21, 2020)

What's in /etc/make.conf and/or /etc/src.conf?


----------



## cds1 (Feb 21, 2020)

I forgot to mention I came across this post, so I emptied my make.conf and don't have a src.conf


----------



## PMc (Feb 21, 2020)

cds1 said:


> Is the CPU just too old now?



That definitely not.
Figuring it out... if I remove my CPUTYPE from /etc/make.conf, I get this one:

```
cc -c -x assembler-with-cpp -DLOCORE -O2 -pipe -fno-strict-aliasing  -g -nostdinc  -I. -I../../.. -I../../../contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD  -MF.depend.support.o -MTsupport.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999  -Werror ../../../amd64/amd64/support.S
```

So, how did You get that `-march=native` in there? 
But, thats not really the problem....
Question is, how You try to do this kernel build? You often cannot just build a kernel from a newer release. You first have to build the compiler and tools from that release, and then use these to build the kernel. Otherwise, strange things may happen. There is some advice at the end of /usr/src/UPDATING, but I for my part, when switching releases, prefer to do the strict sequence: make buildworld, make buildkernel, make installkernel, go single-user, make installworld (add the appropriate parameters and mergemasters).


----------



## cds1 (Feb 21, 2020)

I've compiled both with and without my original make.conf .. the code block in the original post was with a make.conf

Here's without a make.conf


```
cc -c -O2 -pipe -fno-strict-aliasing  -g -nostdinc  -I. -I../../.. -I../../../contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD  -MF.depend.sys_machdep.o -MTsys_machdep.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -Werror  ../../../amd64/amd64/sys_machdep.c
--- support.o ---
../../../amd64/amd64/support.S:1112:2: error: instruction requires: AVX-512 ISA
vmovdqa64 %zmm0, %gs:0x3c0
^
../../../amd64/amd64/support.S:1113:2: error: instruction requires: AVX-512 ISA
vpxor %zmm0, %zmm0, %zmm0
^
../../../amd64/amd64/support.S:1116:2: error: instruction requires: AVX-512 DQ ISA
vorpd (%rdx), %zmm0, %zmm0
^
../../../amd64/amd64/support.S:1117:2: error: instruction requires: AVX-512 DQ ISA
vorpd (%rdx), %zmm0, %zmm0
^
../../../amd64/amd64/support.S:1129:2: error: instruction requires: AVX-512 ISA
vmovdqa64 %gs:0x3c0, %zmm0
^
*** [support.o] Error code 1

make: stopped in /usr/src/11.3/sys/amd64/compile/GENERIC
--- sys_machdep.o ---
ctfconvert -L VERSION -g sys_machdep.o
--- machdep.o ---
ctfconvert -L VERSION -g machdep.o
--- pmap.o ---
ctfconvert -L VERSION -g pmap.o
1 error

make: stopped in /usr/src/11.3/sys/amd64/compile/GENERIC
```


----------



## PMc (Feb 22, 2020)

Okay. I tried and couldn't reproduce that error.
The current CPU should not matter, because that compiler should also be able to cross-compile. So it seems somehow your build-environment is hosed.
So I would start over with a consistent src-tree, doing `make buildworld` first, and then make buildkernel.
And, btw, this /usr/src/11.3/sys/amd64/compile/GENERIC doesn't look sane to me. The src tree is full of relative symlinks, it should be kept in one piece at the proper place. If You want separate trees for different versions to build, then I would suggest You do it this way - that's how it works for me:

make a build directory, like /build/11.3
install the currently running system into that place, e.g. `make installworld DESTDIR=/build/11.3` (**) (or just copy the base installation to there).
unpack the source tree into /build/11.3/usr/src
`chroot /build/11.3`
`cd /usr/src; make buildworld; make buidlkernel KERNCONF=GENERIC`
then, when successful, for installation take /build/11.3/usr/src and /build/11.3/usr/obj and place them at /usr/src and /usr/obj of whereever the stuff shall be installed (jail, different system, ...). Then run the proper installation command.
(**) Addendum: I forgot something: At this point, after doing the make installworld, it is also necessary to do `make distribution DESTDIR=/build/11.3` to properly populate /etc/ and friends in the chroot.


----------



## cds1 (Mar 1, 2020)

I've been working on this off and on as time permits, but I can't get past this error.


```
# make buildworld
--------------------------------------------------------------
>>> World build started on Sun Mar  1 14:46:05 UTC 2020
--------------------------------------------------------------

--------------------------------------------------------------
>>> Rebuilding the temporary build tree
--------------------------------------------------------------
rm -rf /usr/obj/usr/src/tmp
rm -rf /usr/obj/usr/src/lib32
mkdir -p /usr/obj/usr/src/tmp/lib
mkdir -p /usr/obj/usr/src/tmp/lib/casper
mkdir -p /usr/obj/usr/src/tmp/usr
mkdir -p /usr/obj/usr/src/tmp/legacy/bin
mkdir -p /usr/obj/usr/src/tmp/legacy/usr
mtree -deU -f /usr/src/etc/mtree/BSD.usr.dist  -p /usr/obj/usr/src/tmp/legacy/usr >/dev/null
mtree: unknown user `root'
mtree: failed at line 6 of the specification
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src
```

That error occurs while building world in the chroot'd (/build/11.3) /usr/src. Any suggestions? 

Thanks for your help thus far.


----------



## PMc (Mar 3, 2020)

cds1 said:


> mtree: unknown user `root'
> ...
> That error occurs while building world in the chroot'd (/build/11.3) /usr/src. Any suggestions?



Ups - big sorry! This is certainly my fault, I forgot a step. See addendum above.


----------



## cds1 (Mar 11, 2020)

I've made a bit more progress on this. For posterity's sake, devfs needs to be available in the chroot'd environment

e.g.
`mount -t devfs devfs /build/11.3/dev/`

I'm unable to get past this build error now. I've blown away the 11.3 source, and started with a fresh pull. Same result


```
c++  -O2 -pipe -march=native -I/usr/obj/usr/src/tmp/usr/src/lib/clang/libllvm -I/usr/src/contrib/llvm/lib/Target/AArch64 -I/usr/src/contrib/llvm/lib/Target/ARM -I/usr/src/contrib/llvm/lib/Target/Mips -I/usr/src/contrib/llvm/lib/Target/PowerPC -I/usr/src/contrib/llvm/lib/Target/Sparc -I/usr/src/contrib/llvm/lib/Target/X86 -I/usr/src/lib/clang/include -I/usr/src/contrib/llvm/include -DLLVM_BUILD_GLOBAL_ISEL -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DNDEBUG -DLLVM_DEFAULT_TARGET_TRIPLE=\"x86_64-unknown-freebsd11.3\" -DLLVM_HOST_TRIPLE=\"x86_64-unknown-freebsd11.3\" -DDEFAULT_SYSROOT=\"/usr/obj/usr/src/tmp\" -DLLVM_TARGET_ENABLE_AARCH64 -DLLVM_TARGET_ENABLE_ARM -DLLVM_TARGET_ENABLE_MIPS -DLLVM_TARGET_ENABLE_POWERPC -DLLVM_TARGET_ENABLE_SPARC -DLLVM_TARGET_ENABLE_X86 -DLLVM_NATIVE_ASMPARSER=LLVMInitializeX86AsmParser -DLLVM_NATIVE_ASMPRINTER=LLVMInitializeX86AsmPrinter -DLLVM_NATIVE_DISASSEMBLER=LLVMInitializeX86Disassembler -DLLVM_NATIVE_TARGET=LLVMInitializeX86Target -DLLVM_NATIVE_TARGETINFO=LLVMInitializeX86TargetInfo -DLLVM_NATIVE_TARGETMC=LLVMInitializeX86TargetMC -ffunction-sections -fdata-sections -MD -MF.depend.Support_YAMLParser.o -MTSupport/YAMLParser.o -Qunused-arguments -I/usr/obj/usr/src/tmp/legacy/usr/include  -std=c++11 -fno-exceptions -fno-rtti -stdlib=libc++ -Wno-c++11-extensions  -c /usr/src/contrib/llvm/lib/Support/YAMLParser.cpp -o Support/YAMLParser.o
/usr/src/contrib/llvm/lib/Support/YAMLParser.cpp:2156:7: error: expected expression
      /? Set this to end iterator.
      ^
/usr/src/contrib/llvm/lib/Support/YAMLParser.cpp:2156:8: error: expected expression
      /? Set this to end iterator.
       ^
/usr/src/contrib/llvm/lib/Support/YAMLParser.cpp:2156:14: error: expected ':'
      /? Set this to end iterator.
             ^
             :
/usr/src/contrib/llvm/lib/Support/YAMLParser.cpp:2156:8: note: to match this '?'
      /? Set this to end iterator.
       ^
/usr/src/contrib/llvm/lib/Support/YAMLParser.cpp:2156:10: error: use of undeclared identifier 'Set'
      /? Set this to end iterator.
         ^
4 errors generated.
*** Error code 1

Stop.
make[4]: stopped in /usr/src/lib/clang/libllvm
*** Error code 1

Stop.
make[3]: stopped in /usr/src/lib/clang
*** Error code 1

Stop.
make[2]: stopped in /usr/src
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src
```


----------



## tingo (Mar 11, 2020)

What version is clang on your 11.2 system? See /usr/src/UPDATING  for details.


----------



## cds1 (Mar 11, 2020)

> What version is clang on your 11.2 system? See /usr/src/UPDATING for details.




```
~$ clang --version
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
Target: x86_64-unknown-freebsd11.2
Thread model: posix
InstalledDir: /usr/bin
```

I've been through UPDATING. Should I be using  WITHOUT_CLANG and WITH_GCC to get 11.3 off the ground?


----------



## tingo (Mar 11, 2020)

Nope, your clang seems new enough. You could try the advice in entry 20141231 about building and installing libc++ first, perhaps it helps. Not sure, your system setup seems strange somehow. Do you have enough free disk space to build everything?


----------

