# SSE/SSE2 instructions in libcrypto after build with CPUTYPE=pentium-mmx



## DRRB (Oct 27, 2022)

Hi,

I built iso images to install 13.1 on a Pentium 233 MMX machine (no practical use, for fun and to learn how to use the build system).
The commands I used for the build are:


```
make -j12 buildworld buildkernel -DWITHOUT_CLEAN TARGET=i386 TARGET_ARCH=i386 CPUTYPE=pentium-mmx
make TARGET=i386 TARGET_ARCH=i386 CPUTYPE=pentium-mmx release
make TARGET=i386 TARGET_ARCH=i386 CPUTYPE=pentium-mmx DESTDIR=/home/denis/FBSDSRC/my-pentium install
```

I then installed the system with Qemu on an SD card and booted to the physical machine with an IDE/SD adapter.

Everything works almost correctly, but /lib/libcrypto.so.111 uses SSE, SSE2, etc, instructions that the CPU does not support.
So I sometimes get "illegal instruction" errors (with `sudo` for example). And I checked the binary with `elfx86exts` (https://github.com/pkgw/elfx86exts) :


```
$ file /tmp/libcrypto.so.111
/tmp/libcrypto.so.111: ELF 32-bit LSB shared object, Intel 80386, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.1, stripped

$ elfx86exts /tmp/sudo
NOT64BITMODE (push)
CMOV (cmovs)
CPU Generation: Unknown

$ elfx86exts /tmp/libcrypto.so.111 
NOT64BITMODE (push)
SSE2 (pxor)
SSE1 (movups)
AES (aesenc)
SSSE3 (pshufb)
SSE41 (pextrd)
CPU Generation: Unknown
```
Did I miss something about the build?


----------



## SirDice (Oct 27, 2022)

I'm not entirely sure but it may have something to do with the `-DWITHOUT_CLEAN`. If you want to reduce your build times you should use WITH_META_MODE instead. At least try a full build without the `-DWITHOUT_CLEAN` to make sure everything actually gets rebuilt properly. 



			MetaMode - FreeBSD Wiki


----------



## _martin (Oct 27, 2022)

Handbook mentions this:
	
	



```
All Intel®
processors 486 or higher are supported. However, binaries released by the project are compiled
for the 686 processor, so a special build will be needed for 486 and 586 systems.
```
I was not able to find what those special flags are. I'd assume it would require `-march=i586` in CFLAGS.


----------



## DRRB (Oct 27, 2022)

I just rebuilt everything without -WITHOUT CLEAN.

I do see `-march=pentium-mmx` in the options used by the compiler in the build process.

I'm starting to think the problem is with the libcrypto. But that would mean that 13.1 is not fully functional if the CPU is < i686.


----------



## SirDice (Oct 27, 2022)

Keep in mind that i386 is Tier 2 since 13.0. So it doesn't get the same attention as it did before 13.0 when it was still Tier 1.


----------



## _martin (Oct 27, 2022)

DRRB said:


> But that would mean that 13.1 is not fully functional if the CPU is < i686.


It should be according to 13.1 handbook. I can't back that claim up but it's an official docs so that's good enough. 

`llc13 -march=x86 -mattr=help` gives plenty of interesting options, maybe it's worth a try building libcrypt with those only and test that.


----------



## _martin (Oct 27, 2022)

I've i386 VM around so I didn't need to use TARGET specification. On 13.1p2 src I did:
	
	



```
# cd /usr/src/lib/libnetbsd/
# make CPUTYPE=i586
# cd ../libcrypt/
# make CPUTYPE=i586

# elfx86exts /usr/obj/usr/src/i386.i386/lib/libcrypt/libcrypt.so.5
NOT64BITMODE (push)
CPU Generation: Unknown
# file /usr/obj/usr/src/i386.i386/lib/libcrypt/libcrypt.so.5
/usr/obj/usr/src/i386.i386/lib/libcrypt/libcrypt.so.5: ELF 32-bit LSB shared object, Intel 80386, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.1, not stripped
#
```
edit: mhm, but to be fair I have the same results with pentium-mmx. No additional flags give me additional cmov instruction.


----------



## _martin (Oct 27, 2022)

Out of curiosity I'll try to build it on amd64. But first I need to build cross compiling tools 
	
	



```
cd /usr/src
make TARGET=i386 TARGET_ARCH=i386 CPUTYPE=i586 xdev-build
```
and clang takes ages to build. But once that is built I'll do the same commands I did above.


----------



## DRRB (Oct 27, 2022)

`libcrypt` is fine for me too, the problem is with `libcrypto` (`secure/lib/libcrypto`)


----------



## DRRB (Oct 27, 2022)

I think I have something...

In usr/src/secure/lib/libcrypto/Makefile.inc I have :


```
[...]
CFLAGS+=        -DECP_NISTZ256_ASM
CFLAGS+=        -DPOLY1305_ASM
.elif defined(ASM_i386)
CFLAGS+=        -DOPENSSL_IA32_SSE2
CFLAGS+=        -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT
CFLAGS+=        -DOPENSSL_BN_ASM_GF2m
CFLAGS+=        -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM
CFLAGS+=        -DRC4_ASM
CFLAGS+=        -DMD5_ASM
[...]
```

This `OPENSSL_IA32_SSE2` is not a good sign


----------



## _martin (Oct 27, 2022)

DRRB said:


> `libcrypt` is fine for me too, the problem is with `libcrypto` (`secure/lib/libcrypto`)


Ok, now that's important -- I was building the wrong library then. I'm building the correct one on my i386 VM to see the results. edit: Results are the same as you have.

Let's see what that illegal instruction is. Do you have a coredump? If you're executing in on setuid sudo you need `kern.sugid_coredump` set. Once you have coredump do the `gdb /path/to/binary /path/to/core` and execute `x/4i $pc` in gdb.


----------



## DRRB (Oct 27, 2022)

Of course I had to build and install gdb for the target, otherwise it's no fun 


```
(gdb) x/4i $pc
=> 0x40f520 <getenv>:   endbr32 
   0x40f524 <getenv+4>: push   %ebp
   0x40f525 <getenv+5>: mov    %esp,%ebp
   0x40f527 <getenv+7>: push   %ebx
```

`endbr32` is the culprit. This is a CET instruction that does not exist in pentium.


----------



## _martin (Oct 27, 2022)

Heh, yeah, I bet.

Ok, this is not SSE instruction. Can you do x/4b $pc on it too? For Pentium this should be a noop. This is a compiler problem. Maybe there are some extra CFLAGS parameters to work around this. I never had this problem (don't run any OS on older CPU). It's a wild guess, maybe try adding extra `export CFLAGS=-mmanual-endbr -fcf-protection=branch` and try to build that lib again.

Actually, looking at my lib:
	
	



```
fbsd13i(/usr/obj/usr/src/i386.i386/secure/lib/libcrypto)# readelf -a libcrypto.so.111|grep getenv
0034a9b4 00000307 R_386_JUMP_SLOT     00000000 getenv
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getenv@FBSD_1.0 (17)
    24: 00000000000e9b40    77 FUNC    LOCAL  DEFAULT   14 ossl_safe_getenv
   766: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getenv
fbsd13i(/usr/obj/usr/src/i386.i386/secure/lib/libcrypto)#
```
getenv() will come from libc.

And on my i386 system (provided by FreeBSD) I have 
	
	



```
fbsd13i(/usr/obj/usr/src/i386.i386/secure/lib/libcrypto)# gdb -q /lib/libc.so.7
Reading symbols from /lib/libc.so.7...
Reading symbols from /usr/lib/debug///lib/libc.so.7.debug...
(gdb) x/4i getenv
   0x1440d0 <getenv>:    push   ebp
   0x1440d1 <getenv+1>:    mov    ebp,esp
   0x1440d3 <getenv+3>:    push   ebx
   0x1440d4 <getenv+4>:    push   edi
(gdb)
```
So CFLAGS finetunning is needed. libc was built with clang13 so new compiler is capable of doing this just fine.


----------



## SirDice (Oct 27, 2022)

_martin said:


> This is a compiler problem.


It should be reported as a bug in that case. It's either an issue with the compiler or some options are missing in the Makefile.inc for this library. It may just have been missed as i386 doesn't get as much attention nowadays.


----------



## DRRB (Oct 27, 2022)

All of these object files in `/usr/obj/[...]i386.i386/secure/lib/libcrypto` have SSE instructions :


```
x86cpuid.o
x86-gf2m.o
sha512-586.o
sha1-586.o
poly1305-x86.o
ghash-x86.o
chacha-x86.o
aesni-x86.o
```

All have sources generated by Perl scripts. For example, for `x86cpuid.o` we have `/usr/src/crypto/openssl/crypto/x86cpuid.pl` :


```
[...]
for (@ARGV) { $sse2=1 if (/-DOPENSSL_IA32_SSE2/); }
[...]
        if ($sse2) {                 
                &and    ("ecx",1<<26|1<<24);    # check SSE2 and FXSR bits      
                &cmp    ("ecx",1<<26|1<<24);
                &jne    (&label("no_sse2"));                      
                &pxor   ("xmm0","xmm0");
                &pxor   ("xmm1","xmm1");
                &pxor   ("xmm2","xmm2");
```

`pxor` is an SSE instruction found in `x86cpuid.o`
And we see `-DOPENSSL_IA32_SSE2` again.

I will try to build by removing `-DOPENSSL_IA32_SSE2` in `/usr/src/secure/lib/libcrypto/Makefile.inc`.


----------



## _martin (Oct 27, 2022)

SirDice said:


> It should be reported as a bug in that case


I've added more information in my post. It's a problem how that was built (problem on a compiling side of things) and not a problem in the source code. i386 FreeBSD 13.1 works just fine on that CPU. Here just some extra compiler flags are needed.


----------



## DRRB (Oct 27, 2022)

> Can you do x/4b $pc on it too?




```
(gdb) x/4b $pc
0x40f520 <getenv>:      0xf3    0x0f    0x1e    0xfb
```


----------



## _martin (Oct 27, 2022)

But MMX is certain set of SSE instructions. You don't have problem (at least here) with SSE. Can you run the gdb command on libc the same way I did ? On that physical box, are you able to run `/usr/bin/who` command?


----------



## DRRB (Oct 27, 2022)

Yes `/usr/bin/who` works fine.


```
$ gdb -q /lib/libc.so.7
Reading symbols from /lib/libc.so.7...
(No debugging symbols found in /lib/libc.so.7)
(gdb) x/4i getenv
   0x140770 <getenv>:   push   %ebp
   0x140771 <getenv+1>: mov    %esp,%ebp
   0x140773 <getenv+3>: push   %ebx
   0x140774 <getenv+4>: push   %edi
```

I launched a clean build, I have to wait to be able to install `base-dbg` to have `libc.so.7.debug` (if needed at some point).


----------



## _martin (Oct 27, 2022)

OK, that is strange as you can see getenv is different here. On that crashdump can you share two additional outputs ? 
	
	



```
bt
info files
```
I'd like to make sure that getenv() is coming from libc.

Actually your question and subsequent talk was about the SSE/libcrypto that the obvious thing was missed. How was the sudo built. That's why I asked now about info files. Look at the sudo I have:
	
	



```
fbsd13i(~)# readelf -a /usr/local/bin/sudo|grep geten
    12: 000000000040efa0   178 FUNC    GLOBAL DEFAULT   14 getenv
fbsd13i(~)#
```
It brings its own getenv. 
	
	



```
fbsd13i(~)# gdb -q /usr/local/bin/sudo
Reading symbols from /usr/local/bin/sudo...
(No debugging symbols found in /usr/local/bin/sudo)
(gdb) x/4i getenv
   0x40efa0 <getenv>:    endbr32
   0x40efa4 <getenv+4>:    push   ebp
   0x40efa5 <getenv+5>:    mov    ebp,esp
   0x40efa7 <getenv+7>:    push   ebx
(gdb)
```
So the focus should be on port build and not the system.

Actually you could "bruteforce" and use very (very) ugly command to check if base is ok:`find /lib -type f -exec objdump -d {} \; | grep endbr`. You can use that on PATH too to check other binaries. objdump comes from binutils.


----------



## DRRB (Oct 27, 2022)

```
[New LWP 100071]
Core was generated by `sudo'.
Program terminated with signal SIGILL, Illegal instruction.
Privileged opcode.
#0  0x0040f520 in getenv ()
(gdb) bt
#0  0x0040f520 in getenv ()
#1  0x208f4150 in ?? () from /lib/libthr.so.3
#2  0x208f4007 in ?? () from /lib/libthr.so.3
#3  0x2043b24f in ?? () from /libexec/ld-elf.so.1
#4  0x2043965a in ?? () from /libexec/ld-elf.so.1
#5  0x204374ae in ?? () from /libexec/ld-elf.so.1
(gdb) info files
Symbols from "/usr/local/bin/sudo".
Local core dump file:
        `/usr/home/denis/tmp/./sudo.core', file type elf32-i386-freebsd.
        0x0042f000 - 0x00431000 is load1
        0x00431000 - 0x00433000 is load2
        0x2044b000 - 0x2044c000 is load3
        0x2044c000 - 0x2044d000 is load4
        0x2044d000 - 0x2044e000 is load5
        0x2044e000 - 0x20470000 is load6
        0x20482000 - 0x20483000 is load7
        0x20483000 - 0x20484000 is load8
        0x2049d000 - 0x2049e000 is load9
        0x2049e000 - 0x2049f000 is load10
        0x204a9000 - 0x204aa000 is load11
        0x204aa000 - 0x204ab000 is load12
        0x206fa000 - 0x20710000 is load13
        0x20710000 - 0x20711000 is load14
        0x20711000 - 0x20715000 is load15
        0x208ba000 - 0x208be000 is load16
        0x208be000 - 0x208bf000 is load17
        0x208bf000 - 0x208c4000 is load18
        0x208c4000 - 0x208e4000 is load19
        0x20902000 - 0x20903000 is load20
        0x20903000 - 0x20904000 is load21
        0x20904000 - 0x20940000 is load22
        0x20a00000 - 0x20c00000 is load23
        0xffbdf000 - 0xffbff000 is load24
        0xffbff000 - 0xffc00000 is load25
        While running this, GDB does not access memory from...
Local exec file:
        `/usr/local/bin/sudo', file type elf32-i386-freebsd.
        Entry point: 0x40dd78
        0x00400194 - 0x004001a9 is .interp
        0x004001ac - 0x004001f4 is .note.tag
        0x004001f4 - 0x00401184 is .dynsym
        0x00401184 - 0x00401376 is .gnu.version
        0x00401378 - 0x004013e8 is .gnu.version_r
        0x004013e8 - 0x00401abc is .gnu.hash
        0x00401abc - 0x0040228c is .hash
        0x0040228c - 0x00402fe8 is .dynstr
        0x00402fe8 - 0x00403058 is .rel.dyn
        0x00403058 - 0x00403778 is .rel.plt
        0x00403778 - 0x00409cc0 is .rodata
        0x00409cc0 - 0x0040a4a4 is .eh_frame_hdr
        0x0040a4a4 - 0x0040cb0c is .eh_frame
        0x0040db10 - 0x0042df84 is .text
        0x0042df84 - 0x0042df90 is .init
        0x0042df90 - 0x0042df9c is .fini
        0x0042dfa0 - 0x0042edf0 is .plt
        0x0042fdf0 - 0x0042fdf8 is .ctors
        0x0042fdf8 - 0x0042fe00 is .dtors
        0x0042fe00 - 0x0042fe04 is .jcr
        0x0042fe04 - 0x0042fe08 is .init_array
        0x0042fe08 - 0x0042fef8 is .dynamic
        0x0042fef8 - 0x0042ff00 is .got
        0x0042ff00 - 0x0043029c is .got.plt
        0x0043129c - 0x00431a80 is .data
        0x00431a80 - 0x00432978 is .bss
        0x20470174 - 0x2047018c is .note.tag in /lib/libutil.so.9
        0x2047018c - 0x2047134c is .dynsym in /lib/libutil.so.9
        0x2047134c - 0x20471584 is .gnu.version in /lib/libutil.so.9
        0x20471584 - 0x20471604 is .gnu.version_r in /lib/libutil.so.9
        0x20471604 - 0x2047193c is .gnu.hash in /lib/libutil.so.9
        0x2047193c - 0x20472224 is .hash in /lib/libutil.so.9
        0x20472224 - 0x20472dbb is .dynstr in /lib/libutil.so.9
        0x20472dbc - 0x20473004 is .rel.dyn in /lib/libutil.so.9
        0x20473004 - 0x20473644 is .rel.plt in /lib/libutil.so.9
        0x20473644 - 0x2047450c is .rodata in /lib/libutil.so.9
        0x2047450c - 0x204748c0 is .eh_frame_hdr in /lib/libutil.so.9
        0x204748c0 - 0x20475b74 is .eh_frame in /lib/libutil.so.9
        0x20476b80 - 0x20480d38 is .text in /lib/libutil.so.9
        0x20480d38 - 0x20480d44 is .init in /lib/libutil.so.9
        0x20480d44 - 0x20480d50 is .fini in /lib/libutil.so.9
        0x20480d50 - 0x204819e0 is .plt in /lib/libutil.so.9
        0x204829e0 - 0x204829e8 is .ctors in /lib/libutil.so.9
        0x204829e8 - 0x204829f0 is .dtors in /lib/libutil.so.9
        0x204829f0 - 0x204829f4 is .jcr in /lib/libutil.so.9
        0x204829f4 - 0x204829f8 is .init_array in /lib/libutil.so.9
        0x204829f8 - 0x20482b8c is .data.rel.ro in /lib/libutil.so.9
        0x20482b8c - 0x20482c4c is .dynamic in /lib/libutil.so.9
        0x20482c4c - 0x20482c74 is .got in /lib/libutil.so.9
        0x20483c74 - 0x20483cd4 is .data in /lib/libutil.so.9
        0x20483cd4 - 0x20484000 is .got.plt in /lib/libutil.so.9
        0x20484000 - 0x20485e8d is .bss in /lib/libutil.so.9
        0x20486174 - 0x2048618c is .note.tag in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2048618c - 0x204873cc is .dynsym in /usr/local/libexec/sudo/libsudo_util.so.0
        0x204873cc - 0x20487614 is .gnu.version in /usr/local/libexec/sudo/libsudo_util.so.0
        0x20487614 - 0x204876b4 is .gnu.version_r in /usr/local/libexec/sudo/libsudo_util.so.0
        0x204876b4 - 0x20487c24 is .gnu.hash in /usr/local/libexec/sudo/libsudo_util.so.0
        0x20487c24 - 0x2048854c is .hash in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2048854c - 0x204898b7 is .dynstr in /usr/local/libexec/sudo/libsudo_util.so.0
        0x204898b8 - 0x20489c48 is .rel.dyn in /usr/local/libexec/sudo/libsudo_util.so.0
        0x20489c48 - 0x2048a0c8 is .rel.plt in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2048a0c8 - 0x2048b9d5 is .rodata in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2048b9d8 - 0x2048c034 is .eh_frame_hdr in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2048c034 - 0x2048ded0 is .eh_frame in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2048eed0 - 0x2049bb68 is .text in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049bb68 - 0x2049bb74 is .init in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049bb74 - 0x2049bb80 is .fini in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049bb80 - 0x2049c490 is .plt in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049d490 - 0x2049d498 is .ctors in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049d498 - 0x2049d4a0 is .dtors in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049d4a0 - 0x2049d4a4 is .jcr in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049d4a4 - 0x2049d4a8 is .init_array in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049d4a8 - 0x2049d730 is .data.rel.ro in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049d730 - 0x2049d810 is .dynamic in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049d810 - 0x2049d844 is .got in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049d844 - 0x2049da90 is .got.plt in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049ea90 - 0x2049eb38 is .data in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049eb38 - 0x2049efd4 is .bss in /usr/local/libexec/sudo/libsudo_util.so.0
        0x2049f174 - 0x2049f18c is .note.tag in /usr/local/lib/libintl.so.8
        0x2049f18c - 0x2049f76c is .dynsym in /usr/local/lib/libintl.so.8
        0x2049f76c - 0x2049f828 is .gnu.version in /usr/local/lib/libintl.so.8
        0x2049f828 - 0x2049f868 is .gnu.version_r in /usr/local/lib/libintl.so.8
        0x2049f868 - 0x2049f928 is .gnu.hash in /usr/local/lib/libintl.so.8
        0x2049f928 - 0x2049fc20 is .hash in /usr/local/lib/libintl.so.8
        0x2049fc20 - 0x204a010a is .dynstr in /usr/local/lib/libintl.so.8
        0x204a010c - 0x204a01ec is .rel.dyn in /usr/local/lib/libintl.so.8
        0x204a01ec - 0x204a043c is .rel.plt in /usr/local/lib/libintl.so.8
        0x204a043c - 0x204a13cc is .rodata in /usr/local/lib/libintl.so.8
        0x204a13cc - 0x204a15b8 is .eh_frame_hdr in /usr/local/lib/libintl.so.8
        0x204a15b8 - 0x204a1ea0 is .eh_frame in /usr/local/lib/libintl.so.8
        0x204a2ea0 - 0x204a8118 is .text in /usr/local/lib/libintl.so.8
        0x204a8118 - 0x204a8124 is .init in /usr/local/lib/libintl.so.8
        0x204a8124 - 0x204a8130 is .fini in /usr/local/lib/libintl.so.8
        0x204a8130 - 0x204a85e0 is .plt in /usr/local/lib/libintl.so.8
        0x204a95e0 - 0x204a95e8 is .ctors in /usr/local/lib/libintl.so.8
        0x204a95e8 - 0x204a95f0 is .dtors in /usr/local/lib/libintl.so.8
        0x204a95f0 - 0x204a95f4 is .jcr in /usr/local/lib/libintl.so.8
        0x204a95f4 - 0x204a95f8 is .init_array in /usr/local/lib/libintl.so.8
        0x204a95f8 - 0x204a9628 is .data.rel.ro in /usr/local/lib/libintl.so.8
        0x204a9628 - 0x204a96e8 is .dynamic in /usr/local/lib/libintl.so.8
        0x204a96e8 - 0x204a9728 is .got in /usr/local/lib/libintl.so.8
        0x204aa728 - 0x204aa734 is .data in /usr/local/lib/libintl.so.8
        0x204aa734 - 0x204aa868 is .got.plt in /usr/local/lib/libintl.so.8
        0x204aa868 - 0x204ab995 is .bss in /usr/local/lib/libintl.so.8
        0x204ac174 - 0x204ac18c is .note.tag in /lib/libcrypto.so.111
        0x204ac18c - 0x204bddac is .dynsym in /lib/libcrypto.so.111
        0x204bddac - 0x204c0130 is .gnu.version in /lib/libcrypto.so.111
        0x204c0130 - 0x204c02f0 is .gnu.version_d in /lib/libcrypto.so.111
        0x204c02f0 - 0x204c0360 is .gnu.version_r in /lib/libcrypto.so.111
        0x204c0360 - 0x204c798c is .gnu.hash in /lib/libcrypto.so.111
        0x204c798c - 0x204d07a4 is .hash in /lib/libcrypto.so.111
        0x204d07a4 - 0x204e6368 is .dynstr in /lib/libcrypto.so.111
        0x204e6368 - 0x204fa348 is .rel.dyn in /lib/libcrypto.so.111
        0x204fa348 - 0x204fe8f0 is .rel.plt in /lib/libcrypto.so.111
        0x204fe8f0 - 0x20543788 is .rodata in /lib/libcrypto.so.111
        0x20543788 - 0x2054f1e4 is .eh_frame_hdr in /lib/libcrypto.so.111
        0x2054f1e4 - 0x20584878 is .eh_frame in /lib/libcrypto.so.111
        0x20586000 - 0x206f0b18 is .text in /lib/libcrypto.so.111
        0x206f0b18 - 0x206f0b29 is .init in /lib/libcrypto.so.111
        0x206f0b2c - 0x206f0b38 is .fini in /lib/libcrypto.so.111
        0x206f0b40 - 0x206f96a0 is .plt in /lib/libcrypto.so.111
        0x206fa6a0 - 0x206fa6a8 is .ctors in /lib/libcrypto.so.111
        0x206fa6a8 - 0x206fa6b0 is .dtors in /lib/libcrypto.so.111
        0x206fa6b0 - 0x206fa6b4 is .jcr in /lib/libcrypto.so.111
        0x206fa6b4 - 0x206fa6b8 is .init_array in /lib/libcrypto.so.111
        0x206fa6b8 - 0x20710880 is .data.rel.ro in /lib/libcrypto.so.111
        0x20710880 - 0x20710958 is .dynamic in /lib/libcrypto.so.111
        0x20710958 - 0x20710f08 is .got in /lib/libcrypto.so.111
        0x20711f08 - 0x20712b18 is .data in /lib/libcrypto.so.111
        0x20712b18 - 0x20714df8 is .got.plt in /lib/libcrypto.so.111
        0x20714df8 - 0x20717c64 is .bss in /lib/libcrypto.so.111
        0x20718194 - 0x207181ac is .note.tag in /lib/libc.so.7
        0x207181ac - 0x20724a3c is .dynsym in /lib/libc.so.7
        0x20724a3c - 0x2072634e is .gnu.version in /lib/libc.so.7
        0x20726350 - 0x20726468 is .gnu.version_d in /lib/libc.so.7
        0x20726468 - 0x2072c304 is .gnu.hash in /lib/libc.so.7
        0x2072c304 - 0x20732754 is .hash in /lib/libc.so.7
        0x20732754 - 0x2073c6c0 is .dynstr in /lib/libc.so.7
        0x2073c6c0 - 0x207419a0 is .rel.dyn in /lib/libc.so.7
        0x207419a0 - 0x20743590 is .rel.plt in /lib/libc.so.7
        0x20743590 - 0x207567c6 is .rodata in /lib/libc.so.7
        0x207567c8 - 0x2075ef74 is .eh_frame_hdr in /lib/libc.so.7
        0x2075ef74 - 0x207860a4 is .eh_frame in /lib/libc.so.7
        0x207870b0 - 0x208b5c08 is .text in /lib/libc.so.7
        0x208b5c08 - 0x208b5c14 is .init in /lib/libc.so.7
        0x208b5c14 - 0x208b5c20 is .fini in /lib/libc.so.7
        0x208b5c20 - 0x208b9410 is .plt in /lib/libc.so.7
        0x208b9410 - 0x208b9430 is .iplt in /lib/libc.so.7
        0x208ba430 - 0x208bade0 is .tdata in /lib/libc.so.7
        0x208bade0 - 0x208bb5f9 is .tbss in /lib/libc.so.7
        0x208bade0 - 0x208bade8 is .ctors in /lib/libc.so.7
        0x208bade8 - 0x208badf0 is .dtors in /lib/libc.so.7
        0x208badf0 - 0x208badf4 is .jcr in /lib/libc.so.7
        0x208badf4 - 0x208bae00 is .init_array in /lib/libc.so.7
        0x208bae00 - 0x208be644 is .data.rel.ro in /lib/libc.so.7
        0x208be644 - 0x208be648 is .fini_array in /lib/libc.so.7
        0x208be648 - 0x208be718 is .dynamic in /lib/libc.so.7
        0x208be718 - 0x208beb90 is .got in /lib/libc.so.7
        0x208bfb90 - 0x208c2528 is .data in /lib/libc.so.7
        0x208c2528 - 0x208c3334 is .got.plt in /lib/libc.so.7
        0x208c3340 - 0x208e3064 is .bss in /lib/libc.so.7
        0x208e4174 - 0x208e418c is .note.tag in /lib/libthr.so.3
        0x208e418c - 0x208e5c1c is .dynsym in /lib/libthr.so.3
        0x208e5c1c - 0x208e5f6e is .gnu.version in /lib/libthr.so.3
        0x208e5f70 - 0x208e6088 is .gnu.version_d in /lib/libthr.so.3
        0x208e6088 - 0x208e60f8 is .gnu.version_r in /lib/libthr.so.3
        0x208e60f8 - 0x208e690c is .gnu.hash in /lib/libthr.so.3
        0x208e690c - 0x208e765c is .hash in /lib/libthr.so.3
        0x208e765c - 0x208e98a4 is .dynstr in /lib/libthr.so.3
        0x208e98a4 - 0x208e9de4 is .rel.dyn in /lib/libthr.so.3
        0x208e9de4 - 0x208ea1ac is .rel.plt in /lib/libthr.so.3
        0x208ea1ac - 0x208ea95c is .rodata in /lib/libthr.so.3
        0x208ea95c - 0x208eb498 is .eh_frame_hdr in /lib/libthr.so.3
        0x208eb498 - 0x208ee938 is .eh_frame in /lib/libthr.so.3
        0x208ef940 - 0x20900e88 is .text in /lib/libthr.so.3
        0x20900e88 - 0x20900e94 is .init in /lib/libthr.so.3
        0x20900e94 - 0x20900ea0 is .fini in /lib/libthr.so.3
        0x20900ea0 - 0x20901640 is .plt in /lib/libthr.so.3
        0x20902640 - 0x20902648 is .ctors in /lib/libthr.so.3
        0x20902648 - 0x20902650 is .dtors in /lib/libthr.so.3
        0x20902650 - 0x20902654 is .jcr in /lib/libthr.so.3
        0x20902654 - 0x2090265c is .init_array in /lib/libthr.so.3
        0x2090265c - 0x2090287c is .data.rel.ro in /lib/libthr.so.3
        0x2090287c - 0x20902954 is .dynamic in /lib/libthr.so.3
        0x20902954 - 0x209029bc is .got in /lib/libthr.so.3
        0x209039bc - 0x20903a94 is .data in /lib/libthr.so.3
        0x20903a94 - 0x20903c84 is .got.plt in /lib/libthr.so.3
        0x20903cc0 - 0x2090d028 is .bss in /lib/libthr.so.3
        0x20431174 - 0x2043118c is .note.tag in /libexec/ld-elf.so.1
        0x2043118c - 0x2043134c is .dynsym in /libexec/ld-elf.so.1
        0x2043134c - 0x20431384 is .gnu.version in /libexec/ld-elf.so.1
        0x20431384 - 0x2043149c is .gnu.version_d in /libexec/ld-elf.so.1
        0x2043149c - 0x20431570 is .gnu.hash in /libexec/ld-elf.so.1
        0x20431570 - 0x20431658 is .hash in /libexec/ld-elf.so.1
        0x20431658 - 0x2043184d is .dynstr in /libexec/ld-elf.so.1
        0x20431850 - 0x20432168 is .rel.dyn in /libexec/ld-elf.so.1
        0x20432168 - 0x20435b6c is .rodata in /libexec/ld-elf.so.1
        0x20435b6c - 0x20435d90 is .eh_frame_hdr in /libexec/ld-elf.so.1
        0x20435d90 - 0x2043648c is .eh_frame in /libexec/ld-elf.so.1
        0x20437490 - 0x2044aaf7 is .text in /libexec/ld-elf.so.1
        0x2044baf8 - 0x2044becc is .data.rel.ro in /libexec/ld-elf.so.1
        0x2044becc - 0x2044bf44 is .dynamic in /libexec/ld-elf.so.1
        0x2044bf44 - 0x2044bfa4 is .got in /libexec/ld-elf.so.1
        0x2044cfa4 - 0x2044d10c is .data in /libexec/ld-elf.so.1
        0x2044d10c - 0x2044d118 is .got.plt in /libexec/ld-elf.so.1
        0x2044d118 - 0x2044dc24 is .bss in /libexec/ld-elf.so.1
```


----------



## _martin (Oct 27, 2022)

Yop, as I said above and you just confirmed, getenv() is coming from sudo itself and not base.

```
(gdb) x/4i $pc
=> 0x40f520 <getenv>:   endbr32
   0x40f524 <getenv+4>: push   %ebp

   0x0040db10 - 0x0042df84 is .text
```
Address 0x40f520 is within range of .text of the sudo binary.

edit: I was not able to find a way to do this in clang. Self healing problem with gcc (11.3) though:
	
	



```
cd /usr/ports/security/sudo
export CC=/usr/local/bin/gcc
make install clean
```
and all is working ok (no endbr instruction).

Now endbr instruction has its place, especially in setuid binaries as it limits the attack vector in case of an issue in the binary.


----------



## DRRB (Oct 27, 2022)

This explains why openssl works well :


```
$ ldd /usr/bin/openssl
/usr/bin/openssl:
        libssl.so.111 => /usr/lib/libssl.so.111 (0x204c3000)
        libcrypto.so.111 => /lib/libcrypto.so.111 (0x20544000)
        libc.so.7 => /lib/libc.so.7 (0x207b0000)
        libthr.so.3 => /lib/libthr.so.3 (0x2097c000)

$ openssl dgst -sha512 .bashrc 
SHA512(.bashrc)= c6f8279c3344bf53a0c40ecbd9c636f363551b4c3ee9aa451a8d332248cb86fea5d74bf57b1c710f47daefad07f8904a453c54851836dd50edb4f1849d0dfdb0
```

I automatically assumed the problem was with libcrypto. I didn't think to try another tool using the library.


----------



## DRRB (Oct 27, 2022)

So it's a port problem. I'll stick with `doas` on this old machine. Thank you very much for your help Martin.


----------



## _martin (Oct 27, 2022)

From security perspective it doesn't matter much if you use doas or sudo. I'll try to find out if there are options for clang to generate code without endbr though.

So it's the `-fcf-protection` in CFLAGS that enables this. `-fcf-protection=none` explicitly disables it. Even if I do this explicitly as env variable, or even do a global settings in /etc/make.conf port has its own logic. configure detects it's possible to use it and enables it no matter what. Snippet of a port compilation:
	
	



```
/bin/sh ../../libtool  --mode=compile cc -c -I../../include -I../.. -I. -I.  -DLIBDIR=\"/usr/local/lib\" -DLOCALEDIR=\"/usr/local/share/locale\"  -D_PATH_SUDOERS=\"/usr/local/etc/sudoers\"  -D_PATH_CVTSUDOERS_CONF=\"/usr/local/etc/cvtsudoers.conf\"  -DSUDOERS_UID=0 -DSUDOERS_GID=0  -DSUDOERS_MODE=0440 -DZLIB_CONST -D_FORTIFY_SOURCE=2 -fcf-protection=none -march=i586  -g -O0 -fstack-protector-strong -fno-strict-aliasing  -fvisibility=hidden   -fstack-protector-strong -fstack-clash-protection -fcf-protection ./fmtsudoers.c
```
Where you can see both protection none and *-fcf-protection* enabled. I don't know, I guess libtool or something is screwing this up. Personally I would think that if I do `-march=i586` (not shown in this snippet) compiler should not use that feature or fail if feature that cpu doesn't support is requested.

Do you have compiler on your old box? If so would you be willing to make this test? Compile this test code I wrote:
	
	



```
#include <stdio.h>

void hello();

int main() {
    void (*gr)(void) = &hello;
    gr();

    return 0;
}

void hello() {
    printf("hello\n");
}
```
And compile it first with `cc -g -o test test.c` and then with `cc -g -fcf-protection -o test test.c`. Then do the comparison with `objdump -d test` and look for the endbr instruction in the 2nd example.


----------



## DRRB (Oct 27, 2022)

Yep. Here it is :


```
$ objdump -d testmartin | grep -i endbr

$ objdump -d testmartinPROT | grep -i endbr
  401720:       f3 0f 1e fb             endbr32 
  401750:       f3 0f 1e fb             endbr32
```

And, of course :


```
$ ./testmartin
hello

$ ./testmartinPROT 
Illegal instruction (core dumped)
```


----------



## _martin (Oct 27, 2022)

But did you compile it on that actual HW ? I was curious myself so I connected my old jig I use for some stuff. The oldest I've got is Celeron 500MHz with 128MB ram. Interestingly enough 13.1 booted up without any issue. But as it's i686 endbr is not a problem.

The next best thing I've got is qemu. I've emulated pentium processor:
	
	



```
CPU: Pentium/P55C (1497.62-MHz 586-class CPU)
  Origin="GenuineIntel"  Id=0x543  Family=0x5  Model=0x4  Stepping=3
  Features=0x8003bf<FPU,VME,DE,PSE,TSC,MSR,MCE,CX8,APIC,MMX>
```
But it has no problem with the endbr instruction. I was not able to find a way to disable it.


----------



## DRRB (Oct 28, 2022)

> But did you compile it on that actual HW ?



Yes :


```
$ sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu'
hw.machine: i386
hw.model: Pentium/P55C
hw.ncpu: 1
hw.machine_arch: i386

$ grep -A 4 CPU /var/run/dmesg.boot
CPU: Pentium/P55C (233.87-MHz 586-class CPU)
  Origin="GenuineIntel"  Id=0x543  Family=0x5  Model=0x4  Stepping=3
  Features=0x8001bf<FPU,VME,DE,PSE,TSC,MSR,MCE,CX8,MMX>
real memory  = 268435456 (256 MB)
avail memory = 232763392 (221 MB)
```

If I understand correctly, the `-fcf-protection` option generates an `endbr32` instruction which only exists with i686+ and should not be used with an i586/i486/i386 CPU. `-fcf-protection` is used by default if `--disable-hardening` is not specified with the `configure` script.

Is it a compiler issue that shouldn't allow the option if `-march` < i686 ?
Or a sudo autoconf file issue that shouldn't always use `-fcf-protection` ?
Or a port issue that should make `--disable-hardening` an option for the build ?


----------



## _martin (Oct 28, 2022)

This is interesting. While qemu does emulate CPU seemingly the same as you have it can still understand endbr instruction. This has something to do with the guest isolation probably.

In my opinion clang should not generate code on your machine for your CPU that is not able to run (with -march at least). But that's only my opinion, this would be a question for clang devel team. Not sure if anybody would bother to check this for your CPU but if they are willing to share their opinion on this, that would be great.

Yes, correct, it's the flow control mechanism that mitigates certain types of exploits (use of indirect jumps). There are other mitigations that should be allowed (stack guard for example) and I'm not sure what does `--disable-hardening` macro do. In this scenario you'd be better off with gcc.

My Makefile I used for test:
	
	



```
CFLAGS=-g -march=i586 -fcf-protection

test:    test.c
    cc $(CFLAGS) -o test test.c

clean:
    rm -f *.o test
```


----------



## _martin (Oct 28, 2022)

I was googling around and indeed, for gcc this was a bug filled and fixed 98667. The resolution is what I'd expect: `x86: Error on -fcf-protection with incompatible target`.

I don't have high hopes anybody is going to touch it but I did open a PR for it: PR 267401.


----------



## _martin (Nov 4, 2022)

Issue is now fixed on upstream LLVM.


----------



## DRRB (Nov 18, 2022)

FYI. There is a similar situation regarding the use of the `-O2` flag. I opened an issue on LLVM GitHub: 59061.


----------



## _martin (Nov 18, 2022)

I guess these CPUs are not used that often any more. And it seems virtual ones emulate things they should not.


----------



## Crivens (Nov 18, 2022)

You might be surprised. I once reported an illegal instruction in a compiler because it was trapping in valgrind but not on silicon.


----------

