# Accidentally broken libssl



## heximal (Apr 25, 2020)

Hi.
I decided to upgrade redmine from ports, did everything as usual (`portsnap fetch`, `portsnap update` first, then `make install`).
Installation were completed without errors.
After that I discovered apachectl refuses to run with following error:
`httpd: Syntax error on line 145 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: Shared object "libssl.so.9" not found, required by "mod_ssl.so"`

There appeared `/usr/local/lib/libssl.so.11` and no `libssl.so.9` in `/usr/local/lib` (as I understand, redmine upgraded libssl). 
I tried to symlink so.9 to so.11 but in this case mod_ssl fails with 
`Undefined symbol "get_rfc3526_prime_8192"` error. It looks like 9 & 11 are not 100% backward compatible.

I'm lucky that apache is still running, but if I try to restart it, it'll fail, there is no doubt.

Can anyone help?

FreeBSD 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64


----------



## getopt (Apr 25, 2020)

heximal said:


> FreeBSD 10.3-RELEASE-p11


... is no more supported. See








						Release Information
					

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.




					www.freebsd.org
				





			Chapter 23. Updating and Upgrading FreeBSD


----------



## heximal (Apr 25, 2020)

What about all depended software? Like apache (specifically mod_ssl)? As I understand they are statically linked to libssl.so.9? 

`ldd /usr/local/libexec/apache24/mod_ssl.so`

produces following:


```
/usr/local/libexec/apache24/mod_ssl.so:
    libssl.so.9 => /usr/local/lib/libssl.so.9 (0x801638000)
    libcrypto.so.9 => /usr/local/lib/libcrypto.so.9 (0x801a00000
    libcrypt.so.5 => /lib/libcrypt.so.5 (0x801ee5000)
    libthr.so.3 => /lib/libthr.so.3 (0x802105000)
    libc.so.7 => /lib/libc.so.7 (0x800821000)
```

`make.conf` fragment:

`DEFAULT_VERSIONS+=ssl=openssl`


----------



## getopt (Apr 25, 2020)

pyret said:


> but bailing wire and duct tape holds most of the world together.


... until it falls apart. Good practice is running a supported FreeBSD version.


----------



## heximal (Apr 25, 2020)

pyret said:


> Or link libssl.so.9 and the others from the new directory to where Apache expects them. Hackey as can be, but bailing wire and duct tape holds most of the world together.


This leads to errors like `Undefined symbol 'xxx'` as I mentioned in my first post. I tried to link it to libssl.so.11 and libssl.so.7 which I managed to find across the system.


----------



## heximal (Apr 25, 2020)

getopt said:


> Good practice is running a supported FreeBSD version.


Yes, I totally agree, but unfortunately don't feel myself confident enough to perform upgrade on production server, although I previously upgraded it from 9 to 10. It wasn't easy for me at all. Thanks to all computer gods I managed to make it up and running.
More than three years of uptime since that moment)


----------



## getopt (Apr 25, 2020)

Oh, you have got a production server that is obsolete. Read this:








						Topics about unsupported FreeBSD versions
					

The FreeBSD Forums cater primarily to end-users and systems administrators. As such, the Forums focus almost exclusively on FreeBSD versions that are officially supported according to the official FreeBSD website. Since resources are scarce, the FreeBSD Forums strongly suggest that anyone asking...




					forums.freebsd.org
				




For what reason do you 'feel not confident enough for upgrading'?
The purpose of this Forum is to assist on such tasks.


----------



## heximal (Apr 25, 2020)

getopt said:


> For what reason do you 'feel not confident enough for upgrading'?


System administration is not my primary branch, unfortunately.


----------



## getopt (Apr 25, 2020)

heximal said:


> although I previously upgraded it from 9 to 10. It wasn't easy for me at all.


Obviously your resources were sufficient for that. Don't you know how to start or what exactly is your problem? Did you break your server or is it still functioning?


----------



## heximal (Apr 26, 2020)

getopt said:


> Did you break your server or is it still functioning?



The server is ok now, but it I try to restart for instance apache service it won't start again.
I'm trying to find out whether it's better to restore previous libssl version or do something different. I would appreciate if somebody give me a hint.


----------



## heximal (Apr 26, 2020)

If I upgrade to FreeBSD 11 will it fix all all my problems?


----------



## getopt (Apr 26, 2020)

heximal said:


> If I upgrade to FreeBSD 11 will it fix all all my problems?


Probably, but no guaranty. 
First make a backup and be sure you can restore from it.
Upgrading from 10 to 11 was not critical here. From 11 to 12 may be more tricky. But my experience says nothing on what may expect you.
You also could setup a new box with 12.1-RELEASE and restore config files and data to it. Some additional work may be necessary, this way you do not need to touch your production server until you have a working new system.


----------



## heximal (May 2, 2020)

OK, I upgraded FreeBSD to 11.3
As I expected apache started to fail on launch. The error it produces is

`httpd: Syntax error on line 145 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: Shared object "libssl.so.9" not found, required by "mod_ssl.so"`

I did

`# portsnap fetch update
# pkg-static install -f pkg
# portmaster -afG`

and finally

`/usr/sbin/freebsd-update install`


----------



## heximal (May 2, 2020)

Is it ok I upgraded from 10.3 right to 11.3? or I should first upgrade to all version between 10.3 and 11.3?


----------



## PMc (May 2, 2020)

heximal said:


> What about all depended software? Like apache (specifically mod_ssl)? As I understand they are statically linked to libssl.so.9?


They're not statically linked. They're just built with that version, and different major numbers on shared libs are not supposed to be compatible.


heximal said:


> If I upgrade to FreeBSD 11 will it fix all all my problems?


It will fix no problems, it will just be a version that gets fixes for known exploits, and most people seem to like that.


heximal said:


> Is it ok I upgraded from 10.3 right to 11.3? or I should first upgrade to all version between 10.3 and 11.3?


That should work.

As far as I see, your main shortcoming is that you have a prod machine, but no test machine - where you could run a backup, run an upgrade, then restore the backup, and whatever else until you see how it all works out.

The next fancy question is how you obtained that apache24 in the first place. Because mine is locally built, and on 11.3 it runs with libssl.so.8 (from base). Probably your apache24 is built to use ssl from ports (which afaik is not the default). So probably somebody configured that (probably because they needed a newer ssl version at that time). Which makes me wonder how you usually update your apache (on a prod server there is some reason to update it occasionally).

And anyway, after upgrading, it might be a good idea to run `pkg check -d -n -a` to see what else is amiss.


----------



## heximal (May 2, 2020)

PMc said:


> The next fancy question is how you obtained that apache24 in the first place. Because mine is locally built, and on 11.3 it runs with libssl.so.8 (from base). Probably your apache24 is built to use ssl from ports (which afaik is not the default). So probably somebody configured that (probably because they needed a newer ssl version at that time). Which makes me wonder how you usually update your apache (on a prod server there is some reason to update it occasionally).


i had apache24 installed via pkg. Just now I tried to build it from ports, but got compilation errors.
after upgrade to 11.3  `ldd /usr/local/libexec/apache24/mod_ssl.so` says mod_ssl linked to libssl.so.8 (on 10.3 it was linked to libssl.so.9)

`ldd /usr/local/libexec/apache24/mod_ssl.so
/usr/local/libexec/apache24/mod_ssl.so:
    libssl.so.8 => /usr/lib/libssl.so.8 (0x80123b000)
    libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
    libcrypt.so.5 => /lib/libcrypt.so.5 (0x801a76000)
    libthr.so.3 => /lib/libthr.so.3 (0x801c95000)
    libc.so.7 => /lib/libc.so.7 (0x800825000)`





PMc said:


> And anyway, after upgrading, it might be a good idea to run `pkg check -d -n -a` to see what else is amiss.



here is the output of `pkg check -d -n -a`:

`pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Checking all packages: 100%
freetds-msdblib is missing a required shared library: libiconv.so.3`


----------



## heximal (May 2, 2020)

pyret said:


> LMAO I gave you an option that most likely would have fixed your problem, and if it did or didn't, it posed essentially no risk.  You didn't listen to advice before.



What did I do wrong? I followed your recommendations.


----------



## richardtoohey2 (May 4, 2020)

If you are now building from ports - do you _have_ to use LibreSSL?

If you build with OpenSSL you might get things back up and running.


----------



## heximal (May 4, 2020)

What finally helped me to make my server up and running again.
1. I deinstalled openssl installed previously with `pkg` util (`pkg delete openssl`)
2. Just in case I deinstalled openssl port and built it again
`cd /usr/ports/security/openssl/
make deinstall clean
make install clean`

3. Reinstalled Apache Portability Library from ports (without this step I had no success)
`cd /usr/ports/devel/apr1
make deinstall clean
make install clean`

4. Reinstalled apache from ports
`cd /usr/ports/www/apache24/
make deinstall clean
make install clean`

This fixed all static linking issues. Thanks to all digital gods there is no more symlink workarounds in my system. This gives me a hope next OS upgrade won't be so hard.


----------



## SirDice (May 4, 2020)

heximal said:


> This fixed all static linking issues.


Please learn the difference between static and dynamic linking. There is no static linking involved here.





__





						Dynamic linker - Wikipedia
					






					en.wikipedia.org
				












						Static and Dynamic Linking in Operating Systems - GeeksforGeeks
					

A Computer Science portal for geeks. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice/competitive programming/company interview Questions.




					www.geeksforgeeks.org


----------



## D-FENS (May 4, 2020)

heximal said:


> What finally helped me to make my server up and running again.
> 1. I deinstalled openssl installed previously with `pkg` util (`pkg delete openssl`)
> 2. Just in case I deinstalled openssl port and built it again
> `cd /usr/ports/security/openssl/
> ...



You mixed binaries which were not built against each other and this created a binary mismatch.
Your Apache mod_ssl.so was installed via a binary that was built by FreeBSD and your libssl.so was compiled by you via the port. In this case mod_ssl.so calls into libssl.so and expects a certain binary interface (ABI) but libssl.so has an incompatible one.
Both the dependency and the dependent binary need to be built against each other (either install both via pkg, or build both of them from ports).
Of course, the same applies for everything that depends on libssl.so - recursively! This might be quite a long list.

You could also try to build the libssl port and configure it identically to the FreeBSD build server (what you download via pkg). Unfortunatelly I have not done this and I can't tell you exactly how to do it.


----------



## richardtoohey2 (May 4, 2020)

Unless you changed /etc/make.conf I don't think you've done what you think you've done.

You've built Apache and the APR against the system's OpenSSL, and you've separately installed the port OpenSSL.

But easier to use the system OpenSSL (and it should make future upgrades easier) and if you're back in business then that's good.


----------

