# Building mosquitto-auth-plug on FreeBSD



## holala (Jan 15, 2021)

Hello. I'm struggling with mosquitto-auth-plug on FreeBSD. I'm following the (pretty basic) instruction provided by the authors. I do successfully build mosquitto-1.6.9.
I do successfully build also mosquitto-auth-plug, but later I'm getting undefined symbols, when mosquitto starts and loads the auth-plug.so



```
1610714616: mosquitto version 1.6.9 starting
1610714616: Config loaded from /etc/mosquitto/mosquitto.conf.
1610714616: Loading plugin: /home/mzk/mosquitto-auth-plug/auth-plug.so
1610714616: Error: Unable to load auth plugin "/home/mzk/mosquitto-auth-plug/auth-plug.so".
1610714616: Load error: /home/mzk/mosquitto-auth-plug/auth-plug.so: Undefined symbol "mosquitto_client_id"
```

What worries me is that when I do ldd on a linux machine, there are a lot more dependent libraries than on my FreeBSD machine. Here is the output from Ubuntu:

```
ldd auth-plugin.so 
    linux-vdso.so.1 =>  (0x00007ffc017c8000)
    libcdb.so.1 => /usr/lib/x86_64-linux-gnu/libcdb.so.1 (0x00007f0cbba17000)
    libmysqlclient.so.20 => /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 (0x00007f0cbb460000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0cbb243000)
    libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f0cbaf6e000)
    libhiredis.so.0.13 => /usr/lib/x86_64-linux-gnu/libhiredis.so.0.13 (0x00007f0cbad61000)
    libpq.so.5 => /usr/lib/x86_64-linux-gnu/libpq.so.5 (0x00007f0cbab31000)
    libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f0cba8e0000)
    libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f0cba673000)
    libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f0cba22e000)
    libmosquitto.so.1 => /usr/lib/x86_64-linux-gnu/libmosquitto.so.1 (0x00007f0cba01e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0cb9c54000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0cb9a50000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f0cb9836000)
    libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f0cb95ce000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0cb924c000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0cb8f43000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0cb8d2d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0cbbe28000)
    libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f0cb8ae3000)
    liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f0cb88d4000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f0cb86b9000)
    libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f0cb849e000)
    libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f0cb825d000)
    libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f0cb7f2d000)
    libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f0cb7cfa000)
    librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f0cb7ade000)
    libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f0cb78a8000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0cb76a0000)
    libcares.so.2 => /usr/lib/x86_64-linux-gnu/libcares.so.2 (0x00007f0cb748f000)
    libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f0cb71bd000)
    libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f0cb6f8e000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f0cb6d8a000)
    libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f0cb6b7f000)
    libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f0cb6976000)
    libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f0cb66ec000)
    libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f0cb644a000)
    libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f0cb6217000)
    libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f0cb6001000)
    libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f0cb5d9d000)
    libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f0cb5b8a000)
    libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f0cb5957000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f0cb56d7000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f0cb54d3000)
    libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f0cb52aa000)
    libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f0cb509b000)
    libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f0cb4e50000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f0cb4c18000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f0cb4a10000)
```

And here is the output from my FreeBSD machine:

```
ldd ./auth-plug.so 
./auth-plug.so:
    libmariadb.so.3 => /usr/local/lib/mysql/libmariadb.so.3 (0x800683000)
    libcrypto.so.111 => /lib/libcrypto.so.111 (0x800e00000)
    libc.so.7 => /lib/libc.so.7 (0x80024e000)
    libm.so.5 => /lib/libm.so.5 (0x8006e4000)
    libssl.so.111 => /usr/lib/libssl.so.111 (0x800716000)
    libz.so.6 => /lib/libz.so.6 (0x8007ae000)
    libthr.so.3 => /lib/libthr.so.3 (0x8007ca000)
```

I know I shouldn't compare the linux package as it might have been built with all the mosquitto-auth-plug backends, but it is a hint for me that probably on FreeBSD the library dependencies are not properly resolved.

Since I'm very far from porting (and building) software, I'll be very helpful if someone can give me some hints on this.

Thanks!


----------



## SirDice (Jan 15, 2021)

Can you post a link to the instructions so other people can have a look too? It's often a case of making sure dependencies are installed prior to the build and setting the correct options for `./configure` (if it uses configure).


----------



## holala (Jan 15, 2021)

Sure.

I'm using this fork of mosquitto-auth-plug
----








						GitHub - coldfire84/mosquitto-auth-plug: Authentication plugin for Mosquitto with multiple back-ends (MySQL, Redis, CDB, SQLite3)
					

Authentication plugin for Mosquitto with multiple back-ends (MySQL, Redis, CDB, SQLite3) - GitHub - coldfire84/mosquitto-auth-plug: Authentication plugin for Mosquitto with multiple back-ends (MySQ...




					github.com
				



----

The building instructions are untouched:


> In order to compile the plugin you'll require:
> 
> 
> 
> ...



BTW I found unresolved, but closed issue posted on the original author's github page:








						mosquitto_client_id: symbol not found · Issue #397 · jpmens/mosquitto-auth-plug
					

Here is my mosquitto.conf: persistence true persistence_location /mosquitto/data/ log_dest file /mosquitto/log/mosquitto.log ... auth_plugin /mosquitto/src/mosquitto-auth-plug/auth-plug.so auth_opt...




					github.com
				




The mosquitto source code can be downloaded from https://mosquitto.org/files/

The install instructions are:


> The following packages can be used to add features to mosquitto. All of them
> are optional.
> 
> * openssl
> ...



There is no readme.txt file inside the mosquitto archive. The text in readme.md is no more informative, just saying that install instructions can be found at https://mosquitto.org/download/ which is untrue.

This isn't a big deal since I'm not sure that the problem is in mosquitto source. Neither I do believe it is in mosquitto-auth-plug, but I suspect something with the library linking...


----------



## SirDice (Jan 15, 2021)

holala said:


> ```
> Config loaded from /etc/mosquitto/mosquitto.conf.
> ```


That needs patching. Third-party software has to be installed with a /usr/local prefix, the config should therefor be in /usr/local/etc/mosquitto/mosquitto.conf. 

What does `make printconfig` output?


----------



## holala (Jan 15, 2021)

SirDice said:


> That needs patching.


mosquitto loads the config file with the -c argument, so I'm fine with it now. I agree that once ported it should utilize /usr/local/etc/

Do you believe that the problem is related to mosquitto and not to mosquitto-auth-plug?

This is printconfig for mosquitto-auth-plug

```
gmake printconfig
Selected backends:         MySQL
Using mosquitto source dir: ../mosquitto-1.6.9
OpenSSL install dir:        /usr

If you changed the backend selection, you might need to 'make clean' first

CFLAGS:  -I/usr/local/include  -I../mosquitto-1.6.9/src/ -I../mosquitto-1.6.9/lib/ -fPIC -Wall -Werror -DBE_MYSQL -I/usr/local/include/mysql -I/usr/local/include/mysql/mysql -I/src -DDEBUG=1 -I/usr/include
LDFLAGS: -L/usr/local/lib   -L../mosquitto-1.6.9/lib/
LDADD:   -L/usr/local/lib/mysql/ -lmariadb -L/usr/lib -lcrypto -lmosquitto
```


----------



## SirDice (Jan 15, 2021)

holala said:


> Do you believe that the problem is related to mosquitto and not to mosquitto-auth-plug?


Looking back at the error it looks like you're running the mosquitto application from your home directory, I suspect that the auth-plugin needs a library from the application and it can't find that. The Linux ldd(1) output seems to confirm that:

```
libmosquitto.so.1 => /usr/lib/x86_64-linux-gnu/libmosquitto.so.1 (0x00007f0cba01e000)
```

You might need to use LD_PRELOAD to temporarily include the application's lib directory in your home directory if you want to test it from there.


----------



## holala (Jan 15, 2021)

Well, the mosquitto application is systemwide installed (gmake install), and I can invoke it from anywhere.
The mosquitto-auth-plug is sitting in my home directory and I'm pointing it in mosquitto.conf.

Do I properly export LD_PRELOAD?
export LD_PRELOAD=/home/mzk/mosquitto-1.6.9/lib/libmosquitto.so

Also, would ldd return also missing libraries? Or it reports only the existing (and found) libraries? I believe it wold tell if can't find some library and makes me curious why libmosquitto.so is missing in my FreeBSD's ldd report...


----------



## SirDice (Jan 15, 2021)

holala said:


> Also, would ldd return also missing libraries? Or it reports only the existing (and found) libraries?


It shows a list of libraries that are linked to the executable or library you're examining. It would show 'missing' if they can't be found on the system.

For some reason your auth-plug.so hasn't been linked with libmosquitto.so.1, that's probably the reason for the "undefined Symbol" error.


----------



## holala (Jan 15, 2021)

> SirDice said:
> 
> 
> > For some reason your auth-plug.so hasn't been linked with libmosquitto.so.1, that's probably the reason for the "undefined Symbol" error.


That's exactly what I think happens. But I don't understand why the compiler or the linker stay calm.

On macOS they mention the use of additional flags:
`CFG_LDFLAGS = -undefined dynamic_lookup`

But I'm not sure this is gcc compatible. Also no idea what it does.


----------



## SirDice (Jan 15, 2021)

holala said:


> But I don't understand why the compiler or the linker stay calm.


It does appear to get the right directories for CFLAGS and LDFLAGS:

```
CFLAGS:  -I/usr/local/include  -I../mosquitto-1.6.9/src/ -I../mosquitto-1.6.9/lib/ -fPIC -Wall -Werror -DBE_MYSQL -I/usr/local/include/mysql -I/usr/local/include/mysql/mysql -I/src -DDEBUG=1 -I/usr/include
LDFLAGS: -L/usr/local/lib   -L../mosquitto-1.6.9/lib/
```
Both include ../mosquitto-1.6.9/lib/
And LDADD includes libmosquitto

```
LDADD:   -L/usr/local/lib/mysql/ -lmariadb -L/usr/lib -lcrypto -lmosquitto
```
So everything indicates it should be linked. One thing you might try:

```
LDADD:   -L/usr/local/lib/mysql/ -lmariadb -L/usr/lib -lcrypto -L/usr/local/lib -lmosquitto
```
I assume there is a libmosquitto.so.1 in /usr/local/lib.


----------



## holala (Jan 15, 2021)

SirDice, it seems that somehow I managed to compile it without the mosquitto lib. My current ldd output shows the usage of libmosquitto.so:



```
/auth-plug.so:
    libmariadb.so.3 => /usr/local/lib/mysql/libmariadb.so.3 (0x800683000)
    libcrypto.so.111 => /lib/libcrypto.so.111 (0x800e00000)
    libmosquitto.so.1 => /usr/local/lib/libmosquitto.so.1 (0x8006e4000)
    libc.so.7 => /lib/libc.so.7 (0x80024e000)
    libm.so.5 => /lib/libm.so.5 (0x8006ff000)
    libssl.so.111 => /usr/lib/libssl.so.111 (0x800731000)
    libz.so.6 => /lib/libz.so.6 (0x8007c9000)
    libthr.so.3 => /lib/libthr.so.3 (0x8010f2000)
```

However mosquitto still fails to start with the same error:

```
1610729680: Loading plugin: /home/mzk/mosquitto-auth-plug/auth-plug.so
1610729680: Error: Unable to load auth plugin "/home/mzk/mosquitto-auth-plug/auth-plug.so".
1610729680: Load error: /home/mzk/mosquitto-auth-plug/auth-plug.so: Undefined symbol "mosquitto_client_id"
```


----------



## SirDice (Jan 15, 2021)

Is /usr/local/lib/libmosquitto.so.1 from the same version as the one in ../mosquitto-1.6.9/lib/? The issue may be version dependent, I can imagine a newer (or older) version of a library not having a certain symbol.


----------



## holala (Jan 15, 2021)

`$ md5 /usr/local/lib/libmosquitto.so.1
MD5 (/usr/local/lib/libmosquitto.so.1) = c5ac5da880fd6299ec74de8c037f80bb
$ md5 /home/mzk/mosquitto-1.6.9/lib/libmosquitto.so.1 
MD5 (/home/mzk/mosquitto-1.6.9/lib/libmosquitto.so.1) = c5ac5da880fd6299ec74de8c037f80bb`

I'll try the same version on my linux box to verify whether the problem is mosquitto related or not (some users mention that earlier version works OK). But I'll need some time to get back home, so I'll be away for a while.


----------



## holala (Jan 15, 2021)

On Linux it builds and starts as expected (not tested if actually works, but it doesn't quit with error)...


----------



## holala (Jan 15, 2021)

It went deeper than I thought it would...

1. I forgot that I installed the mosquitto binary package, so I removed it. No change (I removed everything and built and installed again).
2. I switched the compiler from gcc9 to gcc 7.5 (as on my linux box). No change.
3. I did ldd and nm on auth_plug.so and interestingly on both OSes nm shows "mosquitto_client_id" (and few others) as "U" (the symbol is undefined). This doesn't stop it from working on Linux, however, despite on both OSes the same symbols are shown as undefined.
Some light on the horizon from this post:





						239128 – audio/swhplugins: fix shared library errors
					






					bugs.freebsd.org
				




There some libs failed to load due unresolved symbols.



> Log:
> Three plugins used C89 inline semantics, which resulted in unresolved
> symbols with C99 compilers.
> 
> ...


Any suggestions are welcome.

#######

Edit: just checked the compiler and I totally forgot that FreeBSD now uses clang by default. Added CC=gcc7 in both mosquitto and mosq-auth-plug, rebuilt everything. Again no luck.
Also I addred -export-dynamic in the LDFLAGS for mosq-auth-plug. From the man page:


> Pass the flag -export-dynamic to the ELF linker, on targets that
> support it. This instructs the linker to add all symbols, not only
> used ones, to the dynamic symbol table. This option is needed for
> some uses of "dlopen" or to allow obtaining backtraces from within
> a program.


###########

Edit 2:
I found that doing `cmake` inside the mosquitto dir did the job! Idk why and I'll be glad if someone explains, but here is the quick reference:
Download mosquitto 1.6.12
tar xvf...
from inside the dir you could do gmake uninstall (as root)
then cmake . (the current directory is the source directory)
gmake
gmake install (as root)
ldconfig

then clone https://github.com/coldfire84/mosquitto-auth-plug.git
cp config.mk.in config.mk
edit MOSQUITTO_SRC = ../mosquitto-1.6.12 (according to your path)
CFG_LDFLAGS =-L/usr/local/lib
gmake

Set auth_plugin in mosquitto.conf to point to your newly built auth-plug.so
and it should work now...
Whew, another day spent in software. Btw if the mosquitto readme didn't insist so much that cmake should be used on windows or Mac and make on other OSes, it would have spared me some 12 hours :/ As SirDice states, experience...


----------

