# /usr/local/libexec/mysqld: Undefined symbol "fdatasync"



## Michael Bushey (Apr 19, 2018)

Mysql broke. From local.err:


```
2018-04-17T22:57:18.020349Z 0 [Warning] Could not increase number of max_open_files to more than 32768 (request: 32929)
2018-04-17T22:57:18.020567Z 0 [Warning] Changed limits: table_open_cache: 16303 (requested 16384)
2018-04-17T22:57:18.181213Z 0 [Note] /usr/local/libexec/mysqld (mysqld 5.7.21-20-log) starting as process 36792 ...
/usr/local/libexec/mysqld: Undefined symbol "fdatasync"
```

I uninstalled mysql and installed percona but it's exactly the same.


```
root@server:/var/db/mysql# pkg info | grep -i percona
percona57-client-5.7.21.20     Multithreaded SQL database (client)
percona57-server-5.7.21.20     Multithreaded SQL database (server)
root@server:/var/db/mysql# freebsd-version 
11.1-RELEASE-p9
```
I've tried searching for `Undefined symbol "fdatasync"` but most responses have to do with jails. I've never used jails, just `pkg install percona57-server-5.7.21.20`

Any ideas?


----------



## Michael Bushey (Apr 19, 2018)

```
root@server:/var/db/mysql# uname -a
FreeBSD server.local 11.1-RELEASE-p4 FreeBSD 11.1-RELEASE-p4 #0: Tue Nov 14 06:12:40 UTC 2017    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
```
Not sure why uname is showing 11.1-RELEASE-p4 and freebsd-version is showing 11.1-RELEASE-p9


----------



## SirDice (Apr 19, 2018)

Michael Bushey said:


> Not sure why uname is showing 11.1-RELEASE-p4 and freebsd-version is showing 11.1-RELEASE-p9


Reboot the machine. You updated the system but haven't rebooted yet. So the new kernel isn't active. It will probably also solve the fdatasync(2) issue.


----------



## Michael Bushey (Apr 21, 2018)

I've rebooted several times.


----------



## Bobi B. (Apr 21, 2018)

Can you provide the output from the following commands: `ldd /usr/local/libexec/mysqld`, `objdump -T /lib/libc.so.7 | grep fdatasync`?

Was this machine originally installed with FreeBSD 11 or it has been upgraded?


----------



## Michael Bushey (Apr 24, 2018)

Absolutely! Thanks Bobi. 
No output from the objdump/grep command. The machine was originally FreeBSD 9 maybe, it's been upgraded a few times.


```
ldd /usr/local/libexec/mysqld
/usr/local/libexec/mysqld:
        liblz4.so.1 => /usr/local/lib/liblz4.so.1 (0x801db6000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x801fd4000)
        libssl.so.8 => /usr/lib/libssl.so.8 (0x8021f3000)
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x802600000)
        libz.so.6 => /lib/libz.so.6 (0x802a6c000)
        libm.so.5 => /lib/libm.so.5 (0x802c85000)
        librt.so.1 => /usr/lib/librt.so.1 (0x802eb0000)
        libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x8030b5000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x8032b8000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x80357f000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x80379d000)
        libthr.so.3 => /lib/libthr.so.3 (0x8039ab000)
        libc.so.7 => /lib/libc.so.7 (0x803bd3000)
        libelf.so.2 => /lib/libelf.so.2 (0x803f87000)
```


----------



## Bobi B. (Apr 24, 2018)

Michael Bushey said:


> ldd /usr/local/libexec/mysqld
> /usr/local/libexec/mysqld:
> liblz4.so.1 => /usr/local/lib/liblz4.so.1 (0x801db6000)
> ...
> ...


Dynamic loader finds libc.so.7 in the right location.


Michael Bushey said:


> No output from the objdump/grep command.


Meaning, that fdatasync(2) function is not in libc.so.7; should be there since FreeBSD 11.1.


Michael Bushey said:


> The machine was originally FreeBSD 9 maybe, it's been upgraded a few times.


Seems like some of your upgrades went wrong. Maybe you're still using libc.so.7 from FreeBSD 11.0?

Could you try the standard way: `freebsd-update fetch`, `freebsd-update install`? But be aware for any warnings/errors that might pop-up and make sure FreeBSD version `freebsd-update` prints is the correct one. Here is what it says on my end:

```
# freebsd-update fetch
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 11.1-RELEASE from update6.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 11.1-RELEASE-p9.
```


----------



## Michael Bushey (Apr 25, 2018)

Thanks for the help Bobi. 

```
# freebsd-update fetch                  
src component not installed, skipped                       
Looking up update.FreeBSD.org mirrors... 3 mirrors found.  
Fetching metadata signature for 11.1-RELEASE from update4.freebsd.org... done.                                         
Fetching metadata index... done.                           
Inspecting system... done.                                 
Preparing to download files... done.
```
Vi comes up with an empty screen, I enter :q. It then displays:
No updates needed to update system to 11.1-RELEASE-p9.

```
# freebsd-update install
src component not installed, skipped
No updates are available to install.
Run '/usr/sbin/freebsd-update fetch' first.
```


----------



## SirDice (Apr 25, 2018)

Try running `freebsd-update IDS`. It's going to complain about a few files in /etc/ (password files) but you can ignore those. I'm wondering if anything else is out of whack.


----------



## Michael Bushey (Apr 25, 2018)

Thanks SirDice, this definately found some problems.

/boot/boot1.efi has SHA256 hash c21307c44f9d5df0bf551fa528fcff3f0915af94447daf54773bf5234da79188, but should have SHA256 hash 59c51ee0b0842e4e9ccf66e0fd53f3e80467259932b725ecf90fb098d257ceb7.                                                
/boot/boot1.efifat has SHA256 hash ecbc9a22b67c3c78c5b419cb1debfcd7d9eec457f4676ae299a18bea2b1c0cf4, but should have SHA256 hash cfe3a38c96493d002cf69b5c8574e2dd76399ad3c9caf93b70ce0c4d7ddef3c2.                                             
/boot/gptboot has SHA256 hash 3f87c9f18df23c19a4ebedcc6437c36bc25e22c70c69089a887324804b3cef34, but should have SHA256 hash 5043733291ff6203f08e968d18d39a8b1e20baac113b143cd3b7761707dc351e.                                                  
/boot/gptzfsboot has SHA256 hash 45da037ea8a2e36dcde4a8466f9dcbfcefaab8a5bea6c1f60ee868a12682b052, but should have SHA256 hash 3b8466cfdf1b9fbddf492e8faf5cb77e60f42be87ea4da2e54c8fab8dcc76f73.                                               
/boot/kernel/hv_storvsc.ko has SHA256 hash d9748dcb1837779fbddf08f24e27e996bb032c09fae4db665d985cad3bd62177, but should have SHA256 hash 83a94cf55c4ebcd0d57d2d730060488a01ce644549eff11be9ac5fb95bfcb83e.                                     
/boot/loader has SHA256 hash 5663dd47b88659d9ef5ea30494ad005cc1b1f983db18dfa595117bba9b9d370e, but should have SHA256 hash 731e71871bf601a033f0421e3ce05a43ef1f0f6d296e4efd2ed270f58eb02e49.                                                   
/boot/loader.efi has SHA256 hash 339222ec1f4b906af7c5c20fd47566e541a2ce26710e53d2c13bfbda71d7b51c, but should have SHA256 hash f77a59198529a9b220fcc8fc9dd91c0bfbf76519240692527ebc1ca60f1817ee.                                               
/boot/pxeboot has SHA256 hash fca9a3768f4a01a9ed32903a033535e372315f6c3fc6b35d7e91878e1417c8fc, but should have SHA256 hash d285b27404275e8f51f0ae3b6e48254f37d3e8ad0edc0742c31a45aba555a2f2.                          
/boot/userboot.so has SHA256 hash 3f8cf5bbf0cabdc595430dfe63d9d9c3bdd2010c4f1c738fd480e1ab56737d3c, but should have SHA256 hash d45372b22bffc1d6d9041819e921fe47e49c7613e4d6108fb6f6f1137a4bfc02.                   
/boot/zfsloader has SHA256 hash 4c1c81e1d991b93f1bac284b194eb023066affe311b85dfcbd380ae1d0242328, but should have SHA256 hash 6495423af0efc8319bf895f6eb27047dcb79cb4f56409deb96cebe72f99d6326.                

it goes on for quite a bit.


/lib/libc.so.7 has SHA256 hash 3f2611f65ddc841a647e5f1477e5aa1f12cf7821c851184fd25697ee2d44a5a5, but should have SHA256 hash c21274868ab320c4759aaaf4f10e6fb9e7b5aef242adfd64244d797d4eed6865. 


I'm guessing I should be able to query what package each of these files is in and reinstall them.


----------



## SirDice (Apr 25, 2018)

Michael Bushey said:


> I'm guessing I should be able to query what package each of these files is in and reinstall them.


These are part of the base OS, there are no packages. 

Before you updated this machine to 11.1 can you remember what version it was before?


----------



## Michael Bushey (Apr 25, 2018)

Hi SirDice,

Most likely 11.0, I usually try to keep it up to date.


----------



## SirDice (Apr 26, 2018)

I think what happened here is that you did the upgrade but only ran `freebsd-update install` _once_. So only the kernel got updated. The rest of the system is updated by running `freebsd-update install` a _second_ time. The command actually needs to be run a total of _three_ times. 

It's a bit of a long shot but you can try to force the upgrade again with `freebsd-update --currently-running 11.0-RELEASE -r 11.1-RELEASE upgrade`. Then run `freebsd-update install` _three_ times. Don't bother to reboot when it tells you to, you're already running the new kernel so you can skip the reboot.


----------

