# fish shell broken and unable to login with ssh



## joggx (Jul 6, 2022)

I can't log into FreeBSD with SSH.


```
ld-elf.so.1: /usr/local/bin/fish: Undefined symbol "eventfd@FBSD_1.6"
```

Every command I try to execute through (possible with an SSH lib) also ends up with above error.


----------



## Geezer (Jul 6, 2022)

Where is the machine? Do you have physical access to it? If it is remote, can you get a KVM? Is it a root account?

You really need to give more information.


----------



## fraxamo (Jul 6, 2022)

joggx said:


> I can't log into FreeBSD with SSH.
> 
> 
> ```
> ...


Looks like you've been here before. How did you fix it that time?
Have you just done an upgrade of the OS? If so, it looks like you might need to finish the upgrade. What does  `freebsd-version -uk` show?


----------



## joggx (Jul 6, 2022)

It wasn't fixed as the VM and jails continued to work I never touched the host OS. Now one of the bhyve vm's is dead I need to get hands on it again. No command can be executed remotely so I can't find out except going to the machine. SSH access of root seems disabled:


```
End of keyboard-interactive prompts from server
Access denied
```


----------



## joggx (Jul 6, 2022)

Geezer said:


> Where is the machine? Do you have physical access to it? If it is remote, can you get a KVM? Is it a root account?
> 
> You really need to give more information.


It is in a data center. I can have virtual console through DELL idrac.


----------



## Erichans (Jul 6, 2022)

If you have a toor account that's accessible (and doesn't have the fish shell as its default) you could try that.


----------



## SirDice (Jul 6, 2022)

joggx said:


> SSH access of root seems disabled


Yes, it's disabled by default. 



joggx said:


> I can have virtual console through DELL idrac.


That's the only way to fix this. Use the remote console to login as root, then fix the issue (looks like you're running 12.3 packages on an outdated 12.0 or 12.1 system).


----------



## astyle (Jul 6, 2022)

When a lib cannot load even symbols for a file descriptor, that indicates a corruption somewhere. Can that VM be rebooted? A reboot forces the kernel to re-check the hardware addresses where libs are loaded. A reboot is also a chance to run `fsck`. If the system can be rebooted, it can be saved.


----------



## SirDice (Jul 6, 2022)

fraxamo spotted the same question from a while back:


fraxamo said:


> Looks like you've been here before.





joggx said:


> It wasn't fixed as the VM and jails continued to work I never touched the host OS



So, you just left it and now it's biting you in the ass.

Anyway, how to fix this mess? Get remote access through DRAC/IPMI/Remote KVM/whatever. If that's not possible you're going to have to walk/drive/fly to wherever the machine is and login locally. There's no way to solve this _without_ access to the console. 

Once there, login as root (I hope you didn't change root's shell), update your sources in /usr/src/. I suggest going for releng/13.1 (that's 13.1-RELEASE). Build world, build kernel, install kernel, install world, run etcupdate(8), etc. to correctly update the host. The host will require at least one reboot. I suggest not rebooting after the installation of the kernel, just reboot once after the whole system has been updated.


----------



## joggx (Jul 6, 2022)

astyle said:


> When a lib cannot load even symbols for a file descriptor, that indicates a corruption somewhere. Can that VM be rebooted? A reboot forces the kernel to re-check the hardware addresses where libs are loaded. A reboot is also a chance to run `fsck`. If the system can be rebooted, it can be saved.


yes it can be rebooted.


----------



## astyle (Jul 6, 2022)

joggx said:


> yes it can be rebooted.


Good. If you can't update it like SirDice suggests in post #9, you're looking at a complete re-install.


----------



## VladiBG (Jul 6, 2022)

`/rescue/sh`


----------



## joggx (Jul 7, 2022)

SirDice said:


> fraxamo spotted the same question from a while back:
> 
> 
> 
> ...


The root has fish as well. So re-install?


----------



## SirDice (Jul 7, 2022)

joggx said:


> The root has fish as well. So re-install?


No, not needed. But you're going to have to boot to single user mode to change root's shell back to csh(1) or sh(1) before you can do anything.


----------



## joggx (Jul 7, 2022)

I have changed the shell and can log in now.  But when running git I get the same link symbol error:


```
/usr # git clone -b releng/13.1 --depth 1 https://git.freebsd.org/src.git src
Cloning into 'src'...
ld-elf.so.1: /usr/local/bin/git: Undefined symbol "regcomp@FBSD_1.6"
```


----------



## fraxamo (Jul 7, 2022)

Can you post the output of `freebsd-version -ruk`?


----------



## joggx (Jul 7, 2022)

fraxamo said:


> [Can you post the output of `freebsd-version -ruk`?



Every command failed with `Undefined symbol "regcomp@FBSD_1.6"`. The following is uname output:

```
$ uname -a
FreeBSD xxx 13.0-CURRENT FreeBSD 13.0-CURRENT #3 r363383: Thu Jul 23 01:09:38 PDT 2020     root@xxx:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
```


----------



## fraxamo (Jul 7, 2022)

You're running '-CURRENT'. Please see point 2 at this link. That link will also explain where to get support for the version that you're running.


----------



## T-Daemon (Jul 7, 2022)

joggx, You could try the solution from How to resolve ld-elf.so.1: /bin/sh: Undefined symbol "stat@FBSD_1.5".

You can use netcat, nc(1) ( /rescue/nc ) to receive the file and from a 13.0-RELEASE or 13.1-RELEASE system netcat send the file.

For example:

On the affected system boot into single user mode, mount the file system read-write, `cd /tmp`.

On the affected system, the receiving side, make nc(1) listen on port 1234 and wait to write to file:

```
# /rescue/nc -l 1234 > libc.so.7
```
From the sending side:

```
nc -N [IP_address_of_affected_system] 1234 < /lib/libc.so.7
```

Proceed as described in the linked thread:

```
/rescue/chflags noschg /lib/libc.so.7
/rescue/cp libc.so.7 /lib/libc.so.7
```


----------



## joggx (Jul 7, 2022)

T-Daemon

Can I extract libc.so.7 from the ISO and put in somewhere like /root to copy over in single user mode?


----------



## VladiBG (Jul 7, 2022)

can you check if the `/usr/bin/fetch` is working? If it's working you can use it to download the src from the ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.1-RELEASE/src.txz and rebuild. Otherwise you will need to mount the virtual iso using idrac then mount it via /rescue/mount_udf or /rescue/mound_cd9660 and extract the src.txz via /rescue/tar


----------



## joggx (Jul 8, 2022)

VladiBG said:


> can you check if the `/usr/bin/fetch` is working? If it's working you can use it to download the src from the ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.1-RELEASE/src.txz and rebuild. Otherwise you will need to mount the virtual iso using idrac then mount it via /rescue/mount_udf or /rescue/mound_cd9660 and extract the src.txz via /rescue/tar


I extracted the file from ISO with `mdconfig` followed by  `mount` and replace the existing one in single user mode and the error `Undefined symbol "regcomp@FBSD_1.6"` disappeared.


----------



## joggx (Jul 8, 2022)

After `make buildworld` and `make kernel`, I have the error:


```
kldxref /boot/kernel
kldxref: error while reading /boot/kernel/iwlwifi-9000-pu-b0-jf-b0-46.ucode.ko: Bad address
kldxref: error while reading /boot/kernel/iwlwifi-9260-th-b0-jf-b0-46.ucode.ko: Bad address
kldxref: /boot/kernel/kernel: too many segments
```

I ignore the error and go ahead to next step, and get the following output:


```
/usr/src # etcupdate -p
No previous tree to compare against, a sane comparison is not possible.
```

How should I proceed?


----------



## VladiBG (Jul 8, 2022)

Remove the current src tree after /usr/src is empty extract the new src as i describe in my previous post then build the kernel and word and generate new current tree with etcupdate. Then use etcupdate diff.


			Re: etcupdate warning: "No previous tree to compare against, a sane comparison is not possible."
		










						Chapter 25. Updating and Upgrading FreeBSD
					

Information about how to keep a FreeBSD system up-to-date with freebsd-update or Git, how to rebuild and reinstall the entire base system, etc




					docs.freebsd.org


----------



## joggx (Jul 8, 2022)

VladiBG said:


> Remove the current src tree after /usr/src is empty extract the new src as i describe in my previous post then build the kernel and word and generate new current tree with etcupdate. Then use etcupdate diff.
> 
> 
> Re: etcupdate warning: "No previous tree to compare against, a sane comparison is not possible."
> ...


I did `etcupdate extract`. diff gives too much I bypass the patch. The system still boots, but the ssh console shows it is still 13.0-CURRENT, while `uname -a` shows 13.1 release.
`Last login: Fri Jul  8 03:09:57 2022 from X.X.X.X
FreeBSD 13.0-CURRENT (GENERIC) #3 r363383: Thu Jul 23 01:09:38 PDT 2020

> uname -a
FreeBSD ftp0 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-fc952ac22 GENERIC amd64`


----------



## VladiBG (Jul 8, 2022)

You need to merge the etc configs do not ignore the etcupdate otherwise you will end up with non working configuration in /etc. After you generate the diff you can prepare the merge and install kernel then reboot and install the userland.

After that when you have working "freebsd-update" binary you can use it to install on top on that installation 13.1-RELEASE and continue to use binary updates instead of following the CURRENT version via src.


----------

