# Hosed system with ldconfig, how to recover?



## Sephiroth (Aug 14, 2013)

I really did it this time. 

I have been trying to upgrade all my ports, and for whatever reason, libpng would not install- it was failing the tests during the build. I went into the work folder, and tried running `./pngtest` directly, and it was complaining that libz.so.4 was not found. Hmm. I searched the system and found it sitting right there in /lib/lib.

Well, something in the linker path must be screwy, right? So I googled and read some man pages, and started with ldd and moved to ldconfig. Ah-ha! I should just run ldconfig /lib/lib, and add those libraries to the path.... Perfect, now the pngtest works perfect. However, make does not... can't find libc.so.7... uh oh.

I figured I'd hosed something bad (true), but I know that ldconfig is run during boot, so I shut the system down and figured a reboot would bring it back. Boy was I wrong.

So now here I sit, serverless. It can't boot because /bin/sh can't load, it's looking for libc.so.7.

What do I need to do to recover the system? I figure I burn a CD of the system, boot from that, and try and replace some files, but I don't know what to replace or how to go about this. I'm honestly not sure what I did wrong with ldconfig, but I guess it's one of those commands that you need to run with all the paths you need every time.


----------



## Sephiroth (Aug 14, 2013)

I do know what I did wrong with ldconfig, I should have read the man page closer. I missed the -m option, to merge in new entries. Too bad I panicked afterwards. Should have known better, restarts fix problems in Windows, not Unix. Dumb.

Downloading the rescue CD now, what do I need to run to repair the system?


----------



## ta0kira (Aug 14, 2013)

Sephiroth said:
			
		

> Downloading the rescue CD now, what do I need to run to repair the system?


Do you have a statically-linked copy of busybox? (It might even be on the rescue CD you have.) If so,

Start the rescue CD.
Mount your root partition.
Copy busybox to the mounted root partition.
chroot to the mounted root partition, running busybox as the command.
Perform the necessary ldconfig repairs.
This might not be the most FreeBSD-ish solution, but it's a general solution for this sort of problem.

Kevin Barry


----------



## Sephiroth (Aug 15, 2013)

Thanks for the advice. I ended up getting it working by copying the linker hints file from the live CD into the server, which was then able to boot and then fixed itself during boot via the ldconfig rc.d script (`/etc/rc.d/ldconfig start`).

Everything seems to be chugging along again just fine, other than slapd in one of the jails that keeps segfaulting on startup... gah.

So, follow up question, back to the original problem I was trying to solve. One port (graphics/png) won't install, because the tests it builds can't find a library in /lib/lib. How do I tell the system to look in there? I think I need to do this: `ldconfig -m /lib/lib`.

But that hint would go away after a reboot, I think. How does the ldconfig rc script work- seems like there's an environment variable in there somewhere that I need to update.


----------



## SirDice (Aug 15, 2013)

There shouldn't be a /lib/lib. Not as a file or a directory. 

What is the exact error message you're getting?


----------



## kpa (Aug 15, 2013)

I think you could boot from a recovery CD/USB stick and use the /rescue/sh to have a working shell inside the chroot(8).


----------



## ta0kira (Aug 16, 2013)

kpa said:
			
		

> I think you could boot from a recovery CD/USB stick and use the /rescue/sh to have a working shell inside the chroot(8).


I suggested BusyBox because ls, cp, mv, etc. all need the dynamic linker. Surprisingly, so does ldconfig, which I wouldn't have expected. Isn't that a Catch-22?

Kevin Barry


----------



## kpa (Aug 16, 2013)

The /rescue binaries are all statically linked so they don't need a dynamic linker.


----------



## ta0kira (Aug 16, 2013)

kpa said:
			
		

> The /rescue binaries are all statically linked so they don't need a dynamic linker.


But you'd have to copy all of them to your root partition to chroot to it. Just copying sh wouldn't be sufficient.

Kevin Barry


----------

