# everything is "invalid file format" after upgrade from 11.4 to 12.0-RELEASE



## noodlefling (Jul 1, 2021)

I was running 11.4-RELEASE-p12 and ran

`freebsd-update -r 12.0-RELEASE upgrade`

It seemed to run normally, I did the merges, then rebooted.

When it came back up I did this...

`# freebsd-update install
Installing updates...Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
etc...`

That went on for a bit, it complained a lot about missing directories, then finished.  Now I can't do much.

`# uname -a
FreeBSD [REDACTED] 12.0-RELEASE-p13 FreeBSD 12.0-RELEASE-p13 GENERIC  amd64
# freebsd-version -uk
ld-elf.so.1: /lib/libedit.so.7: invalid file format
# freebsd-update fetch
ld-elf.so.1: /lib/libedit.so.7: invalid file format
# pkg info
ld-elf.so.1: /lib/libm.so.5: invalid file format`

Feeling a bit nervous.  Suggestions?


----------



## Tieks (Jul 1, 2021)

Quite a few problems with lib files. I would reboot in single user mode and `fsck /` to make sure the file system is okay. See if fsck gives errors. See what you have then, replace mangled files or consider the rollback option of freebsd-update.


----------



## noodlefling (Jul 1, 2021)

I'd use the rollback option right now if freebsd-update were working. It will take a while to get on-site, but I guess I'm going in!

`# freebsd-update rollback
ld-elf.so.1: /lib/libedit.so.7: invalid file format`


----------



## SirDice (Jul 1, 2021)

You know you can upgrade straight to 12.2? There is no need to install any intermediate versions.

If you've only done a single `freebsd-update install`, then rebooted, only the kernel was updated. I suspect something went wrong with the new kernel. Your old kernel should be in /boot/kernel.old. Not sure when the kernel selection was added but you might be able to select it in the 'beastie' menu. If not, drop to the loader prompt, `unload` and `load /boot/kernel.old/kernel`. If you're booting from ZFS you also need to load ZFS; `load /boot/kernel.old/opensolaris.ko` and `load /boot/kernel.old/zfs.ko`. Then you can `boot` (normal mode) or `boot -s`  (single user mode).


----------



## noodlefling (Jul 1, 2021)

Wasn't sure about which to update to, but up until a few hours ago, I've always been pretty comfortable with version upgrades.  Thanks for the 12.2 tip, though.

That said, I did the first `freebsd-update install`, rebooted, and things seemed fine.  Then I did the second `freebsd-update install` and that's when things got bad.

If I try to boot to single user with the new kernel, it doesn't accept any keyboard input.  If I use a different kernel (from the kernel options on the beastie menu), it recognizes the keyboard, but gives me similar "invalid file format" errors.

I have UFS, not ZFS.  I will try the `unload` thing, but am not optimistic.

I have recent backups, but not sure how to get them from tape if I can't even get into single user.  Wasn't looking for Remedial Update School today, but I guess we can't pick our pitfalls!


----------



## noodlefling (Jul 2, 2021)

Yeah, the `unload` thing had the same result as trying to boot from the kernel in the menu.  I guess we're down to backups.  Yikes!  I suppose I'll need to burn an 11.4 disc?  Haven't done this in ages...


----------



## noodlefling (Jul 2, 2021)

Hmm.  It doesn't want to seem to boot from the DVD drive. Getting a little panicky.

Are there any other tricks you can do from the loader prompt?  Such as replacing all those libraries that don't seem happy?  If the backup kernel is available, are all those .so files hanging around somewhere too?


----------



## richardtoohey2 (Jul 2, 2021)

Don't panic - it will only make things worse (easy to say when not my server/machine).  But try and slow down and take a few deep breaths.

Can you boot off USB key?

Once you get something you can boot off, you'll be able to have enough of an environment to backup (if not already done) and copy files across (like the .so files)/try and rebuild it.

But if you can boot it's also worth checking (as others have advised) that you don't have hardware issues like a failing drive.


----------



## noodlefling (Jul 2, 2021)

Ha! Somehow the first ISO didn't burn correctly.  So, there is hope.  It's just that the system has many redundant chunks of hardware and it takes it forever to reboot, while I sit there nervously hoping it isn't frozen.

Will try the USB key next.  It's a server grade system and kinda pigheaded, so I don't know if that will work.  Second DVD worked in a second system, did not work on first try with the broken system, but will give it another go.

Assuming at some point I get a shell, I'm guessing I should try put 11.4 back on there (so everything else has a greater chance of working).  I partitioned things the old way, so I'm guessing replacing / might do it, if not / and /usr, and maybe /var.

But I'm getting ahead of myself.  At least I have the tape, but I need to get to a place to access it.  All I want to do is put / where it was a few hours ago so at least I can talk to it!


----------



## richardtoohey2 (Jul 2, 2021)

If server-grade you might also have remove console/BMC/iDRAC with Virtual Media boot option?  Any logs, event viewers etc. that might show hardware issues?

Get a bit of a plan together - what do you want to achieve?  Have you got another machine you can experiment with - do the steps on the experimental machine first.

Is your goal to try and rollback to 11.4?  If yes, how are you going to do that?  Again, that's where an experimental machine comes into play.  Build it with 11.4, do the first upgrade step to 12.0, then see if you can get it back to 11.4 using a boot disk/Live Shell.

Or if you want to push onto 12.x, then have you got what you need for that, and how is it going work?  Again, a test machine to experiment on is a good idea.

If you half-installed 12.0 then stay with that - don't jump to 12.2 or anything - just be a bigger mess.

But first step is to find something that will boot, then you can see if you can mount the drive(s), copy anything off, come up with a repair plan.

Good luck.


----------



## noodlefling (Jul 2, 2021)

I just want a functional system again.  If I have that, I have access to tape and can reconfigure anything because I'll have access to /etc and /usr/local/etc/ files.

I've been using a secondary machine to test boot media, but it's too dissimilar (already on 12.2) to do much else with.

Plan is to get installation media working, install 11.4 and then try to get back to where we were before.  User data should be safe.  The hardware appears to be OK.


----------



## noodlefling (Jul 2, 2021)

Whew, booting from the USB stick worked.  I'm not talking to a rock any more.

Not sure I could create a test machine with the appropriate environment if I tried, because I'd have to make it broken and I'm not sure why it broke in the first place.

I'm thinking the re-install is the best bet, given that I have no idea which files are good and which are bad.  And I never got to the step where I update all the packages, so they are all primed for 11.4.


----------



## richardtoohey2 (Jul 2, 2021)

noodlefling said:


> Not sure I could create a test machine with the appropriate environment if I tried, because I'd have to make it broken and I'm not sure why it broke in the first place.


I meant an environment as close as possible - i.e. 11.4 upgraded to first 12.0 boot.  See how to get that back to 11.4.  At least if you botch that test machine up, you can try again.  Botch it up on the server, and you'll just have made your life harder.


----------



## noodlefling (Jul 2, 2021)

Hmm.  Yeah, so, it's scary again.

I booted with USB to 11.4.  I started the install and could see all my partitions, but I'm not totally clear which is which.  Which one is /usr, /var, and such.  I can probably work it out, but that leads me to my second point which is that I ONLY want to blast / and maybe /usr.

Not sure I can get away with that with the install.  I don't really want to blast user data if it's not necessary, even if there is a tape backup.

Any suggestions for simplicity?  If I could totally wipe / and /usr, I'd be happy to pick up the pieces later.  I've always hated FreeBSD's install shell, but since I have to do it so infrequently, it never bothered me much.


----------



## noodlefling (Jul 2, 2021)

Maybe the ideal scenario is to install / in a new place, although all the space is taken.  The user partition is large, but I'd have to resize it.

Looks like the only way to do all this is with the manual partitioning and that's pretty harsh to do under pressure while panicked.  Easy to break things in there.


----------



## richardtoohey2 (Jul 2, 2021)

I thought you would boot to a shell and mount the old drives and check their heatlh - see if you can copy anything off?

If you can see them then maybe you can copy the OS from the boot media to the partitions (if they are healthy).

But yes, I wouldn't try any of this on a live server - try on a test machine.

I wouldn't run the installer - because of what you are seeing - but if you can find the /usr partition on the drive and mount r/w then you can "untar" the usr install set.


----------



## richardtoohey2 (Jul 2, 2021)

e.g. to restore /usr









						Solved - Deleted /usr folder
					

Hi everybody! I'm new in FreeBSD world (always used Linux) and I'm really enjoying it :) I was playing with some configurations when I deleted part of /usr folder. Now I don't have anymore commands like tar, xz, cut, uname..... How can I restore them? I tried to look in the packages but I wasn't...




					forums.freebsd.org
				




Caution - there be dragons here (have I mentioned trying on a test system first?)

Anyway, that's my 2c, good luck.


----------



## noodlefling (Jul 2, 2021)

OK, I'm in the shell.  Having a brain fart because of the panic, but trying to keep a lock on it.

I guess `mount -a` doesn't work because the /etc/fstab is not on the stick.

It's not so much about rescuing the data on the drives, I have the tape.  I just want a functional server again so it can do the userland stuff in the morning.

If the necessary files exist on the stick to make the system bootable again (without every command responding with "invalid file format", I'm happy to nurse it back to health in some boring brute force manner.  Stuck on a lump right now.  Doing some remedial reading that I haven't needed for so long, it's making me feel stupid.


----------



## richardtoohey2 (Jul 2, 2021)

It's a horrible feeling - trying to read/understand when your heart has sunk through your stomach (filled with butterflies) and into your boots.

Again - deep breath - what's your goal?  If you know the backups are good (really?  Are you 100% sure?!) then just go straight to scrubbing the disks and do a fresh install - maybe of 11.4 to start with.  If that goes well, try the upgrade to 12.2, then restore your data on there.  If the upgrade fails, re-install 11.4 and restore your files there.  If the upgrade fails then you've got an issue with this server and 12.2 but a few months until 11.x goes EOL, so time to plan.  If the upgrade works then you've done the job.  But understandbly might just want to get back to solid 11.4 ground.

Every time I've thought my backups are good and I've done a re-install I've missed SOMETHING - some config, special port/package, cron job etc. so ... _maybe_ don't trust yourself on the backups bit yet?

I've manually mounted drives from boot media before but I do it so infrequently I can't give you a useful step-by-step.  I'm not being obtuse or withholding helpful instructions - I just know they are there and work - but don't have them "front of mind".

If UFS and you didn't do any exotic disk set-up, then it won't be a lot to do.  Just keep breathing and reading.


----------



## richardtoohey2 (Jul 2, 2021)

Here's gpt show from a took-the-defaults MBR/BIOS install:
	
	



```
% gpart show
=>        40  7812939696  da0  GPT  (3.6T)
          40        1024    1  freebsd-boot  (512K)
        1064  7803501568    2  freebsd-ufs  (3.6T)
  7803502632     8388608    3  freebsd-swap  (4.0G)
  7811891240     1048496       - free -  (512M)
```
Do you know the device name e.g. da0 or ada0 or mfi0 or mrsas0 etc?


----------



## noodlefling (Jul 2, 2021)

Had a possibly good idea in using the manual installation and only telling it about / and /usr as partitions.  Hopefully it will not blast the others for fun.  If that works, it should have a new / and /usr and everything else should be unmolested.  Then I can recover /etc from tape and hopefully be close-ish to home.

Hope this works!

(Sorry to leave you hanging, it takes forever to reboot and I've chewed through my last metaphorical fingernail!)


----------



## mark_j (Jul 2, 2021)

When you mount the boot disk of the corrupt system, say to /mnt, what does the output of `file /mnt/lib/libmd.so.6` say?


----------



## noodlefling (Jul 2, 2021)

Going down the rabbit hole here...

Now it says something to the effect of

`gptboot no boot loader`

This is not verbatim, but it's rebooting and I'm obviously not on that machine.

Am rebooting to USB stick again now (it takes a while!), so I will try the command above, although I'm not so sure it will be as useful as it might have been a few minutes ago.

Oh, and I guess I'm embarrassed to say that I don't know how to mount the corrupt system to /mnt.  `mount -a` did no magic and I hit a learning wall, so I started down a different path.


----------



## noodlefling (Jul 2, 2021)

OK, I ignored the gptboot thing, remembered that I always had a fight with the installer when getting partitions the way I wanted them, and now I'm up and running with a fresh / and a fresh /usr.

I added the user filesystem to `/etc/fstab` and it was right there, with all the data safe and sound.

So now I just need to pull the config data off the tape and hope the other filesystems survived as well, although since they are all backed up too, hopefully that's not a big deal.

Soon I'll be out of the server room and back in a comfortable office and won't be climbing the walls.

Thanks to everyone who helped out!  Just having a human to talk to and bounce ideas off is very helpful, although I hope never to have to abuse this forum in quite this way again.

On the bright side, if I get things happy again, I will take SirDice's advice and go straight to 12.2, which will save me some time.


----------



## mark_j (Jul 2, 2021)

noodlefling said:


> Going down the rabbit hole here...
> 
> Now it says something to the effect of
> 
> ...



You first need to know the disk device(s) of the corrupt system. They will show up with `gpart show`. Their exact nomenclature will depend on whether they're old bsdlabel, GPT etc.

Then once found, let's assume the boot disk is *da0s1a*, you `mount /dev/da0s1a /mnt`


----------



## richardtoohey2 (Jul 2, 2021)

Sounds like you are making good progress.  And have good backups!

It's worth (on a "quiet" day) having a play with the boot media and learning how to mount the partitions etc. in case you ever have this sort of situation again - it's difficult to keep calm when your head is on fire!

Hope your next update attempt goes smoothly - I've always found freebsd-update to be trouble-free (as with most things on FreeBSD).  But _whisper_ I do always try a few things on test boxes first - never quite 100% the same as production but a good indicator.


----------



## noodlefling (Jul 2, 2021)

I have a really dumb backup system, and by that I mean it doesn't try to do anything cute.  I still `dump` for just these stupid reasons.  It's a blunt instrument and it's easy to understand, and when it breaks, it tends to break in a way that makes sense.

And despite first impressions from this fustercluck, I do test things frequently.  I've done this exact upgrade on multiple machines before without a problem, but none of them are the flagship server, so it's never quite apples-to-apples.  You do what you can with the resources you have.

I'm a little bummed that I still don't know what went wrong in the first place, but I am delighted that my server seems happy and mostly normal.  I'm sure I'm missing a couple of config things that will become clear over time, but it's basically the same guy I knew yesterday.

I didn't even lose user or /usr/local data.  Very, very lucky!

Also this is a good advertisement for the old school method of partitioning drives that has fallen out of fashion.  Having / and /usr on their own partitions was a lifesaver!

Oh, and it turns out I wasn't totally stupid in another way.  The FreeBSD installer allows you to label the partitions, and I gave them sensible names a million years ago so I didn't accidentally blast /var instead of /usr or something like that.  The installer was nice enough to show me those labels, so it was total warm fuzzies when I saw logical things in the right places.


----------



## richardtoohey2 (Jul 2, 2021)

Sounds like a good save! Hopefully upgrade goes smoothly when you try again but as you say then it will be a bit of a head-scratcher as to what happened? Solar flares? Aliens?


----------



## kjpetrie (Jul 2, 2021)

I'm not bothering with 12.x as 13.0 is already at p3, so it'll save a lot of work in 2024 to go straight to 13 now.

Don't forget to read the Release Notes for both 12 and 13 first though, to avoid any unpleasant surprises.


----------



## noodlefling (Jul 2, 2021)

The only reason I went to 12 instead of 13 is that I'd moved so many of the test machines to 12 first, and they've been stable for a while.  I suppose trying to be conservative doesn't always work out, although whatever went wrong probably had nothing to do specifically with 12.  Probably.

I'll report in when I do the 12.2 update.  Might be a little while.  I had a little window when I had some time to do this and thought it wouldn't take more than an afternoon.  At least I've freshened up my disaster recovery skills.

The system seems to be doing fine, so I don't think it was a hardware issue related to the drives, but you never know.

Thanks again for everyone's help.  Blood pressure normal now.


----------



## RodrigoC (Jul 14, 2021)

Hi there,


I've just came to a similar issue but with root on ZFS system:

`root@logpmbackup01p:/usr/home/rod # freebsd-version
11.4-RELEASE-p12`

Then:
`root@logpmbackup01p:/usr/home/rod # freebsd-update upgrade -r 12.0-RELEASE
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 11.4-RELEASE from update1.freebsd.org... done.
............ la la la ......
To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".`

To followed by:
`root@logpmbackup01p:/usr/home/rod # freebsd-update install
src component not installed, skipped
Installing updates...
Kernel updates have been installed. Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.
root@logpmbackup01p:/usr/home/rod # reboot`

An then, the horror:
`root@logpmbackup01p:/usr/home/rod # freebsd-update install
src component not installed, skipped
Installing updates...Bus error (core dumped)
Bus error (core dumped)
............. again and again ........
ld-elf.so.1: /lib/libmd.so.6: invalid file format
Bus error (core dumped)
.............. many times ...........
chflags: ///lib/librss.so.1: No such file or directory
chflags: ///lib/libzfs.so.3: No such file or directory
chflags: ///usr/lib/libarchive.so.7: No such file or directory
.............. a bunch of those .....................
chflags: ///var/db/etcupdate/current/etc/syslog.d/ppp.conf: No such file or directory
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libmd.so.6: invalid file format
ld-elf.so.1: /lib/libedit.so.7: invalid file format
.............
done.`


It doesn't reboot any more on kernel or kernel.old

There are more people with the same issue:









						No way to update FBSD 11 to 12 - Bus error (core dumped) and ld-elf.so.1 - invalid format
					

I had a fresh installed FreeBSD 11 and wanted to update to 12 version i did: pkg update pkg upgrade freebsd-update fetch install freebsd-update -r 12.0-RELEASE upgrade /usr/sbin/freebsd-update install reboot /usr/sbin/freebsd-update install And then: Bus error (core dumped) ld-elf.so.1...




					forums.freebsd.org
				




I don't have hands on the server, but I'll try to get someone to boot it from a 11.4 USB to see whats there.

Any suggestions anyone?


----------



## Emrion (Jul 14, 2021)

I confirm the problem from 11.4-RELEASE to 12.0-RELEASE. I tried in a VM and then, the system booted in single user mode with an error concerning ld-elf.so.1. For me, this occurs after the second `freebsd-update install` at reboot but no Bus error.

I can select /rescue/sh as shell. But there, I have no access to a functional `freebsd-update`.
Hope you have a boot environment from which you can start your system.

That being said, there is no problem to upgrade from 11.4-RELEASE to 12.2-RELEASE.

I think that the point is that 12.0-RELEASE is anterior to 11.4-RELEASE. December 11, 2018 vs June 23, 2020. See here.

In fact, you tried to upgrade to a previous version of FreeBSD.


----------



## noodlefling (Jul 17, 2021)

Emrion said:


> In fact, you tried to upgrade to a previous version of FreeBSD.


I think you hit the nail on the head here.  I did finally give it a second try, going straight to 12.2, and it worked as expected.

Important safety tip!


----------



## igoro (Oct 3, 2021)

I've faced the same issue recently with the same set of actions:
- `freebsd-update -r 12.0 upgrade`
- `freebsd-update install`
- `shutdown -r now`
- `freebsd-update install` # here I've got the same log of core dumps...

I would like to post the way I fixed it in case it might be useful for someone. My target was to get back to 11.4 I had before in order to do upgrade right to 12.2 after this 11.4>12.0 failed manoeuvre. And *this is not about remote fix*, definitely.

- I booted the host from 11.4 USB memstick
- `mount /dev/<root-fs-in-danger> /mnt` # you will have to mount other FSs if it's split onto /var, /usr etc in order to get complete "system tree"
- `cd /mnt/var/db`
- `cp -rp freebsd-update freebsd-update.bak` # let's do some backup of freebsd-update state
- `cd freebsd-update`
- `ln -s install.<?> "$(echo /mnt/ | sha256 -q)-rollback"` # find out the name of your install.<?> dir
- `freebsd-update -b /mnt/ -d /mnt/var/db/freebsd-update/ -f /mnt/etc/freebsd-update.conf rollback`


----------



## Deniska80 (Oct 29, 2021)

floodway said:


> I would like to post the way I fixed it in case it might be useful for someone


Oh, thank's a lot, you saved many hours of my life! Good knoledge of how freebsd-backup internaly works. Now i decied read throught this tool too ^)


----------



## grahamperrin@ (Nov 1, 2021)

RodrigoC said:


> `root@logpmbackup01p:/usr/home/rod # freebsd-update upgrade -r 12.0-RELEASE`





floodway said:


> `freebsd-update -r 12.0 upgrade`



RodrigoC and floodway please, was there the false assumption that a major upgrade must always be to a `.0` version?

(<https://docs.freebsd.org/doc/12.2-R...ding-freebsdupdate.html#freebsdupdate-upgrade> for 12.2-RELEASE lacked clarity.)


----------



## igoro (Dec 6, 2021)

grahamperrin said:


> RodrigoC and floodway please, was there the false assumption that a major upgrade must always be to a `.0` version?
> 
> (<https://docs.freebsd.org/doc/12.2-R...ding-freebsdupdate.html#freebsdupdate-upgrade> for 12.2-RELEASE lacked clarity.)


I personally do not remember any official statement regarding this. I guess, it's not a concern if an instance tries to follow new releases constantly, and if it's outdated for longer period than it needs more caution (yeah, undetermined "caution").

The best answer, I think, could be an official statement from a FreeBSD dev, who better understands this situation from dev cycle perspective and upgrade tech process details. And added to the official documentation you referred. For example, it could be a recommendation like "always do major upgrades to <X+1>.<Latest>", if it really is a correct statement.

If it's hard to reach a dev, I guess, it could be sent as a bug request to patch the doc respectively - it may trigger required discussion, and real change eventually.


----------



## SirDice (Dec 6, 2021)

freebsd-update(8) can upgrade from any[*] -RELEASE version to any other -RELEASE version. You do NOT need install any of the intermediate versions. 

[*] Not entirely correct, there's is a 'lowest' version, before 6.2 freebsd-update(8) didn't exist. But I wouldn't recommend upgrading from 6.2-RELEASE to 13.0-RELEASE any way, it's just too big of a step up. It's perfectly fine however to upgrade from 10.3 to 13.0 or 11.2 to 13.0 skipping over all the 12.x versions. 

There has been one exception, at some point in time freebsd-update(8) had a bug that prevented it from upgrading to 11.x (or above). To fix that bug you need to have at least the latest _patch_ version of a -RELEASE.


----------



## astyle (Dec 6, 2021)

Ahhh... the horror stories about freebsd-update(8) that I constantly see on these forums are enough to scare me away. I personally concluded that this kind of thing, it's OK to practice them in a throwaway VM if you have the time and the hardware to do that. I'd want to study my ass off about freebsd-update(8), and make sure the versions match, and all other pitfalls are avoided - BEFORE diving off a cliff like that.


----------



## SirDice (Dec 6, 2021)

astyle said:


> the horror stories about freebsd-update(8) that I constantly see on these forums are enough to scare me away.


Survivorship bias. People that don't have problems don't post about it.


----------



## astyle (Dec 6, 2021)

SirDice said:


> Survivorship bias. People that don't have problems don't post about it.


Yeah, but the flip side of the same idea: There's not that many blogs that crow about how nice and easy freebsd-update(8) is, and offer their own instructions, reviews, and warnings of pitfalls.


----------



## grahamperrin@ (Dec 9, 2021)

SirDice said:


> can upgrade from any[*] -RELEASE version to any other -RELEASE version.



... can *break* FreeBSD if, as reported in this topic, the upgrade is from 11.4 to 12.0.


Beware of hidden context with XenForo, especially on mobile phones.


----------



## SirDice (Dec 9, 2021)

grahamperrin said:


> can *break* FreeBSD if, as reported in this topic, the upgrade is from 11.4 to 12.0.


That's not an issue caused by freebsd-update(8) and you're taking my remark out of context.


----------



## grahamperrin@ (Dec 25, 2021)

Cross-reference: Can I upgrade an old computer that isn't used that much from 12.1 to 12.3 directly or is skipping point releases not allowed? : freebsd



astyle said:


> … horror stories about freebsd-update(8) …



The situation is less horrible than might be imagined. 

Some stories relate to problems before or after use of the tool: 

not a problem with the tool. 
Some almost certainly relate to misuse of the tool: 

not using the tool as instructed in the FreeBSD Handbook.



SirDice said:


> … People that don't have problems don't post about it.



True, and (generally) people that _do_ have problems *very* often do _not_ disclose what's required for readers to appreciate the true nature of the problem. 

Particularly with freebsd-update, I always try to read between the lines (including prior posts/topics by the reporter) to guess whether there was misuse.

I recently went through some open bugs in Bugzilla, resulting in a closure or two. 

Most memorable (already closed) were the bugs that led to erratum *FreeBSD-EN-21:08.freebsd-update*. Other mentions of freebsd-update can be found at pages such as <https://bokut.in/freebsd-patch-level-table/> …


----------



## VENTDEV (Feb 15, 2022)

Sorry for the Necropost, but Wisdom of the Ancients. This is for anyone that runs across this situation now and in the future with any builds. 

I had the same situation as the OP. I bounced from 11.4 to 12.0 on my way to 12.3, I should have known better, but I digress. I managed to fix it without reinstalling on a remote 1000+ connection server, 1000 miles away, in a Linux shop datacenter. (So even if I paid $200/hr to fix it, they wouldn't be able to via USB.)

When you hosed the system and caused the core dumps and ld-elf.so.1: /lib/libmd.so.6: invalid file format errors, you should have at least 1 shell logged in as root. And your services should be running. You can't launch any programs outside of /rescue/. Whatever you do, don't disconnect or reboot the system. 

So what happened? 11.4 is newer, so the /lib/ files are newer than what 12.0 tried to install. So the 12 /libs/ didn't install. I suspect this will happen anytime you jump from a newer old build to an older new build. You need to install 12.0 /lib/ files, which means you need to grab 12's base.txz. 

Sadly, I don't see anything in /rescue/ that can do that. (It would be wonderful if they included wget or some sort of recovery program in there.) 

This is where your services come into play. Luckily for me, I still had access to PHP, with a 15Mb upload limit. Yay! I download Base.txz on my local system and extracted it. The first job is to get sshd working in case of network failure.  

Run sshd -t. It should spit out some broken libs. Find them in your local extraction of base and upload them to your PHP uploader (you either wrote one using /rescue/vi or you got something like forum software installed. Alternatively, if you have mail service still going, an email could work.). Move those libs to /lib/. I think I needed libssl.so.111, and I also brought in /lib/libedit.so since everything was complaining about it. 

Run sshd, connect to it and log in. If you can, great! You're ready for the headlong charge.

You can either try to get wget running or upload /lib/ piecemeal. I chose the latter in 2 tar files. Make sure you have those files all in one folder, say 12BaseLib, as you're going to get only one chance at this. Using /rescue/cp -R copy everything in your uploaded 12BaseLib folder to the actual /lib/ folder. You're going to be kicked off shell as EVERYTHING will crash on the system because you're copying over libc and what have you. /rescue/cp should still do its job, though. At this point, you wait 10minutes. Then you send a tech support ticket to reboot the system. Wait another 10 minutes, and hopefully, you can log in.

If you can, you're almost home. 

Now you need to Freebsd-update to 13, as that will copy over the most files. You'll run through that entire system, a few more things will fail, like libregex, but the install should go through. Sshd and a few other programs will break again. You will have to manually merge files because diff is broken. When you finish that, it will install, but there will be a few errors. At this point, a few programs will work, notably wget. Download the 13.0 base.txz, extract it to 13Base or something. You'll need to copy over the files you need for sshd to make sure it works again. Then using /rescue/cp move over the stuff in 13Base/usr/lib to /usr/lib and you'll also need to do the /share/ folder. This action shouldn't crash, but it might. At this point, you should be able to run pkg-static upgrade -f, and nothing should break. Finish up the install, verify that sshd works again, reboot. Clean up any config issues you have now with the update. (PHP 8, ugh)

That's what worked for me. All my services are running, the system reboots, no errors in my log files. 

Keep 13Base.txz file around if you run into any missing or malformed files. 

In the future, always bounce to the latest minor builds of Release. And download base.txz files to the system if you have the space. They're pretty handy if you screw something up.


----------

