# Upgrade disaster 11.2-RELEASE -> 12.0-RELEASE



## ahhyes (Dec 12, 2018)

Hi All,

Well, this is a first. I have been blessed with years of FreeBSD upgrades going flawlessly until I went to upgrade a machine last night using the `freebsd-update` utility.

All seemed to go well, first phase (which from memory installs the kernel portion of the upgrade) I did the`freebsd-update install` command which completed without any errors, then told me to reboot and run that same command again, and that's where it all went bad. I rebooted and the machine no longer boots and bootloader seems to crash (I have attached a photo)... Not sure what to do from here.. Any suggestions?


----------



## ahhyes (Dec 12, 2018)

I should mention I do have a ZFS pool on the machine, but the OS is not installed on it. FreeBSD is installed on a regular hard disk formatted with UFS and boots from there.


----------



## johnblue (Dec 12, 2018)

ahhyes said:


> FreeBSD is installed on a regular hard disk formatted with UFS and boots from there.


If you are like me I reach a point of diminising returns when an OS tanks.  Plus, my OCD has a hard time trusting it.  I would get another hard drive and install fresh then mount the old hard drive and cp off what you need.

After prod is back up do your postmortem at your leisure.

Good hunting.


----------



## ahhyes (Dec 12, 2018)

Any ideas what may have gone wrong? I see the boot loader print a list of drive letters briefly (which seems normal), then boom. crash. It never loads the kernel from what I can see. I find it strange the screen is also corrupted, theres flashing ascii characters too in those verticle colored bars (though the camera didnt catch it).


----------



## k.jacker (Dec 12, 2018)

I can't make anything of the output on your screen, but the looks of your screen point towards a severe hardware  problem.

It could be random that it hit you now, but if the computer had long uptime, since last reboot, maybe something build up that didn't show.

I would in the first step, take a pendrive with the FreeBSD installer, to see if the computer will boot. If yes, I'd probable look at `gpart show` to see if it still shows a partition table and maybe rewrite the EFI partition (you booted in EFI mode i guess because it shows in the photo).
If it won't boot from the installer either, see if you can enter the BIOS and load defaults and then check the battery.
If you can't get into the BIOS, check RAM, PSU and so on...


----------



## ahhyes (Dec 12, 2018)

k.jacker said:


> I would in the first step, take a pendrive with the FreeBSD installer, to see if the computer will boot. If yes, I'd probable look at `gpart show` to see if it still shows a partition table and maybe rewrite the EFI partition (you booted in EFI mode i guess because it shows in the photo)



I've never enabled EFI mode to my knowledge, how can you tell its EFI? 

Just strange. This machine spends most of its time powered off, it is primarily a NAS which hasnt exhibited any signs of hardware failure to date. I guess its possible I just suddenly got hit with a hardware failure, but I would have to have been pretty unlucky!


----------



## ShelLuser (Dec 12, 2018)

The screenshot in the first post looks like the kernel crashing, not the bootloader. I assume the menu displays normally?

What happens if you drop down to the ok> prompt (press escape) and then issue the `lsdev` command? Does that list your drives (at least the ZFS pool)?

Also: what happens when you use the menu to boot the previous kernel (kernel.old)?


----------



## ahhyes (Dec 12, 2018)

ShelLuser said:


> The screenshot in the first post looks like the kernel crashing, not the bootloader. I assume the menu displays normally?
> 
> What happens if you drop down to the ok> prompt (press escape) and then issue the `lsdev` command? Does that list your drives (at least the ZFS pool)?
> 
> Also: what happens when you use the menu to boot the previous kernel (kernel.old)?



It feels like bootloader to me (but I could be wrong of course), I only say this because of the "BTX halted" message, and secondly it happens so fast. This system boots from good old spinning rust (which is slow compared to the likes of flash storage), so theres always those few seconds where it loads the kernel (that seems noticably absent in this case). I quickly see the loader print a list of drive letters and its an instant crash.

I will try those commands and see what I find (at work at the moment so cant check right now)


----------



## ahhyes (Dec 13, 2018)

the CS:EIP pair in that register dump is rather alarming - assuming this is for protected mode, it will be codeselector and offset - that looks invalid to me and may be the reason for the crash.


----------



## Wamphyre (Dec 13, 2018)

I suffered the same problem after update. Just disconnect any external drive (USB drives, USB HDDs, etc) and boot again


----------



## ahhyes (Dec 13, 2018)

Wamphyre said:


> I suffered the same problem after update. Just disconnect any external drive (USB drives, USB HDDs, etc) and boot again



No external drives connected to this system. I wonder if the drive ordering has gotten changed somehow.. I wont know until I try the commands that 'shelluser' suggested in their reply. So you had the same type of crash as me?


----------



## Wamphyre (Dec 13, 2018)

Yes, after update from 11.2 to 12-RELEASE I experienced the same "BTX Halted" error. Tried to boot again without any external drive connected, and it worked (with 11.2 wasn't any problem with that)


----------



## drhowarddrfine (Dec 13, 2018)

The only thing we know you did was the last of five or more steps involved with updating. What did you do previous to the install part?


----------



## ahhyes (Dec 13, 2018)

drhowarddrfine said:


> The only thing we know you did was the last of five or more steps involved with updating. What did you do previous to the install part?



I didn't get far into the upgrade before this hit (from memory this was the sequence of events):

`freebsd-update -r RELEASE-12.0 upgrade`

<it downloaded a whole lot of stuff as usual, then I was eventually prompted about merging some config in /etc, nothing eventful there>

`freebsd-update install`

<this part completed with no error and advised me that to continue the upgrade, to reboot and then run `freebsd-update install` again (presumably to install the userland stuff)>
<I rebooted and hit the error as shown in the OP>


----------



## edenist (Dec 13, 2018)

What CPU is in this system if you don't mind me asking?

Over the past few months, I've had all sorts of kernel problems stemming from meltdown/spectre mitigations [the OS implementations along with all sorts of microcode combinations]. 

I believe there are also some NUMA changes in 12.0, and the errata also mentions boot problems with J1900 CPUs. I don't suppose any of these apply to your case?


----------



## ahhyes (Dec 13, 2018)

edenist said:


> What CPU is in this system if you don't mind me asking?
> 
> Over the past few months, I've had all sorts of kernel problems stemming from meltdown/spectre mitigations [the OS implementations along with all sorts of microcode combinations].
> 
> I believe there are also some NUMA changes in 12.0, and the errata also mentions boot problems with J1900 CPUs. I don't suppose any of these apply to your case?



It is a Xeon E5-1650V2 from what I can recall.


----------



## tedbell (Dec 13, 2018)

The upgrade destroyed my machine twice as well as well as a host of other problems.

The first time was an upgrade and with my USB drives connected I got BTX halted. The issue is NOT on my end because I can boot fine with 11.2 with UFS on my boot drive.

The second time was a clean install with the USB image. Installation would not detect my USB network adapter like 11.2 would. Another BTX failure with drives connected. Then it took hours to connect to the internet because 
	
	



```
ifconfig wlan0 create
```
 would not work as it did in 11.2. Then when I somehow managed to connect all the mirrors were messed up and I couldn't even download pkg.

The third time was another upgrade from 11.2 and I couldn't boot period with or without drives connected.

Total disaster. Sticking with 11.2


----------



## Datapanic (Dec 13, 2018)

I prefer to either run fresh installs or upgrades on the FreeBSD RC's on my stuff which I can easily backup and restore first, just in case it doesn't work before doing it with *-RELEASE next versions on my production systems.  Of course, always have snapshots/backups that work before doing it.  

Meanwhile, I'm running Poudriere on a temp server to create new packages for my 12.0 preferences before I can install or upgrade any of my other systems to 12.0.  

Good luck with your upgrades!


----------



## ahhyes (Dec 13, 2018)

A few changes to the bootloader there:
https://www.freebsd.org/releases/12.0R/relnotes.html#boot

Not much that can go wrong with all those changes is there .. I've just written a 12.0-RELEASE memstick image to a usb memory stick and will test this when I am home. I'm curious to see if that even boots. If it doesnt, then it looks like 12.0 has introduced some incompatibility with my hardware. I'm willing to wager a small amount of money that a 11.2-RELEASE image will boot if 12.0-RELEASE fails (gut feeling).


----------



## ahhyes (Dec 13, 2018)

Looking at some of the other forum threads.. It seems I am not alone with 12.0 failing to boot.


----------



## VladiBG (Dec 13, 2018)

Try to boot from 12.0-RELEASE LiveCD. If it's boot normally then try to update your bootstrap code.


----------



## ahhyes (Dec 13, 2018)

ShelLuser said:


> The screenshot in the first post looks like the kernel crashing, not the bootloader. I assume the menu displays normally?
> 
> What happens if you drop down to the ok> prompt (press escape) and then issue the `lsdev` command? Does that list your drives (at least the ZFS pool)?
> 
> Also: what happens when you use the menu to boot the previous kernel (kernel.old)?



`lsdev` doesnt appear to do anything: "No lsdev on 0:ad(0p2)"

Trying to do a `boot kernel.old` from the boot loader is telling me "gptboot: No kernel.old on 0:ad(0p2)"

I tried booting from a 12.0-RELEASE memstick image, on a USB drive, it also crashes with the same error...


----------



## ShelLuser (Dec 13, 2018)

ahhyes said:


> `lsdev` doesnt appear to do anything: "No lsdev on 0:ad(0p2)"
> 
> Trying to do a `boot kernel.old` from the boot loader is telling me "gptboot: No kernel.old on 0:ad(0p2)"


Ok, now _that's_ bizarre. The first error makes me believe that the bootloader might have been damaged or incorrectly installed, yet the second error would also point at your root filesystem.



ahhyes said:


> I tried booting from a 12.0-RELEASE memstick image, on a USB drive, it also crashes with the same error...


At this stage I'd try to boot with a 11.2 image just to check what that says. Because something is seriously bad here.


----------



## ahhyes (Dec 13, 2018)

ShelLuser said:


> At this stage I'd try to boot with a 11.2 image just to check what that says. Because something is seriously bad here.



I am putting my money on that working. lol. Downloading it now. I'll report back on the results.


----------



## VladiBG (Dec 13, 2018)

If 12.0-RELEASE memstick image crash then verify it's checksum and try again. If you are sure that the downloaded image is correct and still cannot boot from it then the only option you have left is to revert back to 11.2. You can also try to update your BIOS.


----------



## ahhyes (Dec 13, 2018)

VladiBG said:


> If 12.0-RELEASE memstick image crash then verify it's checksum and try again. If you are sure that the downloaded image is correct and still cannot boot from it then the only option you have left is to revert back to 11.2. You can also can try to update your bios it may help also.



Its a good image  I checked the sha512sum before I wrote the image to the memory stick


----------



## teo (Dec 13, 2018)

ahhyes said:


> Hi All,
> 
> Well, this is a first. I have been blessed with years of FreeBSD upgrades going flawlessly until I went to upgrade a machine last night using the `freebsd-update` utility.
> 
> ...



 It looks like the system detected Xorg as corrupt and he's not getting past it.


----------



## ahhyes (Dec 13, 2018)

11.2-RELEASE boots perfectly....

Ok. So what on earth is 12.0-RELEASE unhappy about on that hardware? Bizarre.

`freebsd-update` does have a rollback option, will that get me out of trouble here?


----------



## ahhyes (Dec 13, 2018)

teo said:


> It looks like the system detected Xorg as corrupt and he's not getting past it.


Hi - Actually this is a boot issue (see photo on OP) "BTX Halted"


----------



## _martin (Dec 13, 2018)

Hm, Xeon E5-1650V2 is 64b.  Are you using 32b version of FreeBSD ? Segment registers suggest you are already in protected mode (CR0 reg would confirm that). CS:IP seems ok, the contents of the mem where it points to is ivalid (0xff -> bad instruction ,  int 6 ).

Consider using beadm(1) and create new boot environment when doing upgrade. With that you can easily revert/reactivate old env.

UPDATE: ok, I went through your thread again and noticed you mentioned ZFS is not root pool .. never mind my advice on beadm then


----------



## ahhyes (Dec 13, 2018)

_martin said:


> Hm, Xeon E5-1650V2 is 64b.  Are you using 32b version of FreeBSD ? Segment registers suggest you are already in protected mode (CR0 reg would confirm that). CS:IP seems ok, the contents of the mem where it points to is ivalid (0xff -> bad instruction ,  int 6 ).
> 
> Consider using beadm(1) and create new boot environment when doing upgrade. With that you can easily revert/reactivate old env.



As far as I know the system has always been running FreeBSD amd64


----------



## ahhyes (Dec 13, 2018)

hm.. is there any way to remount / from single user mode when booting from a USB drive? it would be nice to remount / and do a rollback?


----------



## Wamphyre (Dec 13, 2018)

I tried (just for testing) a new 12.0-RELEASE installation with external disks attached. Boot fails with "BTX Halted" error.

Removing external drives = It boots well (looks spooky)

I'm still thinking that your problem is related with your disks or BIOS configuration (Try from IDE to AHCI or viceversa)


----------



## Wamphyre (Dec 13, 2018)

Definitely, there's a bug on the BTX loader in FreeBSD 12.0-RELEASE (I suppose). With 11.2 there's no problem with boot process with or without external drives connected


----------



## ahhyes (Dec 13, 2018)

Any ideas how I can revert back to 11.2 at this point? I can boot with the 11.2 usb image but / is already mounted and I can only remount /dev/ada6p2 to another location.


----------



## VladiBG (Dec 13, 2018)

do you have backup?


----------



## ahhyes (Dec 13, 2018)

VladiBG said:


> do you have backup?


Sadly no... This is my NAS machine which I use for backing things up to (amongst other things)


----------



## VladiBG (Dec 13, 2018)

can you boot with the kernel.old?


----------



## ahhyes (Dec 13, 2018)

VladiBG said:


> can you boot with the kernel.old?



`boot kernel.old` doesnt seem to work sadly.. the bootloader is screwed.

if I mount the hard disk after booting from an 11.2-RELEASE memstick image I can see all the files are there under boot  folder. It suggests the hard disk contents itself are fine and the bootloader is screwed up.


----------



## VladiBG (Dec 13, 2018)

from the boot screen press "5: Kernel"  to switch to the old kernel


----------



## ahhyes (Dec 13, 2018)

VladiBG said:


> from the boot screen press "5: Kernel"  to switch to the old kernel


The boot doesnt get that far anymore. Theres no menu, only the early stage BTX loader.


----------



## VladiBG (Dec 13, 2018)

boot with the LiveCD create a directory /tmp/root and mount your freebsd-ufs partition there. Setup your network and make a backup of your files. Then you can reinstall 11.2-RELEASE

EDIT:
After you make a *backup *you can try this (This may be bad advice so try on your own risk)

boot with the 11.2-RELEASE livecd and setup your network
`dhclient hn0` # or use ifconfig to set your ip address/gateway/dns
`setenv UNAME_r 12.0-RELEASE` # this will trick freebsd-update that we are running 12.0-RELEASE
`mkdir /tmp/root` # create a mount point
`mount /dev/ada6p2 /tmp/root` # mount the root partition from your hard-disk
`freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ upgrade -r 11.2-RELEASE` # upgrade(downgrade) 12.0-RELEASE to 11.2-RELEASE
`freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ install` # Install 11.2-RELEASE kernel
Do not reboot run again the install
`freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ install` # Install 11.2-RELEASE userland
`reboot`


----------



## ahhyes (Dec 13, 2018)

VladiBG said:


> boot with the LiveCD create a directory /tmp/root and mount your freebsd-ufs partition there. Setup your network and make a backup of your files. Then you can reinstall 11.2-RELEASE
> 
> EDIT:
> After you make a *backup *you can try this (This may be bad advice so try on your own risk)
> ...



Thank you 

The update process didnt get as far as installing the userland for 12.0-RELEASE - it screwed up after the first phase which was installing the 12.0 kernel (and I am guessing that phase also "upgraded" the bootloader), reverting that phase would probably be enough to fix the system.


----------



## ahhyes (Dec 13, 2018)

Is it possible this might work (after mounting the disk under /tmp/root):

`freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ rollback`


----------



## ahhyes (Dec 13, 2018)

ahhyes said:


> Is it possible this might work (after mounting the disk under /tmp/root):
> 
> `freebsd-update -b /tmp/root/ -d /tmp/root/var/db/freebsd-update/ rollback`



Well I answer my own question, it wont work, freebsd-update complains it cant find the rollback dir, so I ran the same command under `truss` and here is the result:





So `truss` reveals what `freebsd-update` is looking for when it tries to rollback.. And the subsequent `find` command I did shows what rollback directories actually exist.. Interestingly the rollback dir has the right date on it for when I did the upgrade, but `freebsd-update` seems to have other ideas on what it wants to look for.. I guess this is the end of the road for me?


----------



## VladiBG (Dec 13, 2018)

Your boot loader shouldn't be changed by freebsd-update. I think that rollback is only for the updates and won't work for major version changes where you also merge some configuration files. But after you cannot boot from the 12.0 livecd then the bootloader from 12.0 is not working for you. You still can manualy update it using
`gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada#`

edit:
if  everything else fail then just install a fresh copy of 11.2-RELEASE and restore your files from the backup.


----------



## ahhyes (Dec 13, 2018)

VladiBG said:


> Your boot loader shouldn't be changed by freebsd-update. I think that rollback is only for the updates and won't work for major version changes where you also merge some configuration files. But after you cannot boot from the 12.0 livecd then the bootloader from 12.0 is not working for you. You still can manualy update it using
> `gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada#`
> 
> edit:
> if  everything else fail then just install a fresh copy of 11.2-RELEASE and restore your files from the backup.



Thank you for the help so far  Appreciate it.

There absolutely were boot loader changes:
https://www.freebsd.org/releases/12.0R/relnotes.html#boot

Its starting to look like I will have to do what you suggested earlier, get the network running manually and get the data off the drive to somewhere safe and reinstall...


----------



## _martin (Dec 13, 2018)

truss output at least confirms it's a 64b version. Interesting that dump in the first pic shows 32b regs on 64b (even in early stage of the boot ; but I guess it's more interesting to me ..).

If I were you I'd go for fresh install. It maybe a bit pain to rebuild/reinstall, but at least it won't be a mess of old/new stuff.
Steps are easy actually: you boot from the memstick/cd , mount the root partition to /mnt/, do a backup:


```
# cd /mnt
# mkdir old
# mv * old
```
Warning: This effectively messes up your original / as it moves it to /old.  However you don't loose data and you can still access them when needed. If you have some other, non-standard directories there, you could leave them ( basically you need to have fresh /bin /sbin /var /boot /lib /lib32 /usr /etc )

Now just to a fresh install of FreeBSD 11.2:


```
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/base.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/doc.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/lib32.txz
# fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.2-RELEASE/kernel.txz
# sh
# for i in base doc lib32 kernel; do tar --unlink -C /mnt -xvPjf ${i}.txz ; done
# cp /mnt/old/etc/fstab /mnt/etc
# cp /mnt/old/etc/rc.conf /mnt/etc/
# cp /mnt/old/boot/loader.conf /mnt/boot/
```
And reboot to it ..


----------



## gkontos (Dec 13, 2018)

That might be a very long shot! I had the same problem with a version of 10.X in one server while trying to boot with a USB drive connected. I read that you do not have any...

Can you disconnect all your disks and try booting off from the FreeBSD 12.0-RELEASE memstick?


----------



## ahhyes (Dec 13, 2018)

_martin said:


> truss output at least confirms it's a 64b version. Interesting that dump in the first pic shows 32b regs on 64b (even in early stage of the boot ; but I guess it's more interesting to me ..)



I would have expected to see the likes of RAX, RBX etc too... Maybe the register dump for BTX only displays the 32 bit portion of the CPU registers.

And, thank you for your help too, I will look into the reinstall steps given. I am also considering just backing up what I can and reinstalling the OS. The most important stuff is on my zfs array which should be unaffected by a reinstall as long as I backup the critical configuration on the main drive used by the OS first.


----------



## _martin (Dec 13, 2018)

I've put that into my TODO list -- out of my curiosity I'd like to see where it does that (my hunch is that it's early in the boot, page tables are not set yet, using 32b part of regs only).

Yeah, do a backup for sure. It would be even better if you can test this in VM first. I'm using this approach when I want to have cleanly installed system ; basic installation is done from old running system. With ZFS on root you can even prep boot env without beadm. 

When it comes to config I have this one liner to save etc: 

`find /etc /usr/local/etc/ -depth -print | cpio -oaV | gzip -c > `hostname`_etc.cpio.gz`

But it's still nice to have all files at hand by some means (/old, external backup..).


----------



## ahhyes (Dec 13, 2018)

I woke up this morning and had a thought.. What if I booted from the USB drive that has the 11.2-RELEASE image, and mount the hard disk filesystem, then overwrite the /boot/loader on the hard disk with the one from 11.2-RELEASE

Well well well!! That has resurrected my server!!

So the problem is in /boot/loader in 12.0-RELEASE


----------



## ahhyes (Dec 13, 2018)

So it boots normally now. So I am in two minds what to do right now... run `freebsd-update` and let it update the userland to 12.0-RELEASE or attempt a rollback. But I wont be doing either until I have made backups of the OS drive (in case things go pear shaped again).

So right now I have a 12.0-RELEASE kernel and 11.2-RELEASE userland. At least I have gotten access to my data again!


----------



## ahhyes (Dec 14, 2018)

After backing things up, I decided to just run `freebsd-update install` and let the upgrade complete, since the likelyhood of the userland updates causing another boot issue are low. The upgrade has worked. I've bought the server back from the dead and its now running 12.0-RELEASE - but its still using the /boot/loader from 11.2-RELEASE. 

So things will continue to work unless another update comes which overwrites that file again with a bad version. I hope if there is an update to /boot/loader in the near future, it is one that fixes the problem that caused this whole mess for me!


----------



## VladiBG (Dec 14, 2018)

You can test the 13.0-CURRENT boot image to see if you can boot with it.


----------



## ahhyes (Dec 14, 2018)

VladiBG said:


> You can test the 13.0-CURRENT boot image to see if you can boot with it.


I'll give that a try  Something is seriously wrong with /boot/loader in 12.0-RELEASE though...


----------



## cbrace (Dec 14, 2018)

My experience as well. I've reverted to v11.2 for the foreseeable future.


----------



## Datapanic (Dec 15, 2018)

In conjunction with all that - I am trying to get a new install of poudriere to work with 12.0 and it's been rough, ruff, furf.  SSL has been "updated" and borks a lot of ports.  That, and php56 is eol soon.  What a nightmare!  Is there going to be an 11.3??? ??? ?/ ?


----------



## ahhyes (Dec 16, 2018)

VladiBG said:


> You can test the 13.0-CURRENT boot image to see if you can boot with it.


I looked on the ftp and cannot find a 13.0-CURRENT. Any ideas?


----------



## VladiBG (Dec 16, 2018)

https://www.freebsd.org/where.html


----------



## ahhyes (Dec 16, 2018)

VladiBG said:


> https://www.freebsd.org/where.html


Thank you.
I am going to test with https://download.freebsd.org/ftp/sn...URRENT-amd64-20181213-r342020-memstick.img.xz (obviously I will `unxz` this image before writing it to the usb drive)


----------



## ahhyes (Dec 16, 2018)

VladiBG said:


> You can test the 13.0-CURRENT boot image to see if you can boot with it.



Tested: *FreeBSD-13.0-CURRENT-amd64-20181213-r342020-memstick.img*

This image boots fine.


----------



## chrcol (Dec 16, 2018)

Some old members on here may remember when I upgraded a machine to 9.0 and then suffered nasty filesystem corruption caused by a new ufs journaling feature, after that day I learnt my lession and now avoid x.0 releases.  Best of luck to those who are getting their systems back up and running.

If you want to try out 12, I suggest STABLE, as any patches for found problems will go in there.


----------



## ahhyes (Dec 16, 2018)

chrcol said:


> Some old members on here may remember when I upgraded a machine to 9.0 and then suffered nasty filesystem corruption caused by a new ufs journaling feature, after that day I learnt my lession and now avoid x.0 releases.  Best of luck to those who are getting their systems back up and running.
> 
> If you want to try out 12, I suggest STABLE, as any patches for found problems will go in there.



Ouch.... well I am fortunate all my filesystems were intact and it was just a bootloader bug this time... 

To be fair on the devs, I have had years of upgrades to .0 releases and never hit any severe issue, until now. First time for everything eh?


----------



## _martin (Dec 17, 2018)

After upgrading some LAB servers and few VMs I've decided today I'll upgrade my main machine. I even updated BIOS while I was at it.  I'm used to upgrade my FreeBSD boxes using svn (cvs back then), I used the same approach today. And what do you know?






Booting 12.0 went fine, reboot after installworld was not sucessfull (as new /boot files got installed). I updated to amd64 12.0 r342177.
My board: S1200BT (07/25/2018) on Xeon E3-1240.

Luckily I was able to interrupt boot0 in very early stage and boot manually to 11.2 (I'm using ZFS on root but my BIOS (not even in EFI) didn't see virtual media from KVM so it would be painful ; here I saved a trip to DC).

EDIT: I forgot to mention I opened  PR for this: PR 234110


----------



## Quarter Wave Vertical (Dec 18, 2018)

More than 2 weeks ago, I accidentally trashed my system when I attempted to install FreeBSD 12.0-RC3.  I did a fresh re-installation of 11.2 with Mate, but I couldn't get Mate to work properly:

https://forums.freebsd.org/threads/...2-mate-not-working-properly.68430/post-409092

I installed 11.2 on an external HD but still no luck with the problems I had with Mate.

Earlier this evening, I attempted to upgrade that drive from 11.2 to 12.  The problem is that my machine keeps crashing when I attempt this:

`freebsd-update -r 12.0-RELEASE upgrade`

(At least I think I typed "12.0-RELEASE" rather than "12-RELEASE".)

My machine is more than 10 years old, so I'm wondering if that might have something to do with it.  I don't know off-hand what processor it uses but it's definitely Intel.

The reason I'm doing on the external drive is that I don't want to attempt the upgrade just yet on an 11.2 system I already have installed plus I'm hoping to see if I still have the same problems with Mate as I had before.

Does anyone have any suggestions?  Thank you.


----------



## Datapanic (Dec 18, 2018)

One of the big challenges with FreeBSD 12.0-RELEASE is with *OpenSSL 1.1* - several ports were not ready for it and now just don't build or work.  https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=openssl

Additionally, even some windows apps, like LDAP Admin won't talk via ssl/tls to a FreeBSD 12.0-RELEASE running OpenLDAP.


----------



## chrcol (Dec 18, 2018)

Datapanic said:


> One of the big challenges with FreeBSD 12.0-RELEASE is with *OpenSSL 1.1* - several ports were not ready for it and now just don't build or work.  https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=openssl
> 
> Additionally, even some windows apps, like LDAP Admin won't talk via ssl/tls to a FreeBSD 12.0-RELEASE running OpenLDAP.



Definitely a bold move to jump to 1.1 in base.  The solution for ports is probably to install 1.0.x from ports and use that for ports to link to.  When it was proposed a couple of years back to switch the default openssl port to 1.1 thankfully I managed to get them to abandon the idea as I had notice a lot of breakages when compiling against it, even in commonly used software, although I expect now alot of those are fixed, but its clear they not all fixed.

I advise using ports openssl anyway, simply because its fairly frequently updated and its much easier to keep a port updated than something in the base.


----------



## xtaz (Dec 18, 2018)

Regarding OpenSSL they had no choice because older versions will not be supported upstream any longer throughout the whole time that 12.x will be supported. I quote from their website:


```
Version 1.1.0 will be supported until one year after the release of 1.1.1
Version 1.0.2 will be supported until 2019-12-31 (LTS).
Version 1.0.1 is no longer supported.
Version 1.0.0 is no longer supported.
Version 0.9.8 is no longer supported.
```

In regards to reports of peoples boot loader crashing. They changed the loader from forth to lua and I'm guessing this is causing issues with some people.


----------



## _martin (Dec 18, 2018)

As you can find in PR i submitted I tested the 12-stable - issue seems to be fixed there. But as you Datapanic mentioned OpenSSL is causing other issues. 
I don't feel like doing any workarounds to have my needed ports up.  Sidenote: I wonder who decided that `uname -a` should not show the built date and kernel config used.

Too many problems with 12.0 on main machine, I reverted back to 11.2 and will stay there for a while.


----------



## terescoj (Dec 18, 2018)

ahhyes said:


> I woke up this morning and had a thought.. What if I booted from the USB drive that has the 11.2-RELEASE image, and mount the hard disk filesystem, then overwrite the /boot/loader on the hard disk with the one from 11.2-RELEASE
> 
> Well well well!! That has resurrected my server!!
> 
> So the problem is in /boot/loader in 12.0-RELEASE




Thanks for this workaround.  I was stuck with this same error - the inability for the 12.0 /boot/loader to find anything on my disk.  Booting from a memory stick 11.2 and copying over /boot/loader let me boot and I am also in the process of continuing the upgrade.  I have everything backed up, but this (assuming it works through to the end of the upgrade) saved me a whole lot of time that I was thinking I'd need to spend on a reinstall and restore.  I'd be happy to send along information about my hardware if it would be helpful for those tracking this down.  I upgraded two machines: the first had no problems, the second failed to boot with the lua error described in this thread.


----------



## Quarter Wave Vertical (Dec 19, 2018)

Quarter Wave Vertical said:


> More than 2 weeks ago, I accidentally trashed my system when I attempted to install FreeBSD 12.0-RC3.  I did a fresh re-installation of 11.2 with Mate, but I couldn't get Mate to work properly:
> 
> https://forums.freebsd.org/threads/...2-mate-not-working-properly.68430/post-409092
> 
> ...


An update on my previous posting.

I tried upgrading that drive to 12.0 again tonight and no joy.  My machine crashed with a kernel panic.

When the actual upgrading process started, I got the following errors:

```
(da4:umass-sim1:1:0:0):  READ(10). CDB: 28 00 02 58 56 a8 00 00 80 00
(da4:umass-sim1:1:0:0):  CAM status: SCSI Status Error
(da4:umass-sim1:1:0:0):  SCSI status: Check Condition
(da4:umass-sim1:1:0:0):  SCSI sense: MEDIUM ERROR asc:11,0 (unrecovered read error)
(da4:umass-sim1:1:0:0):  Error 5, Unretryable error
g_vfs_done9):da4p2[READ(offset=xxxxxxxxxxx, length=yyyyy)]error =5
g_vfs_done9):da4p2[READ(offset=xxxxxxxxxxx, length=yyyyy)]error =5
```
where xxxxxxxxxxx is a location and yyyyy a size.

I also got this message:

```
sha256: ///usr/lib/debug/boot/kernel/geom_raid3.ko.debug:  Input/output error
```
Still, the system inspection continued and got to the point where files were supposed to be downloaded.  A few moments after that, the crash occurred.

I remain baffled as I was able to install 11.2 direct onto the drive without any apparent problems.


----------



## SirDice (Dec 19, 2018)

Those errors look like they're caused by a faulty memory stick. Have you tried a different stick?


----------



## Quarter Wave Vertical (Dec 20, 2018)

SirDice said:


> Those errors look like they're caused by a faulty memory stick. Have you tried a different stick?


That's a possibility.  I think the HD was out of a scrapped second-hand laptop, so it might be on its last legs.  I put the drive in an enclosure, which I bought brand new, but that's not necessarily a guarantee that things are working properly.

I've got some other external drives I could use to see if that error occurs again.


----------



## Polyatomic (Dec 21, 2018)

_martin said:


> Sidenote: I wonder who decided that `uname -a` should not show the built date and kernel config used.


Straight true forum member _martin, this speciality should make its appearance after building the base.


----------



## Datapanic (Dec 21, 2018)

FreeBSD 12.0-RELEASE's uname -a will show the kernel config:

FreeBSD nas02.somestupiddomainname.net 12.0-RELEASE FreeBSD 12.0-RELEASE VMWARE  amd64

But they definately (yes I know I spelled "definitely" wrong) took away a lot of info compared to FreeBSD 11.x:

FreeBSD nas01.somestupiddomainname.net 11.2-RELEASE-p6 FreeBSD 11.2-RELEASE-p6 #0: Wed Dec  5 09:00:58 MST 2018     root@nas01.somestupiddomainname.net:/usr/obj/usr/src/sys/VMWARE  amd64

So when /etc/rc.d/motd creates a new /etc/motd file, we get

FreeBSD 12.0-RELEASE VMWARE

vs

FreeBSD 11.2-RELEASE-p6 (CUSTOM) #0: Wed Dec  5 09:00:58 MST 2018


----------



## Deleted member 30996 (Dec 21, 2018)

On a new build from scratch I get:


```
$ uname -a
FreeBSD obake 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC  amd64
```


----------



## _martin (Dec 21, 2018)

It's the well known old output

```
# uname -a
FreeBSD node01 11.2-RELEASE-p2 FreeBSD 11.2-RELEASE-p2 #0 r338291: Fri Aug 24 15:10:39 CEST 2018     root@node01:/usr/obj/usr/src/sys/LABNODE  amd64
#
```

vs new


```
# uname -a
FreeBSD node01 12.0-RELEASE FreeBSD 12.0-RELEASE r341819 LABNODE  amd64
#
```

I actually miss the date there. Is it really that relevant ?  One can argue that it doesn't give you too much information ..  I just always liked it there.


----------



## SirDice (Dec 21, 2018)

This change is due to an effort to create "reproducible builds", meaning every build (from the same source) has to result in exactly the same images. A build-time timestamp would make that impossible. 

https://wiki.freebsd.org/ReproducibleBuilds


----------



## Polyatomic (Dec 21, 2018)

Noble and correct members, I send greetings. My intuition was incorrect with regards to the uname output, I can see now. Thanking you SirDice.
`cat /usr/src/tools/build/options/WITHOUT_REPRODUCIBLE_BUILD` gives:

```
.\" $FreeBSD: releng/12.0/tools/build/options/WITHOUT_REPRODUCIBLE_BUILD 338644 2018-09-13 14:53:42Z emaste $
Set to include build metadata (such as the build time, user, and host)
in the kernel, boot loaders, and uname output.
Successive builds will not be bit-for-bit identical.
```


----------



## MarcoB (Dec 21, 2018)

My uname says that I have a GENERIC kernel, but I definitely build it with `KERNCONF=` (on 12-STABLE). Something changed here too.


----------



## SirDice (Dec 21, 2018)

MarcoB said:


> My uname says that I have a GENERIC kernel, but I definitely build it with  KERNCONF= (on 12-STABLE). Something changed here too.


You didn't modify the `ident` in your custom kernel. KERNCONF refers to the filename of the kernel config, the `ident` is shown in the uname(1) output.


----------



## MarcoB (Dec 21, 2018)

You're right but this wasn't the case in earlier versions. In FreeBSD 11 and older, I didn't change the ident either but uname showed the KERNCONF in the uname output.


----------



## SirDice (Dec 21, 2018)

MarcoB said:


> You're right but this wasn't the case in earlier versions.


This has been the case since I started with 3.0.



MarcoB said:


> I didn't change the ident either but uname showed the KERNCONF in the uname output.


Besides the ident it showed the whole path to the kernel config (the actual file).


----------



## MarcoB (Dec 21, 2018)

Well, I never changed the ident and been using STABLE for almost two decades.


----------



## SirDice (Dec 21, 2018)

MarcoB said:


> Well, I never changed the ident and been using STABLE for almost two decades.


Then you've had a really bad habit the past 20 years. The `ident` always had to be modified for custom kernels. Common convention is to use the same name as the kernel config file.


----------



## pva (Dec 25, 2018)

ahhyes said:


> After backing things up, I decided to just run `freebsd-update install` and let the upgrade complete, since the likelyhood of the userland updates causing another boot issue are low. The upgrade has worked. I've bought the server back from the dead and its now running 12.0-RELEASE - but its still using the /boot/loader from 11.2-RELEASE.



I ran into the same issue when upgrading my backup server from 11.2 to 12.0. After installing the 12.0 kernel and bootloader, the BTX loader would hang immediately after enumerating all the BIOS drives. Booting from a 11.2 live USB and replacing the 12.0 loader with the 11.2 one allowed me to boot and finish the upgrade. Everything seems to be working fine now, save for the "frankenloader" setup. I see _martin opened a PR for a similar issue, but I wonder if it's the same bug we're encountering (I'm not seeing a "BTX halted" message, for example). Did you file a PR of your own? If I have the time, I'll try a loader from a recent STABLE snapshot to see whether the problem has already been fixed.

FWIW, this was on a SuperMicro A1SAi-2550F motherboard with a Dell H200 cross-flashed to a LSI SAS9211-8i, and a RAID-Z3 pool of 12 disks. The OS is booted from a UFS filesystem on a USB stick.


----------



## cbrace (Dec 25, 2018)

I'm following this thread because I ran into this issue. 12.0-RELEASE-p1 appears to include an updated bootloader, and I'm wondering what the easiest way would be to test this. For example, can one boot v12.0 from a USB stick, run freebsd-install update to bring it to p1, and reboot from the USB stick again? If boots OK, then presumably one can safely apply the update to a live v11.2 system?


----------



## _martin (Dec 25, 2018)

cbrace ZFS and boot environment make this very easy. I created a new BE, booted to it and tested. When it failed I was able to stop the bootloader and choose the proper BE myself.

pva As mentioned in PR I was testing few versions and got different behavior out of it. So it may depend on the given version/release of FreeBSD you're using. I opened PR to my specific issue. As it's resolved in CURRENT and even STABLE I got not further questions/response.


----------



## Crivens (Dec 26, 2018)

Using external enclosures may not play nice with SMART on the drives.


----------



## chrcol (Dec 27, 2018)

Polyatomic said:


> Noble and correct members, I send greetings. My intuition was incorrect with regards to the uname output, I can see now. Thanking you SirDice.
> `cat /usr/src/tools/build/options/WITHOUT_REPRODUCIBLE_BUILD` gives:
> 
> ```
> ...



I was about to post they should add the ability to override back to the older more sane behaviour, and thankfully it looks like they have done that. 

But as always we really need proper changelogs in FreeBSD, the official changelogs provided are a very tiny snapshot and wont show this kind of information, users shouldnt have to dig into src files to find out about feature changes and config syntax.


----------



## cbrace (Dec 27, 2018)

_martin said:


> cbrace ZFS and boot environment make this very easy. I created a new BE, booted to it and tested. When it failed I was able to stop the bootloader and choose the proper BE myself.


 I need to learn how to do this. Will do my homework.  Thanks for the pointer.


----------



## cbrace (Dec 27, 2018)

_martin said:


> cbrace ZFS and boot environment make this very easy. I created a new BE, booted to it and tested. When it failed I was able to stop the bootloader and choose the proper BE myself.


_martin a quick question: when I upgraded a NAS running v11.2 to v12.0 a short time ago using freebsd-update, on reboot the system hung at the BTX loader stage, right at the start of the boot process. If I had previously created a boot environment before the upgrade, how would I have been able to revert to that earlier boot environment if I could not even get past the BTX stage? I see that the BE can be selected in the BSD loader, but that is at a later part of the boot process. Hope my question is clear.


----------



## _martin (Dec 27, 2018)

cbrace As I mentioned in my previous posts -- press ESC when you see the rotating pipe in the upper left corner ( | / - \ ). That's the boot0 being loaded and booting. You'll be presented with an option to manually write down the location of bootloader.

You'll see something similar to this:

```
-
FreeBSD/x86 boot
Default: rpool/ROOT/11.2:/boot/zfsloader
boot:
```

All this is happening before the main menu you posted in picture.


----------



## Quarter Wave Vertical (Dec 30, 2018)

Quarter Wave Vertical said:


> That's a possibility.  I think the HD was out of a scrapped second-hand laptop, so it might be on its last legs.  I put the drive in an enclosure, which I bought brand new, but that's not necessarily a guarantee that things are working properly.
> 
> I've got some other external drives I could use to see if that error occurs again.


I'm still not altogether sure what the problem was.

I recently bought a refurbished laptop and did a fresh installation of 12.0.  It runs Xfce with no apparent problems, though, when I tried Mate, the results were a bit wobbly.


----------



## Quarter Wave Vertical (Dec 30, 2018)

Crivens said:


> Using external enclosures may not play nice with SMART on the drives.


I installed 11.2 on a different drive with a different machine.  The installation went OK, but booting from that drive didn't work for me.


----------



## Datapanic (Dec 30, 2018)

At this point, with all the negative results with root on ZFS updates from 11.2 to 12.0, I think I'll just sit back and wait for 12.1 to come out.  I have back ups and all that, but I don't want to experience the frustration and lose the time doing what others have gone through.  I want seamlessness!


----------



## Remington (Dec 30, 2018)

Datapanic said:


> At this point, with all the negative results with root on ZFS updates from 11.2 to 12.0, I think I'll just sit back and wait for 12.1 to come out.  I have back ups and all that, but I don't want to experience the frustration and lose the time doing what others have gone through.  I want seamlessness!



I agree especially for production servers as I cannot afford to have downtime to sort this out.  I would wait 4 to 6 months and upgrade with STABLE release since most bugs would have been fixed by that time.


----------



## Crivens (Dec 30, 2018)

To add my 2c to this: I did an upgrade to 12.0 with ZFS on root without problems. But I went the source way. Maybe that is what makes the difference.


----------



## bisi (Apr 5, 2019)

This is so closely related, I don't want to start a new thread, yet it has been 3 months....

So, by way of background, I have a bunch of 11.2 production machines that are falling over (hanging, unresponsive) at semi-regular intervals, after bing upgraded from previous 10.x configurations.  These hangs were always preceded by "out of swap" messages in the logs, but never any significant amount of swap being used, yada yada yada.  All are booted off a UFS formatted SSD, running varying configurations of zfs for backup.  Most backup is via rsync (rsnapshot) tasks on the freebsd boxes, plus NFSv3 exports, to support ghettoVCB writing from esxi servers.  I spent a lot of time troubleshooting, with no joy, so eventually built some scripts that watched for the first precursors to the hangs (the out-of-swap messages) and rebooted the servers. With logging.  I didn't learn much, except that it seemed to happen for some of the machines during the maintenance/checking tasks.  The hangs/out-of-swap messages never happened during rsnapshot or while nfs writes were in progress.  No upgrade to the 11.2 stream fixed the issue.

To get to the point of this thread, I've sucessfully upgraded my way out of *that* morass with a couple of boxes, going to 12.0-RELEASE p1 and p3. After waiting more than 3x the longest no-hanging period, I concluded I am back on solid ground.  Now I've bumped into this issue, best described by pva. (although I also have an older machine that behaves like the OP reported). For this new machine (only in production for about 6 months), I decided to see if the issue was hardware related.  I'm thinking not.  After upgrading the kernel (`freebsd-update -r 12.0-RELEASE upgrade`) and rebooting,  the boot loader stops with the enumeration of the drives.  There are 5 drives total (all SATA).



If I unplug one of the drives, the system gets to the boot menu, and will even boot (zpool is exported and zfs is turned off in rc.conf).  It doesn't matter which drive I unplug, or which sata port I plug the drives into.  These are all WD Red Pro 4TB drives (except the SSD, of course).

Only the kernel has been upgraded so far.

I did notice that the upgrade changed`/etc/defaults/rc.conf`, without giving me any chance to edit it beforehand (`Does this look reasonable (y/n)?`), so I'm wondering if that changed something that relates to the number of drives. Especially the `cfumass_` entries.  I've attached the output of that as a file

The earlier boxes that upgraded successfully were all raidz-1 or mirrors (only 3 or 4 drives, total).  .

So, since I have backup, etc, and can boot from an external usb stick to 11.2 and import/export the zpool and all the config info, I'm going to try the route of replacing the apparently-faulty bootloader in 12.0-RELEASE with the working one from 11.2.

My biggest question at the moment is whether this is worthwile making a problem report, or if the other issues have it covered already.  This is not a machine that I can experiment with -- it needs to go back into production tomorrow.   I'm betting that I can reproduce the issue, however, with another machine.

---EDIT---
So this is really odd... (OK -- more like WTF?!)  When I disconnect  the drives making the zfs pool, and boot with  just the SSD...

```
$ uname -a
FreeBSD nas0.domainnme.tld 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC  i386
```
Clearly something really funky happened with the upgrade process.  This *was* an amd64 install when it was 11.2.   Just want to mention that I have a recipe for these upgrades, and I copy/pasted the same update command string that worked two prior times.


----------



## VladiBG (Apr 5, 2019)

You can boot using 11.2 and replace the boot loader.


----------



## bisi (Apr 9, 2019)

I can confirm that using the 11.2 boot loader works -- my co-worker did that with another UFS+raidz2 machine (which was upgraded to escape the silent hangs, preceded by (bogus?) out-of-swap messsages).  

So, this quick fix raises the question -- can anybody tell us at what point the faulty bootloader will be re-installed during an upgrade?

In the case I am writing about here, I had ended up with an i386 kernel (at least according to uname), so I bailed and re-installed (having  full backup of all the config and other modifications).  I will be sticking with 11.2 p9 and the "out-of-swap" problem with that box. Fingers crossed that 11.3 will rectify the problem that got me here.


----------



## SirDice (Apr 10, 2019)

bisi said:


> can anybody tell us at what point the faulty bootloader will be re-installed during an upgrade?


If by bootloader you mean the contents of the MBR and/or freebsd-boot partition, nowhere. You always have to update those by hand. You can easily verify this, /usr/sbin/freebsd-update is a rather large shell script. So anyone can read it.


----------

