# "device vanished" when upgrading from FreeBSD 12.2 to 13.0



## molofishy (Jun 29, 2021)

I could not ssh into my FreeBSD after upgrading (see attached photo -text is displayed indefinitely). I see there is a long thread about this issue already; but is there a quick fix? And if not how can I revert back to 12.2?






Thanks


----------



## grahamperrin@ (Jun 29, 2021)

Please retry attaching, or sharing, the photograph.

If remote access (ssh) is not possible, when will you have physical access?

Did you create a ZFS boot environment?


----------



## bakul (Jun 30, 2021)

The attached photograph showed fine a few hours ago...

Having read the referenced bugs thread, it seems there is a solution (see the last message in the thread). I don't have a machine that hangs sin this manner but I at least verified that replacing 13.0 release /usr/src/sys/dev/usb/usb_hub.c with the version from the stable/13 branch doesn't break things for me. The only difference in this file is some additional error checking so I am not surprised.

molofishy has a few options, none of them ideal but not too bad either.

Revert to 12.2 and wait till 13.1. The issue here is wait (assuming the fix is in 13.1).
Save the current kernel and the source file usb_hub.c in a safe place `cd /usr/src/; sudo make kernel`. The issue here is that before doing freebsd-update you have to restore the kernel and src file and then, if there are any kernel changes, rebuild the kernel.
Run the latest stable/13 snapshot kernel. Generally newer kernel deal with older userland just fine. The issue here is that freebsd-update won't work well if there are any kernel changes. You will have to find a later snapshot and install it.
Note that there is some danger of foot-shooting with 2. but if you are comfortable building kernels locally, it is something worth trying. I'd definitely recommend using boot environment. Either with bectl (if root on zfs) or beadm(if root on ufs?). 3. is easier than 2. but protection against some future exploit may be an issue for a few days.


----------



## richardtoohey2 (Jun 30, 2021)

The picture was there, went away, and is now back - but in case it gets shy again, I've manually copied (don't _think_ any typos!) down the message - basically this repeated forever

```
uhub_reattach_port: giving up port 1 reset - device vanished: change 0xfb status 0x7fb
uhub_reattach_port: giving up port 2 reset - device vanished: change 0xfb status 0x7fb
uhub_reattach_port: giving up port 3 reset - device vanished: change 0xfb status 0x7fb
uhub_reattach_port: giving up port 4 reset - device vanished: change 0xfb status 0x7fb
```


----------



## grahamperrin@ (Jun 30, 2021)

Thanks,



bakul said:


> … bectl (if root on zfs) or beadm(if root on ufs?). …



`beadm` is essentially for boot environments with ZFS, not UFS. Two slightly different manual pages:

<https://www.freebsd.org/cgi/man.cgi?query=beadm&sektion=1&manpath=FreeBSD+13.0-RELEASE+and+Ports> | beadm(1)
<https://www.freebsd.org/cgi/man.cgi?query=beadm&sektion=8&manpath=FreeBSD+13.0-RELEASE+and+Ports> | beadm(8)






						254466 – sysutils/beadm-devel section (1) and sysutils/beadm section (8) for the two manual pages for beadm; consequences
					






					bugs.freebsd.org
				




*UFS*

UFS Boot Environments | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗 (2021-04-02)


----------



## mark_j (Jun 30, 2021)

bakul said:


> The attached photograph showed fine a few hours ago...
> 
> Having read the referenced bugs thread, it seems there is a solution (see the last message in the thread). I don't have a machine that hangs sin this manner but I at least verified that replacing 13.0 release /usr/src/sys/dev/usb/usb_hub.c with the version from the stable/13 branch doesn't break things for me. The only difference in this file is some additional error checking so I am not surprised.
> 
> ...



You can always have multiple kernels lying around.
Option 2 is what I would do. Build the kernel with the patches and DO NOT do a freebsd-update at all, or at least until the patch makes it into the next release (13.1?).

Any patches for security issues can easily be patched in and another build of the kernel performed. It just takes a lot longer than a binary upgrade, but if you have a stable machine because of building the kernel yourself, then I know what I would opt for.


----------



## bakul (Jun 30, 2021)

grahamperrin thanks for the correction!



mark_j said:


> Build the kernel with the patches and DO NOT do a freebsd-update at all


This would mean switching source based updates and not everyone would want to do that. And once you are on this path, it is hard to switch back to freebsd-update. Given that 13.0-RELEASE-p3 just came out....


----------



## Emrion (Jun 30, 2021)

bakul said:


> grahamperrin thanks for the correction!
> 
> 
> This would mean switching source based updates and not everyone would want to do that. And once you are on this path, it is hard to switch back to freebsd-update. Given that 13.0-RELEASE-p3 just came out....


Not hard. You have just to wait until the patch is included in a release and simply use `freebsd-update -r the_release_name upgrade install`.

But, it's difficult to follow the security patches. When I was in this case, I just never applied one (because you have to compile the kernel and/or the world and it may be long and resource consuming).


----------



## T-Daemon (Jun 30, 2021)

It's perfectly practicable to use freebsd-update(8) to binary update 'world' only and build a custom kernel with official and custom patches from source afterwards if necessary. To evaluate necessity inspect the changes made to the tree viewed after (git) updating /usr/src.

To exclude the kernel form binary update set  the 'Components' option in freebsd-update.conf(5) accordingly, (exclude source also) i.e.:

```
# Components of the base system which should be kept updated.
#Components src world kernel
Components world
```

To "switch back" from source based kernel updates, modify /etc/freebsd-update.conf.

And to reduce kernel build time, eventually into a few minutes, depending on the CPU, enable META_MODE.


----------



## molofishy (Jul 10, 2021)

bakul said:


> The attached photograph showed fine a few hours ago...
> 
> Having read the referenced bugs thread, it seems there is a solution (see the last message in the thread). I don't have a machine that hangs sin this manner but I at least verified that replacing 13.0 release /usr/src/sys/dev/usb/usb_hub.c with the version from the stable/13 branch doesn't break things for me. The only difference in this file is some additional error checking so I am not surprised.
> 
> ...


Thanks for all the support FreeBSD community! 
Option 2 sounds too advanced for me (unless you can give be step by step instructions from the start screen), so I will probably try for Option 1: "Revert to 12.2". 
How can I revert to 12.2 with freebsd-update if I cannot even get into any command prompt?

The start screen looks like:

```
1. Boot Multi user
2. Boot Single user
3. Escape to loader prompt
4. Reboot
5. Cons: Video

Options:
6. Kernel: default/kernel (1 of 2)
7. Boot Options
```
1. & 2. result in the infinite "device vanished" issue. 
3. provides with me a list of commands, none of which are freebsd-update
5. & 6. toggle between two options on the welcome screen: 5. (video / duel); 6. (kernel / kernel old)
7. gives me 5. Options:


```
1. Back to main menu
2. Load System Defaults
3. Safe Mode
4. Single user
5. Verbose.
```

Thank you very much and apologies for my rudimentary questions.


----------



## T-Daemon (Jul 10, 2021)

molofishy said:


> Option 2 sounds too advanced for me (unless you can give be step by step instructions from the start screen),


I could give you a step by step guide but it's not a trivial process. To give you an idea what to expect see list:

- Booting a FreeBSD installer image
- Mounting the root partition of the affected sytem
- chroot(8) into affected system
- Install devel/git if not installed
- Clone the 13.0-RELEASE source code
- Fetch patches 1 and 2 (before booting FreeBSD installer, saved on the installer image or extra USB stick)
- Apply patches
- Build, install new kernel . Depending on the CPU this can take up to 1 hour or more
- Exit chroot(8)
- Reboot into system with patched kernel

And every time a security and/or errata patch is released the source code needs to be updated, the kernel rebuild.  That's not a system maintenance which can be called user friendly.

If installing  12.2-RELEASE fresh isn't an option, I suggest you downgrade the affected system. If you have valuable data on the affected system it would be prudent to make a backup before first if not already done. This applies also when installing 12.2 fresh.

To the downgrade process, in short:

- Boot a FreeBSD installer image, 12.2 or 13.0 if file system is UFS2, 13.0 if ZFS
- At the FreeBSD installer 'Welcome' dialog drop to 'Shell'
- Configure the network interface with dhclient(8) <interface  >
- Create a temporary mount point, i.e. /tmp/a
- Mount the affected system on that mount point. How to mount depends on the file system (UFS2 or ZFS)
- In this step, optional make a backup of the data on a mounted external disk for example
- chroot(8) into affected systems mount point
- Execute freebsd-update(8)
- After finishing downgrade exit chroot(8), reboot system

If you need more detailed instructions please ask.


----------



## bakul (Jul 10, 2021)

molofishy said:


> Thanks for all the support FreeBSD community!
> Option 2 sounds too advanced for me (unless you can give be step by step instructions from the start screen), so I will probably try for Option 1: "Revert to 12.2".
> How can I revert to 12.2 with freebsd-update if I cannot even get into any command prompt?
> 
> ...


Sounds like you can’t even boot. The "kernel old" choice will likely boot your 12.2 kernel but old kernel and new userland program combination usually doesn't work all that well....

I think you will have to boot from a usb device or a cd or dvd using an image for 12.2. I haven't done this before but perhaps this post can help you? 








						Upgrade disaster 11.2-RELEASE -> 12.0-RELEASE
					

If 12.0-RELEASE memstick image crash then verify it's checksum and try again. If you are sure that the downloaded image is correct and still cannot boot from it then the only option you have left is to revert back to 11.2. You can also can try to update your bios it may help also.  Its a good...




					forums.FreeBSD.org


----------



## T-Daemon (Jul 11, 2021)

molofishy said:


> ... from the start screen ...
> How can I revert to 12.2 with freebsd-update if I cannot even get into any command prompt?


If the file system is UFS2, instead booting a installer image to downgrade the affected system from there, the systems previous 12.2 kernel can be used to boot the system to a command prompt. At the boot menu ("start screen") press the 6 key, the displayed kernel menu must show `kernel.old`):


```
6. Kernel: default/kernel.old (2 of 2)
```

After system boot and log in try `freebsd-update rollback`. Check system version with `freebsd-version -kru` afterwards. freebsd-version(1). If all goes well the output should show

```
12.2-RELEASE-pn
12.2-RELEASE-pn
12.2-RELEASE-pn

(n = patch level)
```

The best solution would be a fresh 12.2-REALEASE install. But if it's not an option apply one of the methods suggested.

Again, if the data on the system is of any value, and if there are no backups made, it's strongly advised to make a backup before any downgrade operation!


----------



## molofishy (Jul 12, 2021)

grahamperrin said:


> molofishy UFS or ZFS?


ZFS - two SSDs with data mirrored across both.


----------



## grahamperrin@ (Jul 12, 2021)

Thanks. 



grahamperrin said:


> Did you create a ZFS boot environment?



molofishy if you did not create a boot environment before things went wrong, now will be a good time to adopt the habit. 

In normal circumstances: create, activate then boot an environment before making changes to it.


----------



## molofishy (Jul 12, 2021)

grahamperrin said:


> Thanks.
> 
> 
> 
> ...


I used ZFS only for users home directories; but the operating system was stored on a separate partition in UFS. I cannot ssh at the moment but I have physical access to the machine.


----------



## Emrion (Jul 12, 2021)

Have you tried to boot on the old kernel as T-Daemon instructed?


----------



## molofishy (Jul 12, 2021)

T-Daemon said:


> If the file system is UFS2, instead booting a installer image to downgrade the affected system from there, the systems previous 12.2 kernel can be used to boot the system to a command prompt. At the boot menu ("start screen") press the 6 key, the displayed kernel menu must show `kernel.old`):
> 
> 
> ```
> ...


Booting using the kernel.old option still results in the never-ending "device vanished" text; no command line interface; and no ability to ssh in.


----------



## bakul (Jul 12, 2021)

Can you boot using a usb memory stick? Use a 12.2 dvd image as described here: https://www.freebsd.org/releases/12.2R/announce/


----------



## T-Daemon (Jul 13, 2021)

molofishy said:


> Booting using the kernel.old option still results in the never-ending "device vanished" text; no command line interface; and no ability to ssh in.


Then you have not many choices left. Or you boot a FreeBSD installer image and try a system downgrade, or install a fresh system.


----------



## molofishy (Jul 15, 2021)

Now when I reboot, the first thing I see is:
"Samsung SSD 960 PRO 2TB
S.M.A.R.T Status Bad, Backup and Replace
Press F1 to Resume..."
I had setup 2 of those 2TB SSDs, with ZFS, with 100% redundancy... surely both didn't fail.. :/

Also, I have booted from a bootable USB with 12.2 on it - the Welcome screen looks a little different, so I know it's booting from the USB... however when I Boot as Multi user (the default),the never-ending "device vanished" text appears once again 

Could it be that one of the SSDs is no longer plugged in properly? Hmmm


----------

