# URGENT - Upgraded to FreeBSD-12.0 and no /boot directory exists



## byrnejb (Mar 23, 2019)

I just encountered a very bad situation at the end of an upgrade from FreeBSD-11.2 to 12.0.  After completing the `pkg-static upgrade -f` I ran `freebsd-update install` one last time and received this message: 
	
	



```
Cannot identify running kernel
```
.

When I run `sysctl -n kern.bootfile` I see: /boot/kernel/kernel but when I `ll /boot/kernel` I see:
	
	



```
ls: /boot/kernel: No such file or directory
```
.  I have located a directory called /bootpool/boot but I can only guess at what this is for.  

At the moment the system is running.  It uses a root on an *encrypted* *zfs* filesystem.  Regular snapshots are taken of the entire fs. 

I am in a very bad situation here.  I really need some help in getting this system back to normal.


----------



## driesm (Mar 23, 2019)

byrnejb said:


> snapshots are taken of the entire fs.



Can't you rollback to a snapshot of the entire system pre-upgrade?


----------



## D-FENS (Mar 23, 2019)

I also had the issue.
Do you have a separate ZFS boot pool? If so, in my case /boot is just a symlink to /bootpool, where the boot pool is mounted. The upgrade messed my symlink up. It was removed and an empty /boot directory was created with a handful of module files in it.
I fixed it by moving the files above to /bootpool and then recreating the symlink.

However, an even better solution would be to change the mountpoint of bootpool to be /boot. Be careful to move the files to the real ZFS pool first.


----------



## byrnejb (Mar 23, 2019)

/boot does not exist:


```
ll /boot
ls: /boot: No such file or directory
```
/bootpool/boot does exist:


```
ll /bootpool/boot 
total 8037
-rw-------  1 root  wheel     512 May 13  2016 ada0p4.eli
-rw-------  1 root  wheel     512 May 13  2016 ada1p4.eli
-rw-------  1 root  wheel     512 May 13  2016 ada2p4.eli
-rw-------  1 root  wheel     512 May 13  2016 ada3p4.eli
-r--r--r--  1 root  wheel    3554 Mar 23 09:20 beastie.4th
-r--r--r--  1 root  wheel    8192 Mar 23 09:20 boot
-r--r--r--  1 root  wheel     512 Mar 22 20:11 boot0
-r--r--r--  1 root  wheel     512 Mar 22 20:11 boot0sio
-r--r--r--  1 root  wheel     512 Mar 24  2016 boot1
-r-xr-xr-x  1 root  wheel   81920 Mar 23 09:20 boot1.efi
-r--r--r--  1 root  wheel  819200 Mar 23 09:20 boot1.efifat
-r--r--r--  1 root  wheel    7680 Mar 23 09:20 boot2
-r--r--r--  1 root  wheel    2126 Mar 23 09:20 brand-fbsd.4th
-r--r--r--  1 root  wheel    2806 Mar 23 09:20 brand.4th
-r--r--r--  1 root  wheel    1188 Mar 23 09:20 cdboot
-r--r--r--  1 root  wheel    6277 Mar 23 09:20 check-password.4th
-r--r--r--  1 root  wheel    1867 Mar 23 09:20 color.4th
drwxr-xr-x  2 root  wheel       3 Mar 23 09:20 defaults
-r--r--r--  1 root  wheel    4053 Mar 23 09:20 delay.4th
-r--r--r--  1 root  wheel     829 Mar 23 09:20 device.hints
drwxr-xr-x  3 root  wheel       3 Mar 23 09:20 dtb
-r--r--r--  1 root  wheel    1614 Mar 23 09:20 efi.4th
-rw-------  1 root  wheel    4096 May 13  2016 encryption.key
-rw-------  1 root  wheel    4096 Mar 23 09:38 entropy
drwxr-xr-x  2 root  wheel       2 Mar 24  2016 firmware
-r--r--r--  1 root  wheel    3695 Mar 23 09:20 frames.4th
-r--r--r--  1 root  wheel   62810 Mar 23 09:20 gptboot
-r--r--r--  1 root  wheel  108162 Mar 23 09:20 gptzfsboot
-r--r--r--  1 root  wheel   14363 Mar 23 09:20 isoboot
drwxr-xr-x  2 root  wheel     879 Mar 23 09:20 kernel
drwxr-xr-x  2 root  wheel     876 Mar 23 09:20 kernel.old
-r-xr-xr-x  3 root  wheel  421888 Mar 23 09:20 loader
-r--r--r--  1 root  wheel    7425 Mar 23 09:20 loader.4th
-rw-r--r--  1 root  wheel    1008 Mar 28  2018 loader.conf
-r-xr-xr-x  2 root  wheel  457728 Mar 23 09:20 loader.efi
-r--r--r--  1 root  wheel     468 Mar 23 09:20 loader.rc
-r-xr-xr-x  1 root  wheel  372736 Mar 23 09:20 loader_4th
-r-xr-xr-x  1 root  wheel  393216 Mar 23 09:20 loader_4th.efi
-r-xr-xr-x  3 root  wheel  421888 Mar 23 09:20 loader_lua
-r-xr-xr-x  2 root  wheel  457728 Mar 23 09:20 loader_lua.efi
-r-xr-xr-x  1 root  wheel  311296 Mar 23 09:20 loader_simp
-r-xr-xr-x  1 root  wheel  332800 Mar 23 09:20 loader_simp.efi
-r--r--r--  1 root  wheel    3110 Mar 23 09:20 logo-beastie.4th
-r--r--r--  1 root  wheel    2636 Mar 23 09:20 logo-beastiebw.4th
-r--r--r--  1 root  wheel    2214 Mar 23 09:20 logo-fbsdbw.4th
-r--r--r--  1 root  wheel    2631 Mar 23 09:20 logo-orb.4th
-r--r--r--  1 root  wheel    2354 Mar 23 09:20 logo-orbbw.4th
drwxr-xr-x  2 root  wheel      17 Mar 23 09:20 lua
-r--r--r--  1 root  wheel     512 Mar 24  2016 mbr
-r--r--r--  1 root  wheel    9260 Mar 23 09:20 menu-commands.4th
-r--r--r--  1 root  wheel   36016 Mar 23 09:20 menu.4th
-r--r--r--  1 root  wheel    6328 Mar 23 09:20 menu.rc
-r--r--r--  1 root  wheel   18597 Mar 23 09:20 menusets.4th
drwxr-xr-x  2 root  wheel       2 Mar 24  2016 modules
-r--r--r--  1 root  wheel     512 Mar 24  2016 pmbr
-r--r--r--  1 root  wheel  423936 Mar 23 09:20 pxeboot
-r--r--r--  1 root  wheel    2675 Mar 23 09:20 screen.4th
-r--r--r--  1 root  wheel    2613 Mar 23 09:20 shortcuts.4th
-r--r--r--  1 root  wheel   36282 Mar 23 09:20 support.4th
-r--r--r--  2 root  wheel  421480 Mar 23 09:20 userboot.so
-r--r--r--  1 root  wheel  345536 Mar 23 09:20 userboot_4th.so
-r--r--r--  2 root  wheel  421480 Mar 23 09:20 userboot_lua.so
-r--r--r--  1 root  wheel    3065 Mar 23 09:20 version.4th
drwxr-xr-x  2 root  wheel       3 Mar 23 09:38 zfs
-r--r--r--  1 root  wheel  262656 Mar 23 09:20 zfsboot
-r-xr-xr-x  3 root  wheel  421888 Mar 23 09:20 zfsloader
```

Now this system rebooted after the first `freebsd-update install` so does this mean that the second `freebsd-update install` caused the problem?  What does /boot look like on FreeBSD-12? Will running `ln -s /bootpool/boot /boot` fix this issue?


----------



## D-FENS (Mar 23, 2019)

Then it's quite easy:
`ln -s /bootpool/boot /boot`

or:
`zfs set mountpoint=/boot bootpool`

I used #1 but #2 is probably cleaner.

And to answer your question: Yes, freebsd-update messed it up. But the issue is quite minor, so I find it annoying yet easy.


----------



## D-FENS (Mar 23, 2019)

By the way, after you fix /boot, run `freebsd-update IDS` one time to see if something funny is going on.
Also use `freebsd-version -kru` to make sure you have all the components up to date.


----------



## byrnejb (Mar 23, 2019)

SOLVED

However I see this as well:


```
freebsd-update IDS
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 12.0-RELEASE from update4.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
/.cshrc has SHA256 hash 3e6203b75cdf31ea6877ddfd690e3e126577982a9f9fd69692e0d22c10134e24, but should have SHA256 hash fa6ca8c03acabab047418c40a61fd6b42f1ae2898ce2e354dd1cf33fc19b9253.
/.profile has SHA256 hash c997e558f0812a6b14811401651c4df0c90d7cbcfd2f384c9d62e108ec576bc5, but should have SHA256 hash 59d4d08c686122ea22cd4eb8f2c671c90f5e63480e46cdfa2cb876d383bff227.

/boot is a symlink, but should be a directory.
```
We know how this happened.  Should this be a hard link instead?


```
/etc/defaults/bluetooth.device.conf has 0444 permissions, but should have 0644 permissions.
/etc/defaults/devfs.rules has 0444 permissions, but should have 0600 permissions.
/etc/defaults/periodic.conf has 0444 permissions, but should have 0644 permissions.
/etc/defaults/rc.conf has 0444 permissions, but should have 0644 permissions.
/etc/group has SHA256 hash c7df410b8ebc4da5b9a9c1bb8287dca50e7a14b9d951d0a341aaa7128492c2da, but should have SHA256 hash 98bb7185f1de6e9c0bda14343915298042d74cb445be47729686a4778363e4bf.
/etc/hosts has SHA256 hash e6ea217db69704967b9f9287d5246ff8a239744a99cc9b906dec0f90ec99c715, but should have SHA256 hash f06f12bd610b6f9793e82c108eff1259bc328e244042a7596d2d69e4022e2e45.
/etc/mail/aliases has SHA256 hash 34452a4607ac1876abd921212e14a58cfa9a969d9da1b93527924a929f892275, but should have SHA256 hash b40dd2567efbdfe3a1853248802899f376b5fc59dde2734bbf0377b169af9b52.
/etc/mail/mailer.conf has SHA256 hash 58adc5945bd93e246b86484d71a892c77ee2a0fd567d83de263992f9ce5e9196, but should have SHA256 hash c6524597f51dfa19032dcc58d024254a52320703b4b79e56e4c23e57e52f7d71.
/etc/master.passwd has SHA256 hash f557ee7048c7fe20ecdb9d0b3f9e44fcb33d6bc0ed4d63a9934ced93cfd73dec, but should have SHA256 hash 0bbe227751c05a429587b9ec170fe31655e545ab5a655a6890c560c5e258f1d8.
/etc/motd has SHA256 hash 8203dbb9a8df4e2cd68d344ab9764b2189e6dc1effb43a0e5f254cd1bacb60a7, but should have SHA256 hash c76d9a02e764e77686f9bf4a9192311b6a0387a088cc414dd312ef6ba069ad7e.
/etc/ntp.conf has SHA256 hash e1d221779ac8818477387c5d3d02e4dbedb515d54e96ece1d8a84ddf23130eb4, but should have SHA256 hash d542201ce7f23cf163dcb21f2e4370a71e20b8b1f9052dbfbcca1fe38bd88543.
/etc/passwd has SHA256 hash 2673f9f000dbaa720b3e407636314570773a7ab987ea2dee9c7b05d2780bd514, but should have SHA256 hash f38246308e1ca2ec8963fd968cf74aaa17bbb1b8f6a41c6b64fd90b21428f221.
/etc/periodic/daily/310.accounting has 0755 permissions, but should have 0555 permissions.
/etc/periodic/monthly/200.accounting has 0755 permissions, but should have 0555 permissions.
/etc/periodic/security/security.functions has 0755 permissions, but should have 0444 permissions.
/etc/pwd.db has SHA256 hash 52f794c1cace3a771b35554e626fcccc118721fb977b987c6082e76cb34ba09e, but should have SHA256 hash 7399088e4b04f239d3afbff5fc4e554c523a2613c38aa233ed068de2f93a0177.
/etc/shells has SHA256 hash b254eca926ef82cbdac3e64e871062772e415943b8a5d1b765d908e73f5a9454, but should have SHA256 hash 52905f4f8ef9a5791000b58bae44937311db4f4e3ce40e2e2395df11e4647409.
/etc/spwd.db has SHA256 hash f70df628428f3a0f75f596a07e72ffc1fa0c26a21ca5010db50d90949e20dfa6, but should have SHA256 hash c6386bf197a0c385072f0e45fdcb267ad5e9abd71cdf865d1699a289b925adcc.
/etc/ssh/sshd_config has SHA256 hash 42769e3059e1854b18667283cca672ed80f8cf4ec0d0e9dc330fbbb80e242c72, but should have SHA256 hash 2179a5f0ca6ad19702f5643be244b04adf965c5bfa2e8c94c851f1a72c90ebf8.
/etc/sysctl.conf has SHA256 hash 77412a8f7be3fac2987d930c6b3a1aca76d3cfe97af310e2793579b3032f9280, but should have SHA256 hash ea854baa79fcaa25b843eb260067449d4533632ff2e6922c3f753d97e006bb44.
/etc/syslog.conf has SHA256 hash 43e9f22af8da9e263852fafeede9b072b773bfe727f6731521ea85a1413f551a, but should have SHA256 hash 52bcf1ed4e5cc68d946259353a9f9004f4ac22788b981a38e06e13d2ed03b73b.
/root/.cshrc has SHA256 hash 3e6203b75cdf31ea6877ddfd690e3e126577982a9f9fd69692e0d22c10134e24, but should have SHA256 hash fa6ca8c03acabab047418c40a61fd6b42f1ae2898ce2e354dd1cf33fc19b9253.
/root/.profile has SHA256 hash c997e558f0812a6b14811401651c4df0c90d7cbcfd2f384c9d62e108ec576bc5, but should have SHA256 hash 59d4d08c686122ea22cd4eb8f2c671c90f5e63480e46cdfa2cb876d383bff227.
/var/db/etcupdate/current/etc/defaults/bluetooth.device.conf has 0444 permissions, but should have 0644 permissions.
/var/db/etcupdate/current/etc/defaults/devfs.rules has 0444 permissions, but should have 0600 permissions.
/var/db/etcupdate/current/etc/defaults/periodic.conf has 0444 permissions, but should have 0644 permissions.
/var/db/etcupdate/current/etc/defaults/rc.conf has 0444 permissions, but should have 0644 permissions.
/var/db/etcupdate/current/etc/periodic/daily/310.accounting has 0755 permissions, but should have 0555 permissions.
/var/db/etcupdate/current/etc/periodic/monthly/200.accounting has 0755 permissions, but should have 0555 permissions.
/var/db/etcupdate/current/etc/periodic/security/security.functions has 0755 permissions, but should have 0444 permissions.
/var/db/ntp is owned by user id 0, but should be owned by user id 123.
/var/db/ntp is owned by group id 0, but should be owned by group id 123.
/var/db/ntp has 0700 permissions, but should have 0755 permissions.
```


----------



## D-FENS (Mar 23, 2019)

I doubt you can make a hard link to another file system.
As I wrote above, the clean solution is to set the mountpoint of bootpool to /boot.
Otherwise, the symlink just works..... until next upgrade


----------



## SirDice (Mar 25, 2019)

byrnejb said:


> Should this be a hard link instead


Directories cannot be hard linked.


----------



## D-FENS (Mar 26, 2019)

roccobaroccoSC said:


> I doubt you can make a hard link to another file system.
> As I wrote above, the clean solution is to set the mountpoint of bootpool to /boot.
> Otherwise, the symlink just works..... until next upgrade



Hi all! I have just tested my own proposal from above (mounting bootpool to /boot) and *IT DOES NOT WORK!*
Be aware that the bootloader actually searches for a specific path: bootpool:/boot/kernel/kernel and if it cannot find it, the machine can't boot.
So basically the structure /bootpool/boot and a symlink /boot to it is exactly how it is supposed to be according to the bootloader implementation.
This comes probably from the default values in /boot/defaults/loader.conf, where a lot of hard coded paths to "/boot" on the bootpool can be seen. Changing them is probably not a very good idea, because this would make the paths at load time and at runtime diverge - not good.

It is not possible to work with the mountpoints because the dataset needs to have a directory "/boot" and we could not mount the bootpool in "/".
The two valid options seem to be:
- either install /boot directly in your zroot pool without a separate bootpool;
- use bootpool, but then you need the symlink /boot -> /bootpool/boot in the root directory.

And last remark: The 12.0-RELEASE upgrade seems to mess up the symlink in the default situation and needs to be patched manually. I am not aware if a bug report is posted, for me the issue was quite minor so I did not bother.


----------



## Sebastian (Apr 11, 2020)

I have updated from 12.0 to 12.1 an came a cross the same problem. I know I had this issue in the past , but symlinking boot to boot pool/boot works but.... should't we create a bug ?


----------



## D-FENS (Apr 19, 2020)

Technically you're right but I don't have the time to do it. Now we have documented it and it's at least a half-way solution.


----------



## Emrion (May 1, 2021)

Sorry to resurrect this old thread, but I just had an interesting journey in MBR scheme with zfs-on-root.

If you want zfs-on-root with MBR, you get something like:

```
gpart show
=>      63  33554369  ada0  MBR  (16G)
        63         1        - free -  (512B)
        64  33554368     1  freebsd  [active]  (16G)

=>       0  33554368  ada0s1  BSD  (16G)
         0   4194304       1  freebsd-zfs  (2.0G)
   4194304   4194304       2  freebsd-swap  (2.0G)
   8388608  25165760       4  freebsd-zfs  (12G)
```
ada0s1a contains the zfs pool named "bootpool". Inside, you find the content you normally have in /boot.
ada0s1d is the working zpool (zroot by default).

Well, MBR has been working badly with zfs since some times...

If you try to install zfs-on-root + MBR from a 13.0-RELEASE disk, you get at reboot: "Missing operating system".
Seems that the install cannot create partitions inside the BSD slice (as a result, there is no boot stage at this place and precisely no magic number (0xaa55) at the end of the first sector of the BSD slice). Bug it is, that's for sure.

FreeBSD 12.2-RELEASE installation is working on this matter, but `devmatch` complains that it can't find the file linker.hints. At this point, the ada0s1a partition isn't mounted when you reach the multiuser command line. You have a symlink /boot -> /bootpool/boot but the zpool bootpool isn't imported and the dataset bootpool inside isn't mounted. Bug it is also, isn't it?

I upgraded that to 13.0-RELEASE. After kernel upgrade, reboot and typed `freebsd-update install`, it complained that it can't identify the running kernel. Well, bug it is...
`zpool import bootpool`
And then, you can finish the upgrade normaly.

So, what the learning about all of this?

*Give up MBR scheme if you want zfs on root. ufs is still working with MBR, but for how long? 
Give up MBR scheme, at least concerning FreeBSD.*


----------

