# Nothing in /boot; nothing in /boot/kernel; system boots regardless



## byrnejb (Oct 17, 2022)

I upgraded a spare host from FreeBSD-12.1 to FreeBSD-13.1p2.   During this update I encountered a problem caused by /boot being a soft link to /bootpool/boot.  By removing  /boot and then adding it back via `mkdir` I was able to complete the upgrade; or so I thought.  I rebooted after the install, ran `freebsd-update install` again and rebooted.

What I have now is a system that boots and says it is _13.1-RELEASE-p2_ but which has 0 contents in /boot. I cannot find the kernel nor any of the kernel modules using `find`.  I am not really sure what to do at this point.  I am not keen on reinstalling FreeBSD and wiping the entire zpool as this was/is a vm host and there are several vm's that, while not running, I do need to retrieve records from.

My present concern is to discover how this happened.   Once that is accomplished then I need to know how to correct this issue.


----------



## Alain De Vos (Oct 17, 2022)

Whas /bootpool a mounted filesystem.
E.g. part of /etc/fstab ?
Is the root filesystem / mounted ?
What is the output of:

```
mount -v /
```


----------



## richardtoohey2 (Oct 17, 2022)

Anything in /var/log/messages e.g.

syslogd: kernel boot file is /boot/kernel/kernel

(This is from a UFS system so not sure if any difference with ZFS)


----------



## Alain De Vos (Oct 17, 2022)

The output of:

```
kenv | egrep "vfs|currdev|kernel_path|kernelname"
```


----------



## W.hâ/t (Oct 17, 2022)

You're probably boot from /bootpool/boot. You should recreate the link to /boot



byrnejb said:


> My present concern is to discover how this happened.





byrnejb said:


> By removing /boot and then adding it back via `mkdir`



I don't use this setup so I can't tell you if there are extra steps to make an upgrade. Othewise it might just be an issue with 'freebsd-update' and symlink to /boot


----------



## byrnejb (Oct 18, 2022)

Alain De Vos said:


> mount -v /


# mount -v /
mount: /: unknown special file or file system


----------



## byrnejb (Oct 18, 2022)

richardtoohey2 said:


> Anything in /var/log/messages e.g.
> 
> syslogd: kernel boot file is /boot/kernel/kernel
> 
> (This is from a UFS system so not sure if any difference with ZFS)




```
grep boot /var/log/messages
Oct 17 15:49:00 vhost03 syslogd: kernel boot file is /boot/kernel/kernel
```


```
# ll /boot/kernel/kernel
ls: /boot/kernel/kernel: No such file or directory
```


----------



## monwarez (Oct 18, 2022)

Does the bootpool still exist ? If so then it means that you are booting from it and then the other pool is mounted via vfs.root.mountfrom
And after the kernel is loaded it does not need to access to the kernel file, so having the /boot is not really necessary to have it to boot.


----------



## byrnejb (Oct 18, 2022)

monwarez said:


> Does the bootpool still exist ? If so then it means that you are booting from it and then the other pool is mounted via vfs.root.mountfrom
> And after the kernel is loaded it does not need to access to the kernel file, so having the /boot is not really necessary to have it to boot.




```
# tree /bootpool
/bootpool

0 directories, 0 files

# tree /boot
/boot
├── entropy
└── kernel

# lsvfs
Filesystem                              Num  Refs  Flags
-------------------------------- ---------- -----  ---------------
devfs                            0x00000071     1  synthetic, jail
cd9660                           0x000000bd     0  read-only
nfs                              0x0000003a     0  network
procfs                           0x00000002     0  synthetic, jail
tmpfs                            0x00000087     0  jail
ufs                              0x00000035     0
msdosfs                          0x00000032     0
zfs                              0x000000de   406  jail, delegated-administration


# cat /etc/fstab
# Device        Mountpoint    FStype    Options        Dump    Pass#
/dev/ada0p3.eli        none    swap    sw        0    0
/dev/ada1p3.eli        none    swap    sw        0    0
/dev/ada2p3.eli        none    swap    sw        0    0
/dev/ada3p3.eli        none    swap    sw        0    0
fdesc            /dev/fd    fdescfs    rw,late,failok    0    0
```


----------



## SirDice (Oct 18, 2022)

`zfs list`


----------



## byrnejb (Oct 18, 2022)

SirDice said:


> `zfs list`




```
# zfs list
NAME                            USED  AVAIL     REFER  MOUNTPOINT
zroot                          1.31T  3.71T      140K  /zroot
zroot/ROOT                     7.36G  3.71T      140K  none
zroot/ROOT/13.1-RELEASE-p2-1   11.6K  3.71T     4.35G  /
zroot/ROOT/_2022-10-17_154256  11.6K  3.71T     4.34G  /
zroot/ROOT/default             7.36G  3.71T     4.34G  /
zroot/tmp                      9.00M  3.71T      232K  /tmp
zroot/usr                       901M  3.71T      140K  /usr
zroot/usr/home                  140K  3.71T      140K  /usr/home
zroot/usr/ports                 900M  3.71T      900M  /usr/ports
zroot/usr/src                   244K  3.71T      140K  /usr/src
zroot/var                      3.46G  3.71T      140K  /var
zroot/var/audit                 244K  3.71T      140K  /var/audit
zroot/var/crash                 256K  3.71T      140K  /var/crash
zroot/var/log                  83.4M  3.71T     27.4M  /var/log
zroot/var/mail                  465K  3.71T      174K  /var/mail
zroot/var/tmp                  3.38G  3.71T     3.38G  /var/tmp
zroot/vm                       1.29T  3.71T     13.3G  /zroot/vm
zroot/vm/inet13                 235G  3.71T      157K  /zroot/vm/inet13
zroot/vm/inet13/disk0           235G  3.71T      235G  -
zroot/vm/inet14                 111G  3.71T      145K  /zroot/vm/inet14
zroot/vm/inet14/disk0           111G  3.71T      111G  -
zroot/vm/inet16                67.8G  3.71T      157K  /zroot/vm/inet16
zroot/vm/inet16/disk0          67.8G  3.71T     67.8G  -
zroot/vm/inet17                 389G  3.71T      163K  /zroot/vm/inet17
zroot/vm/inet17/disk0           389G  3.71T      389G  -
zroot/vm/inet18                 114G  3.71T      145K  /zroot/vm/inet18
zroot/vm/inet18/disk0           114G  3.71T      114G  -
zroot/vm/inet19                 251G  3.71T      157K  /zroot/vm/inet19
zroot/vm/inet19/disk0           251G  3.71T      251G  -
zroot/vm/samba-01               134G  3.71T      157K  /zroot/vm/samba-01
zroot/vm/samba-01/disk0         134G  3.71T      128G  -
zroot/vm/samba-03              2.80G  3.71T      140K  /zroot/vm/samba-03
zroot/vm/samba-03/disk0        2.80G  3.71T     2.80G  -
zroot/vm/samba-04              3.58G  3.71T      140K  /zroot/vm/samba-04
zroot/vm/samba-04/disk0        3.58G  3.71T     3.58G  -
```


----------



## monwarez (Oct 18, 2022)

Are you booting from uefi or legacy bios ?
What does `gpart show` give you ?


----------



## SirDice (Oct 18, 2022)

As this is an encrypted setup, you should also have a bootpool. This contains the _unencrypted_ /boot filesystem. You need this or else you won't be able to boot the machine. It needs unencrypted access in order to load the kernel, once the kernel is active the *.eli drives can be unlocked, and the system continues to boot from the filesystems on zroot. This is why you had a symlink from /boot to /bootpool/boot. /bootpool/boot is _unencrypted_ and used to boot the system, it resides on the bootpool zpool and is separate from the zroot zpool.


----------



## byrnejb (Oct 18, 2022)

The issue with the upgrade was that `freebsd-update` checks to see if `kern.bootfile` is a directory:


```
737          # Figure out what directory contains the running kernel
   738          BOOTFILE=`sysctl -n kern.bootfile`
   739          KERNELDIR=${BOOTFILE%/kernel}
   740          if ! [ -d ${KERNELDIR} ]; then
   741                  echo "Cannot identify running kernel"
   742                  exit 1
   743          fi
```

The upgrade to 13.1 would not proceed unless this was true.  Thus I made the changes to /boot.  I made no changes to /bootpool. It was empty when I first checked it and it remains so.

I can easily rename /boot and create a softlink to /bootpool with that name.  However, that will not solve the problem with `freebsd-update`.  How do I proceed?


----------



## SirDice (Oct 18, 2022)

byrnejb said:


> I made no changes to /bootpool. It was empty when I first checked it and it remains so.


It simply wasn't mounted.


----------



## W.hâ/t (Oct 18, 2022)

```
$ ln -s /boot boot
$ [ -d boot ]
$ echo $?
0
```


----------



## byrnejb (Oct 18, 2022)

Is it correct to infer that IF I link /boot to /bootpool and then restart that the system will come up correctly with FILE] /bootpool[/FILE] mounted?


----------



## SirDice (Oct 18, 2022)

byrnejb said:


> with /bootpool mounted?


No, it's probably not automatically mounted. Which is why it was empty in the first place. Check what your settings are: `zfs list -o name,canmount,mountpoint bootpool`


----------



## byrnejb (Oct 18, 2022)

SirDice said:


> No, it's probably not automatically mounted. Which is why it was empty in the first place. Check what your settings are: `zfs list -o name,canmount,mountpoint bootpool`




```
# zfs list -o name,canmount,mountpoint bootpool
cannot open 'bootpool': dataset does not exis
```


----------



## Alain De Vos (Oct 18, 2022)

You have a running kernel without a mounted root filesystem.

Normally one of these 3:

```
zroot/ROOT/13.1-RELEASE-p2-1   11.6K  3.71T     4.35G  /
zroot/ROOT/_2022-10-17_154256  11.6K  3.71T     4.34G  /
zroot/ROOT/default             7.36G  3.71T     4.34G  /
```
should be mounted via legacy zfs mount.


Can you share the output of:

```
kenv | egrep "vfs|currdev|kernel_path|kernelname"
```

And of

```
zpool get bootfs
```


----------



## byrnejb (Oct 18, 2022)

```
# kenv | egrep "vfs|currdev|kernel_path|kernelname"
currdev="zfs:bootpool:"
kernel_path="/boot/kernel"
kernelname="/boot/kernel/kernel"
vfs.root.mountfrom="zfs:zroot/ROOT/default"

# zpool get bootfs
NAME   PROPERTY  VALUE               SOURCE
zroot  bootfs    zroot/ROOT/default  local
```


----------



## Alain De Vos (Oct 18, 2022)

That looks ok. You boot from the bootpool-zpool and normally this should mount the zroot-zpool.
The output of `mount` & `zfs mount`


----------



## byrnejb (Oct 18, 2022)

```
# mount
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/vm on /zroot/vm (zfs, local, noatime, nfsv4acls)
zroot/vm/samba-03 on /zroot/vm/samba-03 (zfs, local, noatime, nfsv4acls)
zroot/vm/samba-04 on /zroot/vm/samba-04 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet17 on /zroot/vm/inet17 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet13 on /zroot/vm/inet13 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet18 on /zroot/vm/inet18 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet16 on /zroot/vm/inet16 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet14 on /zroot/vm/inet14 (zfs, local, noatime, nfsv4acls)
zroot/vm/samba-01 on /zroot/vm/samba-01 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet19 on /zroot/vm/inet19 (zfs, local, noatime, nfsv4acls)
```


```
# zfs mount
zroot/ROOT/default              /
zroot/var/mail                  /var/mail
zroot/tmp                       /tmp
zroot/var/log                   /var/log
zroot/var/audit                 /var/audit
zroot/usr/ports                 /usr/ports
zroot/var/crash                 /var/crash
zroot/usr/src                   /usr/src
zroot/usr/home                  /usr/home
zroot                           /zroot
zroot/var/tmp                   /var/tmp
zroot/vm                        /zroot/vm
zroot/vm/samba-03               /zroot/vm/samba-03
zroot/vm/samba-04               /zroot/vm/samba-04
zroot/vm/inet17                 /zroot/vm/inet17
zroot/vm/inet13                 /zroot/vm/inet13
zroot/vm/inet18                 /zroot/vm/inet18
zroot/vm/inet16                 /zroot/vm/inet16
zroot/vm/inet14                 /zroot/vm/inet14
zroot/vm/samba-01               /zroot/vm/samba-01
zroot/vm/inet19                 /zroot/vm/inet19
```


----------



## W.hâ/t (Oct 18, 2022)

```
zpool import -f bootpool
```


----------



## byrnejb (Oct 19, 2022)

Importing/bootpool  worked.  However I am still encountering a problem with `freebsd-update`.


```
# freebsd-version
13.1-RELEASE-p2

# uname -a
FreeBSD vhost03.hamilton.harte-lyne.ca 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 GENERIC amd64

# freebsd-update fetch
src component not installed, skipped
Cannot identify running kernel
```

As I wrote above this issue appears to be caused by this code in /usr/sbin/freebsd-update:


```
737          # Figure out what directory contains the running kernel
   738          BOOTFILE=`sysctl -n kern.bootfile`
   739          KERNELDIR=${BOOTFILE%/kernel}
   740          if ! [ -d ${KERNELDIR} ]; then
   741                  echo "Cannot identify running kernel"
   742                  exit 1
   743          fi
```

`sysctl -n kern.bootfile` returns /boot/kernel/kernel which does not exist:


```
# ls -l "$(sysctl -n kern.bootfile)"
ls: /boot/kernel/kernel: No such file or directory
```


```
# ls -l /boot
lrwxr-xr-x  1 root  wheel  9 Oct 18 15:44 /boot -> /bootpool
```

However
# ls -ld /boot/boot
drwxr-xr-x  15 root  wheel  73 Sep 23 09:47 /boot/boot
[/code]

My question is what should /boot point to:  /bootpool  or /bootpool/boot


----------



## Alain De Vos (Oct 19, 2022)

byrnejb said:


> ```
> # kenv | egrep "vfs|currdev|kernel_path|kernelname"
> currdev="zfs:bootpool:"
> kernel_path="/boot/kernel"
> ...



/boot/kernel/kernel exists because you booted from it see the kenv environment variable above.
Maybe you "overmounted".


----------



## W.hâ/t (Oct 19, 2022)

```
grep -A1 'LN_SF.*bootpool' /usr/libexec/bsdinstall/zfsboot
```


----------



## byrnejb (Oct 19, 2022)

```
# grep -A1 'LN_SF.*bootpool' /usr/libexec/bsdinstall/zfsboot
        f_eval_catch $funcname ln "$LN_SF" "$bootpool_name/boot" \
                     $BSDINSTALL_CHROOT/boot || return $FAILURE
```


----------

