# zfs zroot diskspace puzzle



## cbrace (Dec 5, 2021)

Hi all,
I have a simple LAN file server running v12.2. The OS is installed on a 120GB SSD using the ZFS defaults:

```
$ sudo gpart show /dev/ada4
=>       40  234441568  ada4  GPT  (112G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  freebsd-swap  (2.0G)
    4196352  230244352     3  freebsd-zfs  (110G)
  234440704        904        - free -  (452K)
```
I would like to upgrade this to v12.3 shortly. But I seem to have run out of diskspace; bectl cannot create a boot image. In the info below, I see that zroot/ROOT appears to be using 96.5G, but I cannot determine exactly how this space is being used, and how to free it up. I have some earlier BE's still online, which I should probably delete, but these snapshots don't appear to be using that much space, maybe around 7G.

Any ideas for figuring out what is taking up so much space on my zroot/ROOT?

```
NAME                          USED  AVAIL  REFER  MOUNTPOINT
zroot                         100G  5.40G    88K  /zroot
zroot/ROOT                   96.5G  5.40G    88K  none
zroot/ROOT/12.1-RELEASE       212K  10.0G  1.63G  /
zroot/ROOT/12.1-RELEASE-p1    720K  10.0G  3.16G  /
zroot/ROOT/12.1-RELEASE-p10   864K  10.0G  3.56G  /
zroot/ROOT/12.1-RELEASE-p9    684K  10.0G  3.20G  /
zroot/ROOT/12.2-RELEASE       852K  10.0G  4.45G  /
zroot/ROOT/12.2-RELEASE-p1    624K  10.0G  4.44G  /
zroot/ROOT/12.2-RELEASE-p10  12.5G  5.40G  5.38G  /
zroot/ROOT/12.2-RELEASE-p2    980K  12.4G  4.57G  /
zroot/ROOT/12.2-RELEASE-p3    804K  12.4G  4.63G  /
zroot/ROOT/12.2-RELEASE-p4    464K  12.4G  4.71G  /
zroot/ROOT/12.2-RELEASE-p5    568K  12.4G  4.74G  /
zroot/ROOT/12.2-RELEASE-p6    760K  12.4G  4.94G  /
zroot/ROOT/12.2-RELEASE-p7    764K  12.4G  5.18G  /
zroot/tmp                    11.5M   489M  11.5M  /tmp
zroot/usr                    3.58G  5.40G    88K  /usr
zroot/usr/home               3.58G  5.40G  3.58G  /usr/home
zroot/var                    13.9M  5.40G    88K  /var
zroot/var/log                13.7M  36.3M  13.7M  /var/log
zroot/var/mail                104K   920K   104K  /var/mail
```


```
$ zfs list -o space -t all -S usedds | head
NAME                                                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zroot/ROOT/12.2-RELEASE-p10                                5.40G  12.5G     7.16G   5.38G              0          0
zroot/usr/home                                             5.40G  3.58G         0   3.58G              0          0
zroot/var/log                                              36.3M  13.7M         0   13.7M              0          0
zroot/tmp                                                   489M  11.5M         0   11.5M              0          0
zroot/ROOT/12.2-RELEASE-p2                                 12.4G   980K         0    980K              0          0
zroot/ROOT/12.1-RELEASE-p10                                10.0G   864K         0    864K              0          0
zroot/ROOT/12.2-RELEASE                                    10.0G   852K         0    852K              0          0
zroot/ROOT/12.2-RELEASE-p3                                 12.4G   804K         0    804K              0          0
```


----------



## Alain De Vos (Dec 5, 2021)

You could have a look:

```
mkdir /test
mount -t zfs zroot/ROOT /test
du -hs /test/*
```


----------



## Geezer (Dec 6, 2021)

What are all these?


> ```
> zroot/ROOT/12.1-RELEASE       212K  10.0G  1.63G  /
> zroot/ROOT/12.1-RELEASE-p1    720K  10.0G  3.16G  /
> zroot/ROOT/12.1-RELEASE-p10   864K  10.0G  3.56G  /
> ...



All on the same mountpoint, and each using up a few G.


----------



## Eric A. Borisch (Dec 6, 2021)

cbrace said:


> ```
> $ zfs list -o space -t all -S usedds | head
> NAME                                                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
> zroot/ROOT/12.2-RELEASE-p10                                5.40G  12.5G     7.16G   5.38G              0          0
> ...



How about the full output from `zfs list -ro space zroot/ROOT` ?


----------



## grahamperrin@ (Dec 18, 2021)

Geezer said:


> What are all these? …



Boot environments (BEs).



cbrace said:


> … earlier BE's still online, which I should probably delete, but these snapshots don't appear to be using that much space, maybe around 7G. 𠉧…



`bectl list -s -c creation`
`zpool list`


----------



## Andriy (Dec 19, 2021)

cbrace said:


> zfs list -o space -t all -S usedds


Why don't you share the full output of this command or zfs list -o space -t all -S used ?


----------



## mer (Dec 19, 2021)

I would start off with deleting all of the BEs except for the one in use and maybe the one previous.


----------



## grahamperrin@ (Dec 19, 2021)

cbrace said:


> … don't appear to be using that much space, maybe around 7G. …





mer said:


> I would start off with deleting all of the BEs except for the one in use and maybe the one previous.



It's not yet clear how much free space is expected now, or after taking action, which is partly why I'd like to see output from `zpool list`.


----------



## mer (Dec 19, 2021)

Regardless of how it looks, removing boot environments will clean things up and make it easier to figure out where the space is actually being used.
Why do I say this?  Boot Environments are clones (writable snapshots basically) and the space allocation gets a little weird to understand because of the COW behavior.

I'm not arguing that full output is not needed to help the OP,  obviously we need to see it all.  A listing of snapshots from the top level would also be useful because perhaps the OP has some snapshots he's forgotten about somewhere.


----------

