# help with zfs layout/options with boot environments



## jrm@ (Apr 22, 2014)

When I create a new boot environment, I would like it to have its own version of all filesystems (except perhaps /tmp and /var/tmp).   In other words, if I create a new boot environment, boot into it and make changes in /etc/, /usr/local/etc/, or ~/, then boot back into the original boot environment, I don't want to see those changes.  I'm unclear how to set the layout and the canmount options.  Below is what I have now.  I've tried turning canmount off, for example, for tank/usr/home/jrm, but then I don't see those files.  Can you either point me to documentation that describes this or set me straight? Thanks.


```
% zfs list -o name,mountpoint,canmount
NAME                      MOUNTPOINT            CANMOUNT
tank                      none                        on
tank/ROOT                 none                        on
tank/ROOT/default         legacy                      on
tank/tmp                  /tmp                        on
tank/usr                  /usr                       off
tank/usr/home             /usr/home                   on
tank/usr/home/jrm         /usr/home/jrm               on
tank/usr/jails            /usr/jails                  on
tank/usr/local            /usr/local                  on
tank/usr/obj              /usr/obj                    on
tank/usr/ports            /usr/ports                  on
tank/usr/ports/distfiles  /usr/ports/distfiles        on
tank/usr/src              /usr/src                    on
tank/var                  /var                       off
tank/var/audit            /var/audit                  on
tank/var/log              /var/log                    on
tank/var/tmp              /var/tmp                    on
```


----------



## usdmatt (Apr 23, 2014)

Lets take /usr/local as an example.

At the moment, there is a ZFS file system (tank/usr/local) that is mounted on this path. Any files under /usr/local will be sat on this ZFS file system. If you unmount tank/usr/local, they will disappear.

What you can do is create a temporary file system to allow you to move this data off, e.g

`zfs create -o mountpoint=/move tank/move`

You will then need to do the following to move the data back onto your root filesystem:

* Copy the entire contents of /usr/local to /move
* Unmount the tank/usr/home/ dataset (set canmount to no then delete it later when everything is working)
* Copy the files from /move, into /usr/local, which will now be a folder on your root filesystem.

Edit: alternatively you can just change the mountpoint of the file systems so you only have to copy once:


```
zfs set mountpoint=/move tank/usr/local
...copy contents of /move to /usr/local
zfs set canmount=no tank/usr/local
...delete tank/usr/local once everything is up and running
```

It may be best to do this in single user mode. Some of the file systems (especially /var) will be problematic if you try and move them on a running system.


----------



## asteriskRoss (Apr 23, 2014)

There may be an easier way than @usdmatt suggests.  I'm guessing you're using sysutils/beadm, which uses the name of the ZFS dataset when creating an environment, cloning everything under (and including) tank/ROOT/default, your only current boot environment.  Shout if I've misunderstood.

Like @usdmatt, I'll use /usr/local as an example.  You need the ZFS dataset that contains /usr/local to be under the tank/ROOT/default hierarchy so that it is cloned when creating a new boot environment.  You just need to rename it to move it there with `zfs rename tank/usr/local tank/ROOT/default/local`.  You can do the same for your other datasets.  Note that the structure and naming convention of your ZFS datasets does not need to have any relation to the file system hierarchy.  Anything under tank/ROOT will be unique to each boot environment whilst anything directly under tank, such as tank/tmp will be shared between boot environments.

You should also stop using /etc/fstab to mount /.  Edit it to remove the relevant line, then set the mount point on the ZFS dataset with `zfs set mountpoint=/ tank/ROOT/default`.  For neatness you could also set

```
canmount=off
```
 on your placeholder datasets tank, tank/ROOT.

If you're using sysutils/beadm, it will set the canmount property appropriately on your behalf.  Essentially it recursively sets 
	
	



```
# Edit: Changed from "canmount=off", which is functionally similar but technically incorrect 
canmount=noauto
```
 on tank/ROOT/<disabled boot environment> and recursively sets 
	
	



```
canmount=on
```
 on tank/ROOT/<enabled boot environment>.


----------



## jrm@ (Apr 23, 2014)

@asteriskRoss, your assumptions are correct.  To summarize, anything under ${bootpool}/ROOT/default will be in the boot environment snapshot.  I think I was confused after reading Kris Moores's article in the Proceedings of AsiaBSDCon (also in BSDMag) where he says he wants /var to be included in his boot environment, but then he lists it as tank0/var.   It sounds like he should have created tank0/ROOT/default/var.

Thank @asteriskRoss and @usdmatt for taking the time to respond.  I think that clarifies things.

ADDED:


			
				asteriskRoss said:
			
		

> You should also stop using /etc/fstab to mount /.  Edit it to remove the relevant line, then set the mount point on the ZFS dataset with `zfs set mountpoint=/ tank/ROOT/default`.  For neatness you could also set canmount=off on your placeholder datasets tank, tank/ROOT.


I'm not using /etc/fstab to mount anything.  The legacy value for canmount apparently has no affect in this situation.  This is what @vermaden wrote in his HOWTO: FreeBSD ZFS Madness thread,


			
				vermaden said:
			
		

> Generally the option 'legacy' is just the option that I used for the first time for Boot Environments to distinguish the Boot Environment from other ZFS datasets, it has no other function.


----------



## asteriskRoss (Apr 23, 2014)

@jrm, yes, I agree; if you want /var to be cloned in your boot environment and you are using sysutils/beadm, it should be in the tank0/ROOT/default/var dataset. I've not read the article you mention -- can you post a link?  If you've not already seen it, @vermaden's thread on ZFS and sysutils/beadm is worth a read: Thread 31662.  In practice for both server and desktop, I've found I usually only have two boot environments ("master" which is the normal boot environment and "upgrade" which is essentially an environment for testing base and port updates) which it's perfectly feasible to manage by hand.

I also meant to mention that if you're shuffling around ZFS datasets and changing the hierarchy, it can be useful to reset some property values so that they revert to inheriting the value from their new parent.  From the zfs(8) man page:


> zfs inherit [-rS] property	filesystem|volume|snapshot...
> 
> Clears	the specified property,	causing	it to be inherited from	an
> ancestor. If no ancestor has the property set,	then the default value
> ...


----------



## jrm@ (Apr 23, 2014)

Thanks again @asteriskRoss.  Here's the link to the 2014 AsiaBSDCon Proceedings and here is a link to the BSD Magazine Issue with the Boot Environments article.  After `zfs rename tank/usr tank/ROOT/default/usr` and `zfs rename tank/var tank/ROOT/default/var` I have this layout:

```
% zfs list -o name,mountpoint,canmount
NAME                                   MOUNTPOINT            CANMOUNT
tank                                   none                       off
tank/ROOT                              none                       off
tank/ROOT/default                      legacy                      on
tank/ROOT/default/usr                  /usr                        on
tank/ROOT/default/usr/home             /usr/home                   on
tank/ROOT/default/usr/home/jrm         /usr/home/jrm               on
tank/ROOT/default/usr/jails            /usr/jails                  on
tank/ROOT/default/usr/local            /usr/local                  on
tank/ROOT/default/usr/obj              /usr/obj                    on
tank/ROOT/default/usr/ports            /usr/ports                  on
tank/ROOT/default/usr/ports/distfiles  /usr/ports/distfiles        on
tank/ROOT/default/usr/src              /usr/src                    on
tank/ROOT/default/var                  /var                        on
tank/ROOT/default/var/audit            /var/audit                  on
tank/ROOT/default/var/log              /var/log                    on
tank/ROOT/default/var/tmp              /var/tmp                    on
tank/tmp                               /tmp                        on
```
Kris also excluded /usr/ports, /usr/ports/distfiles, /usr/src and a few other datasets from the boot environments.  This seems like a good idea.  However, if I try to rename tank/ROOT/default/usr/ports there is an error about the parent dataset not existing.  I'm guessing this is where the dataset with canmount=off comes in.


----------



## asteriskRoss (Apr 23, 2014)

Thanks for the links, @jrm.  Regarding your error, I suspect the reason you are unable to rename tank0/ROOT/default/usr/ports is because it has a child dataset; tank0/ROOT/default/usr/ports/distfiles.  You need to rename this one first.  There is no need to preserve the filesystem hierarchy (or even names) in your ZFS hierarchy, though your method has the convenience of inheritance of the mountpoint property.  In my previous post, you may have noticed I suggested renaming the dataset for /usr/local to tank0/ROOT/default/local rather than tank0/ROOT/default/*usr*/local -- this was not an error.

/usr/ports/distfiles is an excellent candidate for sharing between boot environments since you won't have to download the same files multiple times (providing you configure your port management tool to retain them).  I choose to include /usr/ports in my boot environment, since this allows different versions of the ports and different configurations for each environment.

To move the dataset containing /usr/ports/distfiles:

```
# zfs rename tank/ROOT/default/usr/ports/distfiles tank/distfiles
# zfs set mountpoint=/usr/ports/distfiles tank/distfiles
```

If you then wanted to rename tank/ROOT/default/usr/ports, you should be able to do so, though you will need to set the mount point for that also since it will no longer be inheriting correctly from tank/ROOT/default/usr.  I hope that all makes sense.  ZFS's flexibility is so awesome that I sometimes find myself getting carried away...


----------



## jrm@ (Apr 23, 2014)

Yes @asteriskRoss, it's making much more sense now.  For anyone reading this thread later, I did have to set canmount=off for some datasets, otherwise a child dataset would cause problems.  Here's an example to demonstrate.   After booting to single user mode:

```
% which which
/usr/bin/which
% zfs mount tank/ROOT/default/usr
which which
which: not found
```
So here is my layout now.

```
% zfs list -o name,mountpoint,canmount
NAME                                      MOUNTPOINT            CANMOUNT
tank                                      none                        on
tank/ROOT                                 none                        on
tank/ROOT/default                         legacy                      on
tank/ROOT/default/usr                     /usr                       off
tank/ROOT/default/usr/home                /usr/home                  off
tank/ROOT/default/usr/home/jrm            /usr/home/jrm               on
tank/ROOT/default/usr/jails               /usr/jails                  on
tank/ROOT/default/usr/local               /usr/local                  on
tank/ROOT/default/usr/obj                 /usr/obj                    on
tank/ROOT/default/usr/ports               /usr/ports                 off
tank/ROOT/default/usr/src                 /usr/src                    on
tank/ROOT/default/var                     /var                       off
tank/ROOT/default/var/audit               /var/audit                  on
tank/ROOT/default/var/log                 /var/log                    on
tank/distfiles                            /usr/ports/distfiles        on
tank/tmp                                  /tmp                        on
tank/var_tmp                              /var/tmp                    on
```
I'm starting to appreciate ZFS's flexibility and power more and more.


----------



## jrm@ (Apr 24, 2014)

The problem I described in the last post simply had to do with files being located in the wrong dataset.  For example, files under /usr/ were stored in the tank/ROOT/default dataset instead of the tank/ROOT/default/usr dataset.  After moving files around things are making more sense.  Thanks again @asteriskRoss.  This has been enlightening.


----------

