# etcupdate has nothing updated?!



## HL1234 (Jun 1, 2021)

`uname -v`
FreeBSD 13.0-RELEASE #0 releng/13.0-ea31abc26: Sun May 16 14:57:18 CEST 2021

I have read etcupdate() carefully. I understand there are different modes. Because I never used it before I have done a bootstarp, but not like it is written in 24.5.6.1. Merging Configuration Files with etcupdate(8) with "etcupdate bootstrap" but with `etcupdate extract` like it is written in the manual. I done this after `make installworld`, because before I hasn't known. It has created the content of the directory /var/log/etcupdate/current After learning the manual I do today:
`script etcupdate.log etcupdate` the output after some time waiting is that:

```
Script started on Tue Jun  1 17:21:59 2021
Command: etcupdate

Command exit status: 0
Script done on Tue Jun  1 17:22:51 2021
```
 It was written that there will be an output with

```
For each file that is updated a line will be output with a leading character 
to indicate the action taken.  The possible actions follow:
       A  Added
       C  Conflict
       D  Deleted
       M  Merged
       U  Updated
```
 but nothing happens . No output.

Also it was written:

```
Default Mode
     The default mode merges changes from the source tree to the destination
     directory.     First,    it updates the "current" and "previous"    trees.    Next,
     it    compares the two trees merging changes into the    destination directory.
     Finally, it displays warnings for any conditions it could not handle automatically....
```
 But there is no new folder ./previous. 

The "best" are in the new log file (I formate it a bit):

```
>>> update command: rerun= tarball= preworld=
rmdir: /var/db/etcupdate/conflicts: No such file or directory
>>> Building tree at /var/db/etcupdate/current with make  -DNO_FILEMON
cd /usr/src/etc; MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE=native
CC="clang -target x86_64-unknown-freebsd13.0
--sysroot=/usr/obj/usr/src/amd64.amd64/tmp
-B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"
CXX="clang++  -target x86_64-unknown-freebsd13.0
--sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"  CPP="clang-cpp -target x86_64-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"  AS="as" AR="ar" LD="ld" LLVM_LINK=""  NM=nm OBJCOPY="objcopy"  RANLIB=ranlib STRINGS=  SIZE="size" STRIPBIN="strip" PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin make  INSTALL="install -N /usr/src/etc" MTREE_CMD="mtree -N /usr/src/etc" METALOG= distrib-dirs

for file in /usr/share/doc/usd/10.exref
               /usr/share/doc/usd/11.edit
               /usr/share/doc/usd/12.vi
               /usr/share/doc/usd/13.viref; do 
    if [ -f /var/db/etcupdate/current/${file} ]; then 
    rm -f /var/db/etcupdate/current/${file};  fi; 
done
mtree -N /usr/src/etc -deU -i -f /usr/src/etc/mtree/BSD.root.dist -p /var/db/etcupdate/current/
./bin missing (created)
./boot missing (created)
./boot/defaults missing (created)
./boot/dtb missing (created)
./boot/dtb/allwinner missing (created)
./boot/dtb/overlays missing (created)
./boot/dtb/rockchip missing (created)
./boot/efi missing (created)
./boot/firmware missing (created)
./boot/loader.conf.d missing (created)
./boot/lua missing (created)
./boot/kernel missing (created)
./boot/modules missing (created)
./boot/uboot missing (created)
./boot/zfs missing (created)
etc..
```
I look for the files:

```
#ll /usr/share/doc/usd/10.exref
gnuls: cannot access '/usr/share/doc/usd/10.exref': No such file or directory
#ll /usr/share/doc/usd/11.edit
gnuls: cannot access '/usr/share/doc/usd/11.edit': No such file or directory
#ll /usr/share/doc/usd/12.vi
gnuls: cannot access '/usr/share/doc/usd/12.vi': No such file or directory
#ll /usr/share/doc/usd/13.viref
gnuls: cannot access '/usr/share/doc/usd/13.viref': No such file or directory
```
I look to the "created" files in ./boot how the log described. There are three directories
`ll /var/db/etcupdate/current/boot` 
-r--r--r-- 1 root wheel 754 Juni  1 17:22 device.hints
`ll /var/db/etcupdate/old/boot` 
-r--r--r-- 1 root wheel 754 Mai  29 20:48 device.hints
`ll /var/db/etcupdate/old/conflicts`
--empty - I can not find the created files as it is written in the /var/db/etcupdate/log file.

And there are files named added.files both.files new.files old.files removed.files that are not explained in the manual. From the names of the files I can figured out for what they stand for, but I don't know what to do with them. 

Can anybody please explain me, how to use this Updatescript /usr/sbin/etcupdate ???


----------



## Alain De Vos (Jun 1, 2021)

And not doing etcupdate i don't see stuff failing.


----------



## zirias@ (Jun 1, 2021)

Am I missing something or didn't you do a base system upgrade between bootstrapping etcupdate and first using it?


----------



## rigoletto@ (Jun 1, 2021)

I don't think a lot of people use etcupdate(8) but mergemaster(8) which does most of the work interactively.


----------



## zirias@ (Jun 1, 2021)

rigoletto@ said:


> I don't think a lot of people use etcupdate(8) but mergemaster(8) which does most of the work interactively.


mergemaster is deprecated and will be unreliable after 13.0-RELEASE. More info:








						Heads-Up: Deprecation of mergemaster
					

So far, I was using mergemaster when upgrading from source; it worked fine, and it's also described in the handbook.  Now I learned it's deprecated and will be removed soon. It might be dangerous after 13.0-RELEASE, because it relies on SVN source tags that don't exist any more with git!  For...




					forums.freebsd.org
				



The handbook was updated a while ago to use etcupdate instead of mergemaster.


----------



## HL1234 (Jun 1, 2021)

Zirias said:


> Am I missing something or didn't you do a base system upgrade between bootstrapping etcupdate and first using it?


I was on FreeBSD 12.2. I fetch the sources for 13.0 from Git then `make build` and `make install`. then I got busy oneself with etcupdate. Before I run it in default mode I do a bootstrap how it is written in the man page with `etcupdate extract`. 
Before fetching the sources /usr/src and /usr/obj was empty.


----------



## HL1234 (Jun 1, 2021)

rigoletto@ said:


> I don't think a lot of people use etcupdate(8) but mergemaster(8) which does most of the work interactively.


mergmaster has gone. It comes no more with FreeBSD since version 13.0. But you can still install it from the ports. But I read here that it can not work with Git, so it makes further no sence to use it. I have to learn a new way. 

With mergemaster I had also much problems. It would help me when:
a) I can build all new configuration files in any place.
b) Can install all new configuration files, that I have not changed, with a command (including this one where only the FreeBSD-Id has changed) (for mergmaster it was `mergmaster -Ui` and or `mergmaster -F`)
c) The rest I will merge it by myself. (with mergmaster the files not copied with the method above gone in /tmp/temproot/etc, so I can take an editor and compare is there really something new between taht and the old files in /etc)

Do anybody know the commands for that.


----------



## HL1234 (Jun 1, 2021)

Alain De Vos said:


> And not doing etcupdate i don't see stuff failing.


sorry, wrong. The configuration files which are in /etc for FreeBSD changes by every major update. I.e. for some years there was no folder /etc/newsyslog.conf.d only the file /etc/newsyslog.conf. Files in the folders /etc/rc.d or /etc/defaults could also changed. I.E I see in /etc/rc.d the files are from 05.12.2020. The files from the new FreeBSD 13.0 install /bin or /sbin are now from 16.05.2021.

Your installed software goes to /usr/local/ After an fresh new complete OS install, without anything else, this folder was empty. After installing software or services, there configuration files are in /usr/local/etc/...


----------



## SirDice (Jun 1, 2021)

HL1234 said:


> mergmaster has gone. It comes no more with FreeBSD since version 13.0.


Nope. It's still there.

```
mergemaster: /usr/sbin/mergemaster /usr/share/man/man8/mergemaster.8.gz /usr/src/usr.sbin/mergemaster
root@molly:~ # uname -a
FreeBSD molly.dicelan.home 13.0-STABLE FreeBSD 13.0-STABLE #29 stable/13-n245802-43acee0cdd5: Sun May 30 20:16:03 CEST 2021     root@molly.dicelan.home:/usr/obj/usr/src/amd64.amd64/sys/MOLLY  amd64
```



HL1234 said:


> But you can still install it from the ports. But I read here that it can not work with Git, so it makes further no sence to use it.


That's not entirely correct. It works fine with git. The problem is the fact that with git there is no version ID any more at the top of the source files, so you can't use that as a basis to compare if files need to be merged or not.


----------



## zirias@ (Jun 2, 2021)

None of the tools has a direct dependency on svn or git or any other VCS, but mergemaster relies on generated ID lines. Therefore it "won't work (reliably) with git", because git doesn't generate them…

HL1234 etcupdate(8) works with a 3-way compare, pretty similar to what a VCS does when merging or rebasing a branch. One of the inputs for that is the old unmodified state. In a nutshell, it figures out the changes you have from the old unmodified state and applies them automatically to the new unmodified state. This doesn't depend on any ID lines and is very comfortable, there's rarely a reason for manual actions, only on "merge conflicts".

The old unmodified state must be recorded _once_ before it can work for the first time, that's what the "bootstrapping" does. If you do that _after_ you upgraded, the old state will be the same as the new state, so it can't work.

Your way out of that: use mergemaster(8) one more time instead. Make sure while merging that you take the new ID lines (empty, only containing `$FreeBSD$`). For upgrade from 12.2 to 13.0, mergemaster will still work correctly. It won't for 13.0 → 13.1 for example, but you now have the old unmodified state (of 13.0) recorded, so next time, etcupdate will work as expected.


----------



## HL1234 (Jun 2, 2021)

SirDice said:


> Nope. It's still there.
> 
> mergemaster: /usr/sbin/mergemaster /usr/share/man/man8/mergemaster.8.gz /usr/src/usr.sbin/mergemaster


You are right! I think I have searched in the wrong place.

SirDice  I have in addition another question. I took a closer look to /var/db/etcupdate/log and the script. I don't see any interaction with Git. But I was amazed that it is building 3 times a lot of stuff from the sources.
Has not all files, also this for the configuration, defaults and start scripts for /etc build with `make buildworld`?
I suppose that this files are anywhere in /usr/obj/usr/src/... or comes with the sources /usr/src/etc/.... Why build them again? (I formatted the log for better reading)

```
>>> Building tree at /var/db/etcupdate/current with make  -DNO_FILEMON
cd /usr/src/etc; MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE=native
    CC="clang -target x86_64-unknown-freebsd13.0
        --sysroot=/usr/obj/usr/src/amd64.amd64/tmp
        -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"
    CXX="clang++
        -target x86_64-unknown-freebsd13.0
        --sysroot=/usr/obj/usr/src/amd64.amd64/tmp
        -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"
    CPP="clang-cpp
        -target x86_64-unknown-freebsd13.0
        --sysroot=/usr/obj/usr/src/amd64.amd64/tmp
        -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"
    AS="as" AR="ar" LD="ld" LLVM_LINK=""  NM=nm OBJCOPY="objcopy"  RANLIB=ranlib STRINGS=  SIZE="size" STRIPBIN="strip"
    PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:
         /usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:
         /usr/obj/usr/src/amd64.amd64/tmp/usr/bin:
         /usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:
         /usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:
         /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:
         /usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec:
         :/sbin:/bin:/usr/sbin:/usr/bin make
     INSTALL="install -N /usr/src/etc"
     MTREE_CMD="mtree -N /usr/src/etc"
     METALOG= distrib-dirs

for file in /usr/share/doc/usd/10.exref
            /usr/share/doc/usd/11.edit
            /usr/share/doc/usd/12.vi
            /usr/share/doc/usd/13.viref; do
    if [ -f /var/db/etcupdate/current/${file} ]; then
        rm -f /var/db/etcupdate/current/${file};
    fi;
done
....
--------------------------------------------------------------
>>> stage 2.2: rebuilding the object tree
--------------------------------------------------------------
cd /usr/src; MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE=native
    CC="clang
...
...
--------------------------------------------------------------
>>> stage 4.4: building everything
--------------------------------------------------------------
[Creating objdir /var/db/etcupdate/current/usr/obj/usr/src/amd64.amd64/etc...]
    cd /usr/src/etc; MACHINE_ARCH=amd64  MACHINE=amd64  CPUTYPE=native
    CC="clang -target x86_64-unknown-freebsd13.0
        --sysroot=/var/db/etcupdate/current/usr/obj/usr/src/amd64.amd64/tmp
        -B/var/db/etcupdate/current/usr/obj/usr/src/amd64.amd64/tmp/usr/bin"
.....
.....
```
 And where are the files:
/usr/share/doc/usd/10.exref
/usr/share/doc/usd/11.edit
/usr/share/doc/usd/12.vi
/usr/share/doc/usd/13.viref On my machine there is no folder /usr/share/doc/usd/?


----------



## HL1234 (Jun 2, 2021)

Zirias said:


> The old unmodified state must be recorded _once_ before it can work for the first time, that's what the "bootstrapping" does. If you do that _after_ you upgraded, the old state will be the same as the new state, so it can't work.


Does that mean I should never delete the files in /var/db/etcupdate/old?
And how does this works with a fresh new install from the sources, because there are no old unmodified state?
----
After finished the update or upgrade I delete /usr/src and /usr/obj. So, with the version 12.2 I would could never do a bootstrap from the old sources. Does `etcupdate`not compare the new files with the installed by coping them in ./previous?
----
An other idea I have is: instead of using mergmaster can I not copy the installed files into /var/db/etcupdate/old then they are different to /var/db/etcupdate/current? I know I must then use `etcupdate -r` because other it will copy ./current to ./old and build ./current again.


----------



## zirias@ (Jun 2, 2021)

HL1234 said:


> Does that mean I should never delete the files in /var/db/etcupdate/old?


This shouldn't hurt as old is only needed _during_ the update of etc, but, why?
Before using etcupdate(8) normally, you just need current, created _before_ doing the system upgrade.



HL1234 said:


> And how does this works with a fresh new install from the sources, because there are no old unmodified state?


Bootstrap before doing the upgrade. How should a "fresh new install" work from sources? You already need a running system to be able to compile.



HL1234 said:


> After finished the update or upgrade I delete /usr/src and /usr/obj.


Why?



HL1234 said:


> An other idea I have is: instead of using mergmaster can I not copy the installed files into /var/db/etcupdate/old then they are different to /var/db/etcupdate/current?


Then your local changes would be included, which defeats the concept of a 3-way compare.


----------



## HL1234 (Jun 2, 2021)

Zirias sorry, but your answers confuse me. 
1) If ./old is needed for an update to /etc why doesn't it hurt when I delete it?
2) Your are right. I forgot, the first install I took an DVD.
3) I only compile world when there is a major upgrade, with some small settings in /etc/src.conf. After that I use for updates `freebsd-update` for the world and if something for the kernel has changed too, I fetch the sources and compile the kernel. On my rent virtual server `make buildworld` has taken more then 5 hours.
4) but would that works?
Or is this the 3-merge way? I.e. in ./old are the files created form the sources for 13.0. When it comes to an upgrade (may 13.1) the files from the new fetched sources go to ./current. Then `etcupdate`compares ./old with ./current and the the files which are new between 13.1 and 13.0 will compared with the installed from ./etc. Have I understood that correctly?
--------------
If I would delete the content of ./current and ./old after installing 13.0 configurations files. And it comes to FreeBSD version 13.1 - and I fetch 13.1 new from the Git sources - does I get then the same problems with `etcupdate` like now ? When the answer is Yes, I must then keep an tarball from the last configuration file installation.


----------



## zirias@ (Jun 2, 2021)

First of all:


HL1234 said:


> 3) I only compile world when there is a major upgrade, with some small settings in /etc/src.conf. After that I use for updates `freebsd-update` for the world and if something for the kernel has changed too, I fetch the sources and compile the kernel. On my rent virtual server `make buildworld` has taken more then 5 hours.


I don't think that's a good strategy. To begin with, there's never a reason to delete /usr/src, it won't be modified by building. Just clone the source tree, check out the branch you want (e.g. `releng/13.0`) and then you can keep it up to date with `git pull`.
Then, if you customize world (by setting things in /etc/src.conf), it probably isn't a good idea to use freebsd-update(8). Or, the other way around, if you only customize the kernel, there's no need to ever build world, just use freebsd-update(8) for upgrades as well. If you want to reduce build times, _don't_ delete /usr/obj but use _meta mode_ for building (load filemon(4) and put `WITH_META_MODE=yes` in /etc/src-env.conf). Especially with patch levels, this is very efficient, as the changes are small.

But back to etcupdate(8):


HL1234 said:


> 1) If ./old is needed for an update to /etc why doesn't it hurt when I delete it?


It's called previous, not old, but anyways: it's needed _during_ the `etcupdate` process. First thing will be to move current to previous and build a new current from the source tree. Then, the 3-way compare (and merge) is done by (simplified) creating a diff of your installed /etc to previous, applying that diff to current and installing the result back to /etc. After that, previous isn't used any more (except when you give `-r` to redo the merge), so it's safe to remove. But again, why would you do that? It's removed/overridden anyways when using `etcupdate` the next time.



HL1234 said:


> If I would delete the content of ./current and ./old after installing 13.0 configurations files. And it comes to FreeBSD version 13.1 - and I fetch 13.1 new from the Git sources - does I get then the same problems with `etcupdate` like now ?


Yes.



HL1234 said:


> When the answer is Yes, I must then keep an tarball from the last configuration file installation.


No. Just don't touch /var/db/etcupdate.

Again, for now, just use mergemaster(8) for a last time. Then make sure to create /var/db/etcupdate/current (with `extract` subcommand) and don't touch it any more. With the next upgrade, etcupdate(8) will _just work_.


----------



## jhb@ (Jun 4, 2021)

The other note is that before running 'etcupdate' to do a merge in the future, run 'etcupdate diff'.  If it gives you an error saying that there's no previous tree to compare against, you need to bootstrap it.  If it gives you a bunch of diffs you don't expect (because you've done upgrades with mergemaster since you've installed), then you also need to bootstrap it and use mergemaster one last time.  However, installs from releases since 10.0 or thereabouts have included a bootstrapped etcupdate database as part of the install, so if you install a new machine or VM you should just be able to use 'etcupdate' to merge changes to /etc after each installworld without needing to bootstrap.  Also, Zirias is right that you shouldn't need to touch /var/db/etcupdate directly but let 'etcupdate' manage that.


----------



## HL1234 (Jun 5, 2021)

Thanks all,
I have now used `mergemaster -F` and the last two days merged the remained files with more differences by hand.
Then I have yet done again `etcupdate extract`. There was no output, but the files in ./current are form today.
I never have seen a folder ./previous like it is described in the manual, but a folder ./old.

```
[/var/db/etcupdate] => ll
total 240K
drwxr-xr-x  5 root wheel    512 June  5 17:03 .
drwxr-xr-x 22 root wheel   1024 June  4 20:23 ..
-rw-r--r--  1 root wheel      0 June  1 17:22 added.files
-rw-r--r--  1 root wheel   7598 June  1 17:22 both.files
drwxr-xr-x  2 root wheel    512 June  1 17:22 conflicts
drwxr-xr-x  7 root wheel    512 June  5 17:04 current
-rw-r--r--  1 root wheel 195711 June  5 17:04 log
-rw-r--r--  1 root wheel   7598 June  1 17:22 new.files
drwxr-xr-x  7 root wheel    512 May  29 20:49 old
-rw-r--r--  1 root wheel   7598 June  1 17:22 old.files
-rw-r--r--  1 root wheel      0 June  1 17:22 removed.files
[/var/db/etcupdate] => ll ./old
total 44K
drwxr-xr-x  7 root wheel  512 May  29 20:49 .
drwxr-xr-x  5 root wheel  512 June  5 17:03 ..
-rw-r--r--  2 root wheel 1023 May  29 20:49 .cshrc
-rw-r--r--  2 root wheel  507 May  29 20:49 .profile
drwxr-xr-x  2 root wheel  512 May  29 20:49 boot
-r--r--r--  1 root wheel 6109 May  29 20:48 COPYRIGHT
drwxr-xr-x 21 root wheel 2048 May  29 20:49 etc
drwxr-xr-x  2 root wheel  512 May  29 20:49 root
drwxr-xr-x  3 root wheel  512 Mai  29 20:49 usr
drwxr-xr-x  3 root wheel  512 Mai  29 20:49 var
```
Again thanks for helping me to solve the problem.


----------

