# Managing your system configuration with svnlite



## ShelLuser (Nov 24, 2018)

Hi gang!

*Editorial*

There are many reasons why I'm passionate about FreeBSD as an operating system and why it's my all time favorite, two of which are fully related to this article:

FreeBSD is a complete system, meaning that you may end up surprised if you stop to think about the sheer functionality provided by _just_ the base system.
Curious? compiler & debugger (nowadays this seems like an afterthought, but I lived the days that you actually had to pay to get your hands on a compiler other than GCC (and I'm not only referring to Microsoft here!)), very extensive & fully customizable mailserver (Sendmail), simple file encryption, a more extensive encryption suite & extensive disk encryption facilities (/usr/bin/enigma, /usr/bin/openssl and /sbin/geli), an intrusion detection system (/usr/sbin/mtree), network monitoring / managing support (/usr/sbin/tcpdump and many other related tools) _and..._ "out of the box" support for version control through Subversion.

FreeBSD doesn't do popularity contests. Of course people keep a check on which tool does the best job, but when the next hip & modern tool comes out then it doesn't "just" jump on another bandwagon because something managed to gain extra popularity (it's also important to realize that we're talking about 'out of the box' functionality, something which is also very easy to customize so that the system fully suits _your_ specific needs).
And for the record: If you have been reading some of my other posts and sporadic shared opinions then you may know that I have a certain bias in favor of Git, definitely not going to deny that. But that doesn't mean that I suddenly stopped caring (or liking) other alternatives. And honestly? Within the context of FreeBSD I think that Subversion is an awesome tool which can really help you to elevate your systems administration tasks to the next level.

Disclaimer: In most of my shared examples you'll see that I'm (ab)using the root user. The only reason I do so is because I'm using my Psi jail to provide all the examples which doesn't have any extra user accounts (mostly because it's not being used on a network). But please keep well in mind that you shouldn't use root for most of these examples; a regular account will work just as fine and it'll be a lot safer!

*What is version control?*
Version control is performed with a version control system ("VCS") and sometimes also referred to as _software configuration management_ ("SCM"). It's basically a discipline most often practiced in software development where you store every change you make to your system and provide some extra comments on why you made those changes. This can help you to keep track of its history, it allows you to revert your changes back to a prior version at any time you'd like (either permanent or simply to study a prior version) and you can always fully check the specific differences between those (previous) versions.

It can also help to share your work with others and work on something together, but that part is beyond the scope of this tutorial.

*Why use VCS for system configuration?*
I assume that you keep backups of your system, but how high is your retention? On my main server I keep daily backups through ZFS snapshots which retention only lasts 7 days. There's also a real (off-site) backup which gets made on a weekly basis with a few incrementals sent during the week. These weekly backups are stored for 4 weeks.

So basically my retention is about a month.

Now imagine that I upgraded PHP from 5.6 to 7.0 a few weeks ago which also included a drastic overhaul of my php.ini file. The new configuration works perfectly but I made a few changes of which I no longer remember why I did that. I also seem to recall that I had better results with some previous settings and although I can look up the properties online (in the documentation) it would be much easier if I could check out my previous configuration again.

Unfortunately this would mean that I'd have to dig this up from my backup server, which can become quite a hassle, so it's probably easier to spend the next 20 minutes skimming through the documentation again.

If only there were a system which could store all these changes in an optimal (redundant) way and which would give us full access to previous versions.

Guess what?  That's _exactly_ what a version control system such as Subversion can do for you!

The advantages should be obvious:

Safeguarding: By letting Subversion track your configuration you can always undo any unwanted changes, or verify these changes against previous versions.
Back up the backup   By including the Subversion data in your main backup you're basically safe keeping it; while it can be quite a hassle to dig into your backup history this doesn't hold true for a Subversion repository (= the place where Subversion stores its data). So you're basically making a backup of your full configuration backup 
Useful on other servers. Though I'm not going to address this here you could provide your data to other (local) servers, which might help to set them up much quicker. For example: instead of having to configure Apache from the ground up you could also grab your existing configuration and use that as a starting point.
Documentation. All it takes is a simple description of why you made a certain change, something which might help you later to verify why you made certain changes.
Support is provided "out of the box". The moment you install a FreeBSD system is the moment when you can access your repository.
*Let's get started!*
Subversion manages data in two different ways. All data gets stored in a so called _repository_; this is somewhat comparable to a database (it basically is a database). Normally you don't have to do much with a repository, but it has some interesting tweaks which we'll check out later. One repository can store many projects, but because of that it will become important to make sure that you keep track of how you stored this data.

In order to access the data (so the actual files which are stored inside the repository) we need a so called _working copy_. This is basically the place where we work with both our data and the main repository, hence its name.

*Creating the repository*
The first thing we need is a repository to store our data, this is where svnliteadmin comes into play. You can basically store this anywhere you want, but I prefer using /var/db/repo:

```
root@psi:~ # svnliteadmin create /var/db/repo
root@psi:~ # ls /var/db/repo/
README.txt      db/             hooks/
conf/           format          locks/
```
Because this repository is going to be used to store server configuration files I *strongly* recommend that you make sure that no unauthorized users will be able to access this. For my own servers I prefer to limit access to the wheel group, but for my company servers we decided to utilize a dedicated group. So, the next step will be: `# chmod 770 /var/db/repo`, which will prevent unauthorized access.

You have a few options here...  Use chmod (and optionally chown) to make sure that wheel (or another group) has full writing permissions. Or use svnliteserve to access the repository. It can be used as a daemon which would mean that you'd have to customize /var/db/repo/conf/passwd in order to set up specific Subversion accounts.

In my example we'll stick with file system access, so to keep people out of our repository we rely on the chmod command shared earlier.

*Deciding what data gets added*
The most obvious locations to add to our new repository are of course /etc and /usr/local/etc. But the latter requires a bit more attention. See, the problem is that although packages never contain actual configuration files they do often provide examples:

```
root@psi:~ # pkg which /usr/local/etc/*.sample
/usr/local/etc/clamd.conf.sample was installed by package clamav-0.100.2
/usr/local/etc/freshclam.conf.sample was installed by package clamav-0.100.2
/usr/local/etc/pkg.conf.sample was installed by package pkg-1.10.5_5
```
These files could easily change whenever we upgrade the software they're related to, which would then immediately trigger our VCS system. Now, this doesn't have to be a bad thing of course: getting alerted to such changes can be a good thing. It's just that we need to decide if we want to set up such a scenario or not.

Another point of attention are binary data files. Take for example /usr/local/etc/sasldb2.db. This is a data file used by Cyrus SASL; a library used for authentication purposes. Or what to think about /etc/pwd.db?

Now, in this scenario these files wouldn't become a big problem; Subversion actually knows how to recognize and handle non-ascii files, even if you don't tell it to. But there is something to say about the risk(s) involved with storing password information in a somewhat insecure setup like our repository (it's not _that_ insecure, but Subversion also wasn't build with high-end security in mind).

So we're going to exclude these files, and for that we need to reconfigure Subversion a bit. This is done by editing the configuration found in ~/.subversion. If this directory doesn't exist yet then you can create it by simply running `svnlite help`:

```
root@psi:~ # ls .subversion
ls: .subversion: No such file or directory
root@psi:~ # svnlite help > /dev/null
root@psi:~ # ls .subversion/
README.txt      auth/           config          servers
```
Now edit config and find & edit this section:

```
### Section for configuring miscellaneous Subversion options.
[miscellany]
### Set global-ignores to a set of whitespace-delimited globs
### which Subversion will ignore in its 'status' output, and  
### while importing or adding files and directories.
### '*' matches leading dots, e.g. '*.rej' matches '.foo.rej'.
global-ignores = *.sample *.db
```
This will make sure that we don't have to worry about those sample and data files anymore.

*Adding the data*
One of the major advantages of Subversion is that it can handle large amounts of data very well. And it's not only about adding the data: working with this data is quite easy too. For example: there are many files within the /etc directory which could become a hindrance if you're only interested in the SSH configuration. Fortunately Subversion accounted for that and it allows you to access individual parts of your data if you need to.

This is why I prefer to simply add the entire /etc and /usr/local/etc structures, especially since we already made sure to exclude some unwanted overhead. However we do need to be careful and make sure that we store both directories within separate locations inside our repository.

Don't worry, it sounds more difficult than it actually is:

```
root@psi:~ # cd /usr/local/etc
root@psi:/usr/local/etc # svnlite import . file:///var/db/repo/etc.local -m "Initial commit of /usr/local/etc"
```
Note that you can omit the . if you want to (Subversion imports the current directory by default) but I prefer not to take any chances. If you want to add more information about this data than just the description then leave out the -m parameter; then your default editor will open which allows you to provide extra information ("meta data").

Don't be fooled by the file:// prefix; we're not adding a single file here but stored our data under the etc.local header. Subversion requires an URL to specify the main repository, and by using file:// we're basically telling Subversion that the repository should be accessed using our current filesystem.

After this command Subversion will show a list of files which it has added to the repository and end with a confirmation:

```
Adding         rc.d
Adding         rc.d/clamav-clamd
Adding         rc.d/clamav-freshclam
Adding         rc.d/imapd
Adding         rc.d/nsd
Adding         slsh.rc
Adding         ssl
Adding         ssl/cert.pem
Committing transaction...
Committed revision 1.
```
The revision is a unique identifier which Subversion uses to keep track of our changes. Since this is the first addition to our repository it got revision number 1, makes sense right? After you do the same for /etc you'll notice that Subversion will now tell you that you commited revision 2.

*Putting our configuration under version control*
Now, adding the data is only step one of the whole procedure, something you can see for yourself if you ask svnlite about the current status:

```
root@psi:/usr/local/etc # svnlite status
svn: warning: W155007: '/usr/local/etc' is not a working copy
```
Therefor we'll need to tell Subversion that we wish to use the data which we just stored. We do that by _checking out_ this data, like so:

```
root@psi:/usr/local/etc # svnlite co --force file:///var/db/repo/etc.local .
```
You will now see that Subversion has extracted several files and directories:

```
E    clamd.conf
E    cyrus.conf
E    freshclam.conf
E    imapd.conf
E    pkg.conf
E    slsh.rc
Checked out revision 2.
```
Congratulations, your /usr/local/etc directory is now under version control!

So what happened here? Subversion is very careful with handling data and it normally expects to 'extract' data into a new (empty) working directory. However, our directory wasn't empty which would have triggered Subversion to ask for confirmation if we really wanted to use the existing data, that's why I used --force to enforce this.

But there's more: despite the use of the parameter it will not force Subversion to blindly overwrite any existing data, _far_ from it:

```
root@psi:/usr/local/etc # stat pkg.conf
3878456435 35404 -rw-r--r-- 1 root wheel 4294967295 2143 "Nov 24 06:41:08 2018" "Oct 18 01:11:17 2018" "Nov  2 15:28:03 2018" "Oct 18 01:11:17 2018" 4096 6 0x800 pkg.conf
root@psi:/usr/local/etc # date
Sat Nov 24 07:00:57 UTC 2018
```
See what I mean? Sure, the access time is just a while ago (I'm setting up these examples while I'm writing this article) but notice the modification timestamp (the sequence is: access time, modification time, creation time & birth time)? Even though we told Subversion to enforce "extraction" it didn't blindly overwrite any of our current data at all.

If you're more familiar with Subversion then you could already tell this from the previous output. That *E* you saw? It doesn't mean 'extracted' which is something I hinted at but it actually means _Existed_; Subversion indicated that the files and directories already existed, and therefor it merely added them under version control, but nothing more.

*Making changes*
So now the only thing you need to remember is that whenever you make a change to the system you should also tell Subversion about it. When in doubt just ask Subversion what the current status is:

```
root@psi:/usr/local/etc # svnlite status
root@psi:/usr/local/etc #
```
Remember: in true Unix fashion 'no news' means 'good news'. So Subversion is telling us that we don't need to worry about the current state of our working copy. If you want more detailed information about our setup then you can use:

```
root@psi:/usr/local/etc # svnlite info
Path: .
Working Copy Root Path: /usr/local/etc
URL: file:///var/db/repo/etc.local
Relative URL: ^/etc.local
Repository Root: file:///var/db/repo
Repository UUID: 6f0b2ca5-adef-e811-9a4c-00123f2e3b36
Revision: 2
Node Kind: directory
Schedule: normal
Last Changed Author: root
Last Changed Rev: 1
Last Changed Date: 2018-11-24 06:41:09 +0000 (Sat, 24 Nov 2018)
```
So lets reconfigure PKG. I'm going to remove all the comments and get rid of the default aliases. Then I'm going to set up a new repository directory:

```
root@psi:/usr/local/etc # cat pkg.conf
# System-wide configuration file for pkg(8)
# For more information on the file format and
# options please refer to the pkg.conf(5) man page

REPOS_DIR [
        "/etc/pkg/",
        "/usr/local/etc/pkg/repos/",
        "/root/repos/"
]
```
So now let's see what Subversion has to say about this:

```
root@psi:/usr/local/etc # svnlite status
M       pkg.conf
```
As you can see it has detected that we modified pkg.conf ('*M*') From here on we have several options...  If you want to see exactly what has changed then you can use `svnlite diff`. If you want to undo these changes then you can use `svnlite revert pkg.conf` after which things will be changed back to the way they were, this can be an invaluable command if you made accidental changes and you don't have a ZFS snapshot at your disposal.

And finally if you wish to keep your changes then you should _commit_ them using: `svnlite commit`. I suggest that you don't use the -m parameter but instead use the full editing screen. This will allow you to add any data you might need; not merely a description but also information about possible problem reports, information if someone approved the change (and who that person is, same applies to reporting, reviewing and submitting) and even specific security related information can be added.

This is one of the reasons why I think that Subversion is a very good tool to use for this, because you can look up all that extra information at a later time if you need to.

Keep in mind though that once you committed a change then you should also update the current working copy again, you can do that by using `svnlite update`. Need an overview of all the recent changes? Couldn't be easier: `svnlite log | less`.

*Allowing someone else to make changes?*
So let's say that our SSH server has gone bonkers and we can't fix this ourselves. Someone else has offered to help us and he can edit the config files for us. However, that would require direct access and since this person isn't a part of our usual team we can't "just" give him access.

Fortunately we have Subversion at our disposal!


```
root@psi:~ # cd /home/helper/
root@psi:/home/helper # svnlite co file:///var/db/repo/etc/ssh
A    ssh/moduli
A    ssh/ssh_config
A    ssh/sshd_config
Checked out revision 3.
root@psi:/home/helper # chmod 666 ssh/*config
```
Note that in this example I made sure that the keys aren't included (by not adding them in the first place) but this is an important aspect because it could be a security risk.

So now our helper can safely edit these files. We could tell him to commit the changes once he's done, but this would mean that he'd need to have access to our repository. So another option is to make him apply the changes after which we'll commit them instead.

The advantage is that our helper never gets their hands on the actual configuration. And thanks to Subversion we can check out _exactly_ what he changed long before actual implementing it (remember `svnlite diff`?). And speaking of which... once we are satisfied with all the changes then we also no longer need to mess around with cp or mv. Just use Subversion to update the actual configuration:

```
root@psi:/etc/ssh # svnlite update
Updating '.':
U    sshd_config
Updated to revision 4.
```
This is also one of the advantages which Subversion provides: once the changes got committed they became immediately available to our main working directory.

It's also an example of what I mentioned earlier on: how Subversion was quite capable of handling large amounts of data. As you can see we never had to check out ('extract') the entire /etc directory. And of course you can expand on this; you could even use such a setup to provide the same configuration across a series of servers.

*Securing our data*
Now, obviously the idea is to include /var/db/repo in your common backup scheme so that the whole repository will be safely stored away. But what about making a security copy just to be safe? Once again Subversion to the rescue:

```
root@psi:~ # svnliteadmin hotcopy /var/db/repo repo.copy
* Copied revision 0.
* Copied revision 1.
* Copied revision 2.
* Copied revision 3.
* Copied revision 4.
root@psi:~ # tar cJf svnrepo.txz repo.copy/
root@psi:~ # rm -r repo.copy/
```
So, the reason why I used svnliteadmin instead of tar right away is because there was no way for me to tell if the repository could be in use. This is somewhat comparable to using vi to edit /etc/passwd during which someone else also changes their password. So even if someone was working on our system configuration then svnliteadmin would have made sure that it couldn't interfere with our backup.

*If you need more help...*
All Subversion commands have build-in help screens which can be shown by using the help command. So if you want to know more details about the commit command then you can use: `svnlite help commit | less`, if you simply need an overview of all admin commands at your disposal: `svnliteadmin help`, and so on.

_And there you have it..._

From here on you can basically continue to commit & update any changes after which you'll eventually build up quite a log. The main advantage though is that even after a year of updating you'll still have access to the way things were back then. And without all the hassle that a regular backup brings along (nor its massive size).

Also: since you already have a repository at your disposal there's no reason why you shouldn't use that for some actual development. Do you maintain your own shell scripts? Maybe it could be a good idea to add those as well!


----------



## Bobi B. (Nov 24, 2018)

Tried to do something similar when working on nanobsd appliances at work and stumbled on two issues: neither svnlite(1), nor svn(1), preserved filesystem objects' ownership and permissions. And some daemons (like sshd(8)) are pretty strict for ownership and permissions of configuration and auxiliary files.

Got any pointers on how to deal with that?


----------



## ShelLuser (Nov 24, 2018)

Good question! Yeah, file system properties are not included within such scenarios, but the cool part about programs such as Subversion (as well as Git) is that you still make this happen through the use of _hooks_.

In the main repository you'll find a directory called hooks which contains several template shell scripts. These hooks (or scripts) get executed by Subversion whenever something happens with the repository. For example: pre-commit, start-commit and post-commit get called during the several stages of the committing process.

If you combine this with mtree(8) (= tool to map a directory hierarchy, also usable as an intrusion detection system) then you could automatically trigger Subversion to update the filesystem data (within mtree context this is called a _specification_) right before actually committing the data. This would ensure that you'd get a snapshot of the whole directory with every commit.

The only bad news is that with Subversion it'll be a little bit harder to automate the process to check the data against the current situation because Subversion doesn't have any update or 'check out' hooks.

So, for example: `mtree -ck uid,gid,time,mode,flags > etc.mtree`. This would create a specification of your current directory and store the mentioned data within etc.mtree. Checking against this data is very simple: `mtree < etc.mtree`.

There is of course a small caveat:

```
root@psi:/usr/local/etc # time mtree -ck uid,gid,time,mode,flags > etc.mtree
0.000u 0.016s 0:00.01 100.0%    112+312k 80+1io 0pf+0w
root@psi:/usr/local/etc # mtree < etc.mtree
etc.mtree:
        modification time (Sat Nov 24 22:28:54 2018, Sat Nov 24 22:28:54 2018)
```
As you can see the time it took mtree to process my /usr/local/etc was next to nothing even though it had approx. 400 entries to go through. The downside though is that the specification will always trigger an alert because it's being modified right when it gets processed itself. Although mtree can be told to ignore patterns you can only do this using an actual ignore file, which would only add extra clutter to the directory. Still, it should be possible to use sed(1) to filter out this data so that you don't get false positives.

Hope this can give you some ideas.


----------



## Bobi B. (Nov 25, 2018)

Thanks! That is a nifty solution. Perhaps the lack of respective hooks can be solved by wrapping svnlite(1) in a shell script: i.e. the admin will not use `svn` directly, but a shell script that will take care to execute pre- and post-fix steps?


----------



## kpa (Nov 25, 2018)

I use rather different approach to system configuration files.  I have everything under a VCS (git)  but I never checkout anything over the live directories.  I use Makefiles that know the installation directories, ownership and file modes. The files also know how to run sanity checks on the files if applicable. For example, this is what I use to install my PF configuration files and reload the rules into PF.


```
PREFIX=/opt

PFCTL=/sbin/pfctl

PF_CONF= pf.conf

SCRIPTS= firewall-log.sh

SERVICE= /usr/sbin/service

all:

install: check install-scripts install-pf-conf

install-scripts:        $(SCRIPTS)
        $(INSTALL) -o root -g wheel -m 755 $> $(PREFIX)/sbin


install-pf-conf: ${PF_CONF}
        $(INSTALL) -o root -g wheel -m 600 $> /etc/pf.conf

check:
        ${PFCTL} -n -f ${PF_CONF}

reload:
        ${SERVICE} pf reload

restart:
        ${SERVICE} pf restart
```

With that I can just first do `sudo make check` to see if my rules have errors, then if all looks good I can do `sudo make install reload` to install and reload the rules. The install target even forces the check before installing anything so you won't mess up your system with faulty rules if you don't remember to run `make check` first. Of course this approach requires more work on the Makefiles but in my opinion it's nicer this way to keep the machinery that does the installation completely separate from the system directories. You also need to have some self discipline so that you never edit the installed files directly but always use this method for installing the files.


----------

