# Compiling a kernel - a walkthrough



## carlton_draught (Dec 12, 2010)

I'm starting this thread as a result of this other thread. Hopefully it helps someone out. Maybe even me. Anyone reading this who knows what they are doing, please feel free to correct me or answer questions as I ask them.



			
				UNIXgod said:
			
		

> When you are ready to compile your kernel start a new thread and I would be more than happy to help you with grokking the config and maybe save you one less headache.



So where am I at now, and what do I want to do?

Where I'm at now:

I have a ZFS root mirror setup, using 8.0-RELEASE.
Have upgraded from 8.0-RELEASE to 8.1-RELEASE.
Have done a `# [url=http://forums.freebsd.org/showpost.php?p=39092&postcount=37]portupdater.sh[/url] yes` in the midst of the former, deleted some libraries, and I think botched my gnome somehow. This is NOT a criticism of portupdater.sh btw, it's very likely a reflection of my own combination of ignorance/stupidity/impatience.

What I want to do:

Compile a custom kernel, and learn how to update that (and world I guess) so that I can do it regularly and easily, so as to have a properly secure and reliable system.
Get system able to boot from 2nd disk, which is apparently a problem with 8.1-RELEASE and ZFS root mirror setups. Test it.
Fix my broken gnome (no gnome panel).
Figure out how to properly backup and restore the updated root mirror (which probably involves figuring out the minimum set of directories/files to backup and the order in which to do so).

I may as well detail what I did to upgrade, for others following along:

Upgrading from 8.0-RELEASE to 8.1-RELEASE
I have managed to upgrade from 8.0-RELEASE to 8.1-RELEASE, by virtue of these threads (and especially this post). The process I took was the following:

I'd recommend doing this first - before we begin, do a snapshot of zroot:
`# zfs snapshot -r zroot@8.0_working_date`

Then
`# freebsd-update upgrade -r 8.1-RELEASE`
(Takes forever to download files. It was at this point that I decided to execute portupdater.sh as well.)
`# freebsd-update install`

Put in your 8.1-RELEASE DVD and do the following:
`# mount -t cd9660 /dev/cd0 /mnt`
`# gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptzfsboot -i 1 da0`
`# gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptzfsboot -i 1 ada0`

Rebooted, and it is now working!
`# shutdown -r now`

Remember to continue with instructions at here:

`# freebsd-update install`
`# shutdown -r now`

Note that we can see that (the kernel?) is a new version:
`# uname -r`

```
8.1-RELEASE
```

Note that to stop gnome coming up automatically if gnome panels aren't working, log in as root, click on something on the desktop and navigate to /etc/rc.conf and put a "#" at the beginning of the line:

```
gnome_enable="YES"
```
and reboot.

My next post will start off the custom kernel compile.


----------



## carlton_draught (Dec 12, 2010)

*Backing up a kernel and testing it.*

Before we begin, do a snapshot of zroot:
`# zfs snapshot -r zroot@8.1_before_custom_kernel_date`

Ok, so on the advice of UNIXgod, I'm doing the following first:


			
				UNIXgod said:
			
		

> If your really paranoid go through the process of loading the cp'd kernel as if you had to. Once you go through that process and see it work move to the step of compiling your own kernel.



So, to copy the kernel:
`# cp -r /boot/kernel /boot/kernel.works`
I also did the following just in case:
`# cp -r /boot/GENERIC /boot/GENERIC.works`

Now to test using an old kernel:

Reboot
Select option 6.
`# unload kernel` 
`# boot /boot/kernel.works/kernel`
Yay, the system boots properly.


----------



## carlton_draught (Dec 12, 2010)

Now, we reboot again using the default kernel.

We are ready to start "Configuring the FreeBSD Kernel".

The next page - "*8.2 Why Build a Custom Kernel?*" details the reasons we are doing this.

So why am I doing this? There are several advantages:

_Faster boot time._
Not a huge selling point, as the machine is on most of the time. It could be nice for testing backups and restores though.
_Lower memory usage._
With 8GB of RAM, this is probably not an issue.
_Additional hardware support._
Irrelevant to me.

Other reasons not listed:

I want a system that is patched as far as security holes go, and I'm guessing that having an up to date kernel is a part of that. I know that every so often kernel updates were part of life under linux, I would suspect this is the case under FreeBSD? (If so, how to know when to update the kernel? Or do it regularly?)
I need to get the boot loader updated through this patch. Someone says that using 8.1 STABLE has this patch included. Let's see, difference between RELEASE and STABLE... I know that doesn't apply to ports, does it apply to both kernel and world or just kernel? According to the latter thread, updating doesn't change RELEASE to STABLE, you need to set an option for that. And if security patches are applied to RELEASE, I'd rather just stick with RELEASE for now, especially if I can just apply necessary patches to RELEASE as well.
Maybe less security holes for the reason that in the same way that disabling services increases security, less stuff that is built into the kernel is less stuff that can go wrong.

Ok, as far as the boot loader patch is concerned, I have gotten stuck so far here. So my suspicions are that rather than figure out why that isn't working, if I compile the kernel I will at the same time solve that problem too. Or if I can't, then I will just have to upgrade to 8.1 STABLE.


----------



## carlton_draught (Dec 12, 2010)

Now we are ready to embark on the next page: 
*8.3 Finding the System Hardware*

Ok, that page lists multiple methods for finding out hardware information. I'm going to go with dmesg as a first resort, printed out to a USB stick which can be read from another computer (the one I am typing this from), so that I can minimize the switching between editing the conf file and finding information (perhaps it would be better to edit the conf file on the other computer, that way I would be free to use pciconf etc.). Nevermind.


Insert the (FAT formatted) USB stick.
Use dmesg to find the device node. However, maybe because it's 8.1, the device node (da3) is automatically spat out.
`# dmesg`
`# mkdir /mnt/usb`
`# mount_msdosfs /dev/da3 /mnt/usb`
`# dmesg > /mnt/usb/dmesg_output_20101212.txt`
Unmount the USB before taking it out.
`# umount /dev/da3`

With out trusty dmesg output in hand, we are ready to embark on the next step, which is:
*8.4 Kernel Drivers, Subsystems, and Modules*

Nothing to do in that one, it's just information that says basically to use modules if one is available.

The next page: 
*8.5 Building and Installing a Custom Kernel*

We skip forward to the "tip" to set things up so that we don't inadvertently erase the custom kernel config in a fit of anger.
`# cd /usr/src/sys/amd64/conf`
`# mkdir /root/kernels`
`# cp GENERIC /root/kernels/WORKSTATION` 
`# ln -s /root/kernels/WORKSTATION`

Now we are ready for what appears to be the bulk of the work in this process: 
*8.6 The Configuration File*
Now, my first choice: do I wuss out and just do a delta (basically a new file specifying to include GENERIC) or do I go through and erase everything that does not apply? Decisions, decisions. Advice?


----------



## UNIXgod (Dec 12, 2010)

Looks like your making progress.

If you would like to know what your nic driver your machine is using look at ifconfig() output.

You can safely remove all other nic drivers.
also remove _DEBUG=-g_

Some scsi drivers are needed for cd/dvd burning(_da_ and _pass_ come to mind). keep those and remove the rest.

Your kernel name should reflect your machines name (if you haven't thought of one here are some suggestions I use character names from hitchhikers guide to the galaxy book and other sci-fi references(i.e. 2001: space odyssey) . I have seen people use lord of the rings references. Don't name it after a living pet or dead rock stars =) ) 

So I have in one of my machines named
/root/kernels/HAL9000

Also there is a field in the kernel config called ident. Replace GERNERIC with your computers name there.

Since you mentioned security is a concern. Removing anything with the line root at the top of the file is safe to remove unless you know you will need it. (i.e. MD_ROOT, NFS_ROOT and so on)

After you have created a slim kernel take the time to study NOTES which is the old lint file. It contains everything possible to add to the kernel from linux filesystem support to altq for firewalls and even old mac(pre unix) filesystem support.

If you would like a easy grep-able file to see all the options without comments you can type


```
make LINT
```

inside your /usr/srs/sys/`uname -m`/conf folder and it will create two files named LINT and LINT-VIMAGE

For stuff I will never need I delete the line. For things I feel are worth keeping in the file but have no immediate need I comment out. (I'm a _vi_ user so flying through the file pressing dd is simpler than insert + hashmark everywhere)

If you want to create multiple kernel configs the second level up can use include syntax as so

for example here would be
/root/kernels/HAL9000-FIREWALL

`# set KERNIDENT=HAL9000`
`# printf "include ${KERNIDENT}\nident ${KERNIDENT}-FIREWALL\n\n" > HAL9000-FIREWALL`
`# grep ALTQ < LINT >> HAL9000-FIREWALL`


```
include HAL9000
ident HAL9000-FIREWALL

options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
options         ALTQ_NOPCC      # Required for SMP build
```

This may help with organization of having a slim kernel and another file acting as a feature filled kernel simply sucking in the slim config.

You could also do the reverse where you have a slim kernel named HAL9000-slim and have the rest of your additions and hacks in a file that suck it in named HAL000. 

Of course you can have everything in one file if you wish. The policy and organization is yours to make.

When you boot into it for the first time run uname -a() and see if output. Here is an example of my uname output:



> FreeBSD *hal9000*.rubyprogrammer.net 8.1-RELEASE FreeBSD 8.1-RELEASE #2: Thu Aug 12 17:55:19 CDT 2010     UNIXgod@hal9000.rubypr*grammer.net:/usr/obj/usr/src/sys/HAL9000  _amd64_



I'll keep following this thread. Post your kernel config here when you have it.


----------



## carlton_draught (Dec 12, 2010)

Thanks UNIXgod. I'll review your post a few times, decide what to do and then post what I've done.


----------



## wblock@ (Dec 12, 2010)

carlton_draught said:
			
		

> Now, my first choice: do I wuss out and just do a delta (basically a new file specifying to include GENERIC) or do I go through and erase everything that does not apply? Decisions, decisions. Advice?



One of the nice things about including GENERIC and using nooptions/nodevice to turn things off is that the config file becomes really a list of just the stuff you care about either disabling or enabling.  

Another feature is that your config automatically includes new GENERIC features and settings.  A full standalone config file has to be checked periodically against GENERIC to keep from missing those changes.


----------



## UNIXgod (Dec 12, 2010)

carlton_draught said:
			
		

> Thanks UNIXgod. I'll review your post a few times, decide what to do and then post what I've done.



I put alot of information into that post. Getting a slim kernel first is most important to your immediate needs. 

Worry about LINT as you need support for more exotic services. Also most of everything you need for desktop usage should be loaded as a module via /boot/loader.conf and controlled by /etc/rc.conf and /etc/sysctl.conf

The usage of _no_ vs just deleting or hash marks is up to you. wblock has a point as in version releases are upgraded GENERIC will change to keep compatibility with ports and add new device drivers.

My policy has been to keep a copy of the old generic and diff it against the new one when hitting point releases. I don't really have that many machines to worry about so my way of doing it may be a modus vivendi.

As with everything in the open source world. The choice and policy is yours to decide.


----------



## carlton_draught (Dec 13, 2010)

Before I begin in earnest, I will address some things UNIXgod and wblock have said.



			
				UNIXgod said:
			
		

> My policy has been to keep a copy of the old generic and diff it against the new one when hitting point releases. I don't really have that many machines to worry about so my way of doing it may be a modus vivendi.
> 
> As with everything in the open source world. The choice and policy is yours to decide.


I will certainly try your way first, as you have offered to help me in this thread and it would be untoward of me not to accept that help. I may also try it wblock's way as well and get the kernel loading time difference between the two. After that I will be in a position to decide which way I want to go in future.



> Your kernel name should reflect your machines name (if you haven't thought of one here are some suggestions I use character names from hitchhikers guide to the galaxy book and other sci-fi references(i.e. 2001: space odyssey) . I have seen people use lord of the rings references. Don't name it after a living pet or dead rock stars =) )


I used to do that sort of thing but I started making things more generic. e.g. "WORKSTATION", "LAPTOP" etc. If I might need to have more of the same thing, I would add a number on the end (e.g. "SERVER1", "SERVER2" etc.) I know, it's not very exciting, or memorable, but somehow I remember them. But I will say that 2001 is one of my favorite movies. I was considering making the "success" beep on zxfer be the famous bars from "Thus Spoke Zarathustra". And then decided against it, as it is too long and would get annoying fast.



> (I'm a vi user so flying through the file pressing dd is simpler than insert + hashmark everywhere)


I first learned vi/vim about 10 years ago or so. It is the ultimate text editor, IMO. The time invested in learning it pays itself back over, and over, and over again. That being said, if you wanted to insert a hashmark at the beginning of a line it would be a fairly simple matter of recording a macro in vim with q, then doing @@ after the first @(macro assignment key). Deleting things would make the file more simple though.


----------



## UNIXgod (Dec 13, 2010)

carlton_draught said:
			
		

> I will certainly try your way first, as you have offered to help me in this thread and it would be untoward of me not to accept that help. I may also try it wblock's way as well and get the kernel loading time difference between the two. After that I will be in a position to decide which way I want to go in future.



I will help you regardless. Both methods will end with the same result. I'm pretty easy going and realize that their are many ways to accomplish the same goal.



			
				carlton_draught said:
			
		

> I first learned vi/vim about 10 years ago or so. It is the ultimate text editor, IMO. The time invested in learning it pays itself back over, and over, and over again. That being said, if you wanted to insert a hashmark at the beginning of a line it would be a fairly simple matter of recording a macro in vim with q, then doing @@ after the first @(macro assignment key). Deleting things would make the file more simple though.



Thanks for the tip. I'll look into that later. Sounds like a good thing to know about vim.


----------



## carlton_draught (Dec 14, 2010)

UNIXgod said:
			
		

> I will help you regardless. Both methods will end with the same result. I'm pretty easy going and realize that their are many ways to accomplish the same goal.


Thanks.


> Thanks for the tip. I'll look into that later. Sounds like a good thing to know about vim.


Oh yes, definitely. To record a vim macro: 


Type "q", then a character assigned to executing that macro (e.g. "a")
Record what you want done at that point. For example, "i#<ESC>j"
Press "q" again. 
To execute once press "@a".
To repeat, press "@@".

Also another thing you can do is simply insert a "#" once, and press "." to do the same action. The macro allows you to do things like making the cursor go down the next line ready to repeat again, enabling you to just hold down the "@" if you so choose.

It is *amazing* what can be done with vim macros, especially with lots of SQL statements, or any structured text file you need to edit in a hurry.


----------



## wblock@ (Dec 14, 2010)

carlton_draught said:
			
		

> I may also try it wblock's way as well and get the kernel loading time difference between the two.



There isn't any loading time difference.  "include GENERIC" happens at build time.  There probably isn't a building speed difference, either.  It's like program source, you can put it all in one big file or use multiple files and include statements for better organization.


----------



## carlton_draught (Dec 14, 2010)

I'll be copying, pasting and annotating in chronological order, so that people who follow get an idea of the order in which to do things, and don't have to flip back and forth. (I'll probably be following along at a future date.)


			
				UNIXgod said:
			
		

> If you would like a easy grep-able file to see all the options without comments you can type
> 
> 
> ```
> ...


Done. These two files now exist.

I just noticed that LINT has 847 lines while GENERIC has 325 lines, some of which are comments and white space. All in all, only 220 lines actually do something. Not so bad. Now I'll proceed through the file "WORKSTATION".


```
cpu  hammer
```
hammer is amd64, basically, whether it's Intel or AMD (mine is Intel).


```
ident  WORKSTATION
```
(Name of my machine.)


```
makeoptions    DEBUG=-g"
```
removed as per UNIXgod's suggestion.

Now I'm following along with 8.6 The Configuration File.


> options SCTP # Stream Control Transmission Protocol


is the first thing not listed. UNIXgod has not mentioned it. Ok, so there are two places to check this, the NOTES file in the same directory as WORKSTATION, and /usr/src/sys/conf/NOTES.

```
SCTP is a NEW transport protocol defined by...
```
Well, none of this means anything to me. Searched google:

```
site:forums.freebsd.org sctp kernel
```
The following link makes it sound like a good idea to include. Next.


----------



## carlton_draught (Dec 14, 2010)

After progressing through a bit, I've decided to put each entry that is not in the handbook, and may either be of interest to someone, or I am not 100% sure about. The goal is to prompt discussion if I'm wrong, and serve as help to someone else or myself in future. "options" or "device" will be omitted for sake of brevity.


```
MD_ROOT
```
hmmm.


			
				UNIXgod said:
			
		

> Since you mentioned security is a concern. Removing anything with the line root at the top of the file is safe to remove unless you know you will need it. (i.e. MD_ROOT, NFS_ROOT and so on)


Ok, done. At this point, I've decided to take my own advice and simply comment out lines I don't use. Vim highlights the commented lines as blue, so it's easy to pick out what to focus on, while allowing the ability to back it out later. Continuing (making notes as to what I comment out, this is what is meant by "remove").


```
NFS...
```
 - removing

```
COMPAT_FREEBSD...
```
 Removing. I've never run anything from FreeBSD 7 or earlier, and I use ports for everything.


```
SCSI_DELAY...
```
 Removing. Will be prepared to put it straight back in if my HDDs are not detected.


```
KTRACE
```
 - Leaving in, don't use it AFAIK, but may need it?


```
STACK...
```
? not in handbook. Assume it goes with KTRACE and leave in.


```
P1003_1B_SEMAPHORES
```
? Nothing in handbook, one of the NOTES says that it is "very experimental". Guessing it's related to the next one, and leaving it in.


```
_KPOSIX_PRIORITY_SCHEDULING
```
  - "Certain applications in the Ports Collection use these", stays in.


```
PRINTF_BUFR_SIZE=128
```
 - not in handbook or either NOTES. Googling as before suggests this. No help for this but interesting. wblock talks about it here. Leaving it in just in case.


```
HWPMC_HOOKS
```
 - not in handbook or NOTES. 
`$ man hwpmc` leaves me none the wiser. Google. zeiz askes same question. This thread suggests to remove it. I'm going to leave it in and see if it doesn't give me problems, as I don't really understand what it does. Perhaps someone can help. Going to take a break here.


----------



## UNIXgod (Dec 14, 2010)

carlton_draught said:
			
		

> Thanks.
> 
> Oh yes, definitely. To record a vim macro:
> 
> ...



Hey man thanks for this. Pretty much opens up a whole new way of using the editor for me as I never took the time to learn vim recording mode. I plan on a vim hacks thread one of these days. Get all of us to share and learn from each other. I ran across some crazy thing the other day that had some drop down menu built inside vim that was able to complete ruby syntax.

On Topic. You also don't need IPV6 (INET6) in your kernel (or any of your ports for that matter)


----------



## carlton_draught (Dec 14, 2010)

UNIXgod said:
			
		

> Hey man thanks for this. Pretty much opens up a whole new way of using the editor for me as I never took the time to learn vim recording mode. I plan on a vim hacks thread one of these days. Get all of us to share and learn from each other. I ran across some crazy thing the other day that had some drop down menu built inside vim that was able to complete ruby syntax.


No problem. Key to using vim macros is to have the macro end up at the next line, and also to use the "move a word" keys when necessary, e.g. "w" and "e", though you've probably already guessed this. It has enabled creating things that would have taken me days otherwise, or I'd have given up. But because the macro is played back identically each time, if you get it right it eliminates the potential for syntax error which is a risk with manually editing text. So faster and more accurate to boot.

It is surprising what is out there. I guess since vim is probably the most popular FOSS editor out there for developers, it stands to reason that it is going to be hacked to do what people want. It would be interesting to read the thread.


			
				UNIXgod said:
			
		

> On Topic. You also don't need IPV6 (INET6) in your kernel (or any of your ports for that matter)


I kind of have a hankering to include it. I should probably not be so idealistic. I know that IPV6 is unlikely to be needed any time soon, but I hear how we are all going to need it soon enough and if only everyone adopted it world peace would break out or something.


----------



## carlton_draught (Dec 15, 2010)

Continuing where we left off...


```
AUDIT
```
 Left in. The only thing said anywhere about it is "#support for BSM audit" in the NOTES, whatever that means. Googling doesn't really help either.


```
MAC # TrustedBSD MAC Framework
```
 Left in. "Support for Mandatory Access Control", in NOTES. Google gives this. Not really sure whether it is used or not.


```
FLOWTABLE # per-cpu routing cache
```
 Left in, suspect important for multiple core machines. Apparently some problems with it. Will monitor to see if I ever get 1 core going to 100% for no reason.


```
INCLUDE_CONFIG_FILE # Include this file in kernel
```
. "Allows you to actually store this config file into the kernel binary itself. See config(8) for more details." Why not, I'll keep it.

Ok, from here it is pretty easy until I get to SCSI. I note that in my dmesg output, it looks like the USB drives use SCSI, and also my SAS/SATA controller devices come up as SCSI, including. Let's have a guess:
`$ dmesg | grep SCSI`

```
Waiting 5 seconds for SCSI devices to settle
(probe0:mpt0:0:0:0): CAM Status: SCSI Status Error
(probe0:mpt0:0:0:0): SCSI Status: Check Condition
da0: <ATA INTEL SSDSA2M040 02HB> Fixed Direct Access SCSI-5 device 
da1: <ATA WDC WD2500JD-00H 2D08> Fixed Direct Access SCSI-5 device 
da2: <ATA WDC WD2500JD-00H 2D08> Fixed Direct Access SCSI-5 device 
cd0: <ASUS DRW-22B2ST 1.00> Removable CD-ROM SCSI-0 device SMP: AP CPU #3 Launched!
umass0:  SCSI over Bulk-Only; quirks = 0x0000
da3: <TDKMedia Trans-It Drive PMAP> Removable Direct Access SCSI-0 device 
umass1:  SCSI over Bulk-Only; quirks = 0x0000
da4: <Verbatim STORE N GO 5.00> Removable Direct Access SCSI-0 device
```
Guessing that I can comment out all the other SCSI controllers but mpt.

Now we come to:

```
# SCSI peripherals
```
I'm going to leave the following: 

```
scbus
da
cd
```
Hopefully this is enough for my HDDs, DVD/CDROM and USB drive access.

Then 

```
# RAID Controllers interfaced to the SCSI subsystem
```
Going to comment out the LSI, as I know the device is actually "ASUS PIKE 1068E LSI Logic Fusion-MPT", as a google search for "ASUS 1068E LSI" brings that up, so it should be enough. And I can't see the string "mfi" when I grep the dmesg for SCSI.

Ok, next. Let's be realistic, I'm actually editing the dmesg output on another computer, so here is what I type:
`$ 
cat dmesg.txt | grep atk`

```
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
```
So I leave both the following in:

```
adkbdc
atkbd
```

Reading through The Configuration File, I decide to remove:

```
psm
kbdmux
splash
agp
cbb
pccard
cardbus
```

No parallel port, so removed all directly under

```
# Parallel port
```

We come to:

```
# PCI Ethernet NICs.
```
I do the following, per UNIXgod's suggestion:
`$ ifconfig`
I see several devices, em0 and em1, so I comment out all but the following:

```
em
```

Wow, I feel so productive, lol. So many to easily comment out!

Now I'm up to:

```
# Wireless NIC cards
```
Since I am not using wireless on this PC (partly due to bad experiences), it's all getting commented out.


```
# Pseudo devices.
```
After consulting handbook, leaving all of these in.


```
bpf
```
Commented out as I use static IP.


```
# USB Serial devices
```
Did a grep of each to make sure in dmesg, couldn't find anything so commented all out.



```
# USB Ethernet
```
Don't have this, commenting out.


```
# USB Wireless
```
Ditto.


```
# Firewire
```
Ditto.

Finished! Just do a check to see what we've removed (how much non-commented, non-blank space lines remain):
`$ cat WORKSTATION | grep -v "^#" | grep -v "^$" | wc`

BTW the above is a useless use of cat. So sue me. :stud

72 lines for WORKSTATION, as compared to the 220 lines of GENERIC. Not bad. Let's see if this works.


----------



## UNIXgod (Dec 15, 2010)

*UNIXgod crosses fingers for carlton_draught*
:beergrin:e:beer​


----------



## carlton_draught (Dec 15, 2010)

Thanks UNIXgod!

Following on from:
*8.5 Building and Installing a Custom Kernel*

Within that, we are up to:
*Building a Kernel*

`# cd /usr/src`
`# make buildkernel KERNCONF=WORKSTATION`


```
ERROR: version of config(8) does not match kernel!
config version = 600007, version required = 600009
```
Googled error.

`# make buildworld`
Near 60 minutes later on a quad-core Xeon, it's coffee grabbing time for those who are partial to that sort of thing.

`# make buildkernel KERNCONF=WORKSTATION`
Finished in under 20 minutes.

`# make installkernel KERNCONF=WORKSTATION`
Very quick.

`# shutdown -r now`


----------



## carlton_draught (Dec 15, 2010)

Yuck. I'm getting an endless scrolling text. Not cool. I pressed <CTRL>C. It eventually ended with:

(snippet)

```
panic: page fault
cpuid = 0
kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address  = 0x18
fault code = supervisor read data, page not present...
```
 etc.


----------



## carlton_draught (Dec 15, 2010)

Ok, reboot, escape to loader prompt. Load old kernel. It's working.

Here is my kernel conf.


----------



## carlton_draught (Dec 15, 2010)

Let's just test it against default GENERIC.

`# cp GENERIC /root/kernels/WORKSTATION-GENERIC`
`# ln -s /root/kernels/WORKSTATION-GENERIC`
`# cd /usr/src`
`# make buildkernel KERNCONF=WORKSTATION-GENERIC`
`# make installkernel KERNCONF=WORKSTATION-GENERIC`

Ok, so that worked. Let's see, what did I inadvertently comment out I am now regretting...


----------



## carlton_draught (Dec 15, 2010)

Ok, I'm going through and removing bits that I've removed before, in order of most confidence I have to least (basically working in reverse order of file, as I'm more sure about devices than various options). To speed up the process, here is what I'm doing. Do as in the previous post, but instead of WORKSTATION-GENERIC, I'm calling it TEST.

After the kernel is built and installed, do the following:
`# cp /root/kernels/TEST /root/kernels/TEST.old`
Then edit TEST file with your favorite editor.

After you have built it once, here is how to build the kernel ultra-fast. See especially this post.

Our testing loop now looks like so:
Reboot.
It works? Remove some more stuff.
`# cp /root/kernels/TEST /root/kernels/TEST.old`
`# vim /root/kernels/TEST`
`# cd /usr/src`
`# make buildkernel KERNCONF=TEST KERNFAST=1`
`# make installkernel KERNCONF=TEST`
`# shutdown -r now`

Ok, I think I found my error. I said I was going to leave the following in, but accidentally commented it out.

```
HWPMC_HOOKS
```

Now I'm compiling (without the KERNFAST=1) TEST as WORKSTATION instead, having renamed WORKSTATION to WORKSTATION.old. Hopefully that works.

Well that's weird. Getting the same error as I first got. Now to check my WORKSTATION.


----------



## wblock@ (Dec 15, 2010)

devel/ccache can be very effective.  I haven't tried KERNFAST or putting /usr/obj in RAM.  With ccache, buildworld on this E8400 dual-core system is about 9 minutes.


----------



## UNIXgod (Dec 15, 2010)

You have a scsi device. you should have this in options:


```
SCSI_DELAY=5000
```

Also turn the one you caught on:

```
HWPMC_HOOKS
```

32-bit binary support is also a good thing to have:

```
COMPAT_FREEBSD32
```

USB_DEBUG can be turned off.

But I have a feeling the last couple are not why your getting the panic.


----------



## wblock@ (Dec 15, 2010)

It's easy to become fixated on only having what's necessary.  But it's a lot easier to get a working custom kernel by just removing unnecessary devices.  For example, look at SCSI controllers and Ethernet cards in GENERIC.  They're pretty easy to identify, and most people won't have more than a few models of either.  All the others can be removed.  Repeat the process, paring away devices and options until it breaks and you can't fix it* or boredom or diminishing returns set in.

* For bicycle repairs, I express this as "tighten it until you pull the threads out, then back it off a quarter-turn."


----------



## carlton_draught (Dec 15, 2010)

Just wondering if this is it... this dmesg excerpt:

```
ada1 at ahcich4 bus 0 target 0 lun 0
```
Would that mean I need to enable the following?

```
device ahc # AHA2940 and onboard AIC7xxx devices
```


----------



## carlton_draught (Dec 15, 2010)

wblock said:
			
		

> It's easy to become fixated on only having what's necessary.  But it's a lot easier to get a working custom kernel by just removing unnecessary devices.  For example, look at SCSI controllers and Ethernet cards in GENERIC.  They're pretty easy to identify, and most people won't have more than a few models of either.  All the others can be removed.  Repeat the process, paring away devices and options until it breaks and you can't fix it* or boredom or diminishing returns set in.


Yes indeed. I've removed the bulk of the entries and it is working still. Keeping on going.


> * For bicycle repairs, I express this as "tighten it until you pull the threads out, then back it off a quarter-turn."


Lol.


----------



## UNIXgod (Dec 15, 2010)

carlton_draught said:
			
		

> Just wondering if this is it... this dmesg excerpt:
> 
> ```
> ada1 at ahcich4 bus 0 target 0 lun 0
> ...



There is a man page for every device. Look at man ahc(4)() and see if that helps you.


----------



## wblock@ (Dec 16, 2010)

carlton_draught said:
			
		

> Just wondering if this is it... this dmesg excerpt:
> 
> ```
> ada1 at ahcich4 bus 0 target 0 lun 0
> ...



No, adaX on ahci is SATA.  ahci channel 4, I suppose.


----------



## wblock@ (Dec 16, 2010)

What I implied but didn't outright say in #26 was "But it's a lot easier to get a working custom kernel by *starting with GENERIC and* just removing unnecessary devices."

In other words, start with a known-good kernel config file and mercilessly hack bits out of it rather than start with an empty file.


----------



## carlton_draught (Dec 16, 2010)

wblock said:
			
		

> What I implied but didn't outright say in #26 was "But it's a lot easier to get a working custom kernel by *starting with GENERIC and* just removing unnecessary devices."
> 
> In other words, start with a known-good kernel config file and mercilessly hack bits out of it rather than start with an empty file.


Oh definitely. I never start from scratch with anything if I have a choice. Much easier to start with something that works (i.e. GENERIC) than try and put stuff in you think will work. You would never know what the extra addition would be that makes it work, but it's always the last thing you took away that broke it.


----------



## carlton_draught (Dec 16, 2010)

UNIXgod said:
			
		

> There is a man page for every device. Look at man ahc(4)() and see if that helps you.


Thanks. I realized after I read the man page that I had the following in /boot/loader.conf

```
ahci_load="YES"
```
which is (duh) the alternative to compiling it in the kernel. I think I prefer to do it that way (in loader.conf), because I like things to still work with GENERIC, if possible.

91 active lines of working kernel so far. So far so good.

Edit:
81 lines now. Difference between previous WORKSTATION config:

```
compat_freebsd32
hwpmc_hooks
ataraid
pass
ses
psm
kbdmux
splash
```
I think we've hit the point of diminishing returns.


----------



## UNIXgod (Dec 16, 2010)

looking at the list I know that splash can be removed. It's just a console based screensaver. basically it's McKusick's BSD Daemon picture moving around the screen.

kbdmux Is only necessary if you use more than one keyboard( strangely I love this hack), I have a laptop which I use a happy hacking keyboard and it works well there and I also have a system I watch vids on one side and program on the other. where I move the screen 180 degrees and the system has on keyboard on either side based on how I'm using the machine.

If you have a PS/2 mouse port leave psm in

I have no idea what ses is but I haven't owned any real scsi gear in a long time so I can't comment on this one.

if you plan on cd and dvd burning the handbook suggests pass, cd, scbus and ata with atapicam_load="YES" in loader.conf

ataraid is "fakeraid" if your motherboard supports it. It can be removed if you never plan on enabling it.

I'm gonna be gone for a couple days. I am interested in in the success of your project. It sounds like your pretty much there.


----------



## carlton_draught (Dec 17, 2010)

UNIXgod said:
			
		

> I'm gonna be gone for a couple days. I am interested in in the success of your project. It sounds like your pretty much there.


Thanks for the help. I'll have a look at it a bit later and maybe remove a few more lines, posting the final kernel config here. As you say, I'm pretty much there.

My immediate goal is to get gnome2 to finish compiling (just did!), so that I can live in gnome again rather than doing everything from 1 tiny console. Easier to cut and paste and stuff etc to forums. From there I'll hark back to my original goals:

Compile a custom kernel, and learn how to update that (and world I guess) so that I can do it regularly and easily, so as to have a properly secure and reliable system. Still need to document what is involved with that. And also figure out and document my own practice for how to cope with updates and new additions to GENERIC, so that I can keep the holy trinity of kernel, world and ports fully patched/updated, including on when to do each or how to keep abreast of when.
Get system able to boot from 2nd disk, which is apparently a problem with 8.1-RELEASE and ZFS root mirror setups. Test it.
Fix my broken gnome (no gnome panel). DONE!
Figure out how to properly backup and restore the updated root mirror (which probably involves figuring out the minimum set of directories/files to backup and the order in which to do so).


----------



## danbi (Dec 17, 2010)

Without going into much details about what options/devices in the kernel to leave/remove I would like to point out just few things:

1. There are times, when you MUST update the kernel AND world at the same time. These are documented in the /usr/src/UPDATING file. There are not many such cases recently, for in the past changes in the networking code for example would leave your without network if you update only the kernel. Make it a habit to check that file each time you update sources.

2. There is not much point to snapshot the root if you only are installing new kernel. The installkernel target will move the old kernel and modules to kernel.old. Of course, if you are going to iterate a lot, it's good idea to have safe working root anyway. But if things break, you will have to boot off some other media anyway 

3. Take a look at nextboot. Using nextboot is advisable for experiments like yours, where you add/remove random parts from the kernel.

4. There is no point in having separate place for kernel config files and having symlinks in /sys/{arch}/conf -- you can leave the config files in there -- nothing is going to overwrite those. Or, if it does, it will overwrite the files via the symlinks as well -- little value.

5. As you have multicore CPU, run (you said you have 4 core CPU):

`# make -j8 buildworld`
`# make -j8 buildkernel`

do not use -j for other targets!

6. It is always smart to do buildworld after changing your sources. It's also smart to do cleanworld before that 

There are of course zillion ways to shoot yourself in the leg. What I described is actually what is documented as 'The FreeBSD way' and you could imagine, for good reason.

PS: 
7. As you are playing with the kernel, you may wish to discover hardware that the GENERIC kernel didn't care about, but your system has present. The easiest and funny way is to load kernel modules, like

`# kldload ichsmb`

If this produces kernel output about new device(s) on the console, you have just hit one 
Study the man page and if you like/want it, add the respective load statement in /boot/loader.conf. Or, add the device to your kernel config.

Have fun!


----------



## carlton_draught (Dec 17, 2010)

danbi said:
			
		

> Without going into much details about what options/devices in the kernel to leave/remove I would like to point out just few things:
> 
> 1. There are times, when you MUST update the kernel AND world at the same time. These are documented in the /usr/src/UPDATING file. There are not many such cases recently, for in the past changes in the networking code for example would leave your without network if you update only the kernel. Make it a habit to check that file each time you update sources.


Thanks. Even though it upsets my desire (and sensibilities?) that everything possible should be automated, I am learning to read and not skim or scan both /usr/src/UPDATING and /usr/ports/UPDATING.



> 2. There is not much point to snapshot the root if you only are installing new kernel. The installkernel target will move the old kernel and modules to kernel.old. Of course, if you are going to iterate a lot, it's good idea to have safe working root anyway. But if things break, you will have to boot off some other media anyway


While that is true, at least I have written out procedures for that so that in the worst case I can get back what I once had. If you can do that, you feel more freedom to experiment (whether the fears are warranted or not), which is ultimately good IMO. And snapshots are easily destroyed, so no harm done.



> 3. Take a look at nextboot. Using nextboot is advisable for experiments like yours, where you add/remove random parts from the kernel.


Sweet! Just so people know, this is included in the system and not a port. Just type
`# man nextboot`



> 4. There is no point in having separate place for kernel config files and having symlinks in /sys/{arch}/conf -- you can leave the config files in there -- nothing is going to overwrite those. Or, if it does, it will overwrite the files via the symlinks as well -- little value.


I don't think anything is going to overwrite those files other than the admin. However, you might delete /usr/src in a fit of anger, which is why I did what is done in the first tip in that section of the handbook.


> 5. As you have multicore CPU, run (you said you have 4 core CPU):
> 
> `# make -j8 buildworld`
> `# make -j8 buildkernel`
> ...


Hey, thanks for that!



> 6. It is always smart to do buildworld after changing your sources. It's also smart to do cleanworld before that


Again, thanks for the tip.



> PS:
> 7. As you are playing with the kernel, you may wish to discover hardware that the GENERIC kernel didn't care about, but your system has present. The easiest and funny way is to load kernel modules, like
> 
> `# kldload ichsmb`
> ...


This is cool. I'm just playing around with a way to automate this discovery of devices. Hopefully I can post a script when done.

I've got LINT pared away to just the names of the modules, to be iterated through in a loop. However, I'm trying to do the following in order to see whether there is any output, and so far seeing nothing in some things I know should be there.

e.g.
`# kldunload snd_hda`
`# kldload snd_hda`
No output.
`# kldunload speaker`
`# kldload speaker`
Nothing. Any comments?


----------



## carlton_draught (Dec 18, 2010)

Following on from that last one:

```
#!/bin/sh
# By scrolling through your terminal output, you should see which kernel module
# load detects some hardware on your system you didn't know about/enable.
ARCH=amd64
# Note that LINT includes everything that is NOT in GENERIC
LINT_DEVICES=$(cat /sys/$ARCH/conf/LINT | grep "^device" | cut -f 3 | sort | uniq)
for device in $LINT_DEVICES
do
  echo "$device"
  kldload "$device"
  sleep 1
  echo
done
```
Note that I did NOT find anything I did not have already enabled. Not a great surprise, as I have a server/workstation mobo and I'm basically using what I want to already. This may be of use to someone.


----------



## carlton_draught (Dec 18, 2010)

UNIXgod said:
			
		

> looking at the list I know that splash can be removed. It's just a console based screensaver. basically it's McKusick's BSD Daemon picture moving around the screen.
> 
> kbdmux Is only necessary if you use more than one keyboard( strangely I love this hack), I have a laptop which I use a happy hacking keyboard and it works well there and I also have a system I watch vids on one side and program on the other. where I move the screen 180 degrees and the system has on keyboard on either side based on how I'm using the machine.
> 
> ...


Ok, I'm commenting out ataraid and splash, the others can stay. Here is what I'm doing now:
`# cp /root/kernels/WORKSTATION /root/kernels/WORKSTATION.works_20101218`
`# make -j8 buildkernel KERNCONF=WORKSTATION`
And while we are at it:
`# make cleanworld`
`# make -j8 buildworld`
Next time, boot the system using the new kernel without checking to see if it exists.
`# nextboot -f -k kernel`
Reboot.
`# shutdown -r now`


----------



## carlton_draught (Dec 18, 2010)

Well, that worked. I've pared it down as far as I want to. Now to tick the rest of the items off the todo list. For those interested, here is the final configuration file, WORKSTATION.


----------



## carlton_draught (Mar 18, 2011)

Considering that I've had a several month gap in being able to work on anything FreeBSD, I've consequently forgotten a lot of this stuff, or at least, it has grown vague. I've spent a few hours reading through this thread, the handbook, various other posts on the forum etc, to try and understand where I was and where I need to go.

So now I'm in a position where I want to upgrade my system to 8.2-RELEASE, and I have a custom kernel (which IIRC was built primarily because I wanted to include some patches that fixed some ZFS issues). I thought I might continue documenting that process in this thread, for my own benefit and the benefit of others hopefully.

Upgrading your system with a custom kernel to a new RELEASE - a walkthrough

One thing I needed to make clear (correct me if I'm wrong) was that you can have a custom kernel and still follow RELEASE (rather than STABLE or CURRENT). This seems to be backed up here. i.e. - the fact that you can upgrade from source indicates that this should be possible.

Such a system that tracks RELEASE will still receive security fixes. e.g.


> While it is true that security fixes also go into the FreeBSD-STABLE branch, you do not need to track FreeBSD-STABLE to do this. Every security advisory for FreeBSD explains how to fix the problem for the releases it affects [1], and tracking an entire development branch just for security reasons is likely to bring in a lot of unwanted changes as well.



So it looks like I will need to first sync my sources, and then make/rebuild world, so to speak. I'll research that and then begin updating this thread.


----------



## carlton_draught (Mar 19, 2011)

Make a backup

Before we do anything, it's a good idea to backup your system. So I'm doing it here (using ZFS pools). I'll first make recursive snapshots on zroot and storage, so I can rollback if necessary.

`# zfs snapshot -r zroot@before_updating20110318`
`# zfs snapshot -r storage@before_updating20110318`

I'm backing up my storage pool to a HDD in an e-SATA HDD dock. I backup my SSD root mirror zroot to storage regularly, so all I need to do is backup what's on storage.

Insert the HDD, which has had a pool created on it called backup06, and two filesystems, backup06/filesystems and backup06/pools. Turn it on, and the HDD is detected by FreeBSD 8.1.
`# zpool import backup06`

Now here's where I show off the script in my sig (which I really need to get included into ports, but one thing at a time). To backup everything including snapshots and properties, and force the destination to have 2 copies to guard against bad sectors, and lzjb compression in order to reduce the size, all while allowing restore of the original properties easily, all I do is the following (whether it is the first time, second time, or nth time):
`# zxfer -dFkPvb -o copies=2,compression=lzjb -N storage/home backup06/filesystems`
`# zxfer -dFkPvb -o copies=2 -N storage/distfiles backup06/filesystems`
`# zxfer -dFkPvb -o copies=2 -N storage/packages backup06/filesystems`
`# zxfer -dFkPvb -o copies=2,compression=lzjb -R storage/zrootbackup backup06/pools`

Note that I could probably just do:
`# zxfer -dFkPvb -o copies=2,compression=lzjb -R storage backup06/pools`
Not sure why I'm not, to be honest. I will figure that out when I finish my guide to installing, backing up and restoring a FreeBSD root mirror system.

Export the backup pool, and turn the HDD dock off.
`# zpool export backup06`

Syncing my sources

Before we can build from source, we need to get that source from somewhere. I'll check the handbook. Note that I'm somewhat following on from dcbdbis's post as well.

I am going to be using csup, which is a faster reimplementation of net/cvsup. We are using it because

it's efficient in terms of only getting the files we need compared to anonymous CVS
using email, and PGP to prevent forged CTM deltas seems complicated. I also can't really find anything on it in the forums, googling 
	
	



```
site:forums.freebsd.org ctm
```
 comes up with nothing.
unlike net/cvsup, there are no dependencies.
If we use a host mirror, there is no load on FreeBSD core infrastructure.

The only potential downside I can see is that AFAIK using a local mirror means that you trust their checksums, which would have higher probability of being compromised than using the original FreeBSD site. It would be nice to get the checksums from there and fetch the other files locally, not sure if that is possible. Would appreciate any enlightenment about this from someone who knows more than I do.

So, let's follow the cvsup link (and we'll just use "csup" instead in commands).

We are going to start with the standard supfile, and modify if necessary.

`# cp /usr/share/examples/cvsup/standard-supfile /root/supfile_20110318`

Now edit /root/supfile_20110318.

Lines changed:

```
*default release=cvs tag=RELENG_8_2
*default host={local [url=http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/cvsup.html#CVSUP-MIRRORS]mirror[/URL]}
```

Now we run csup. No need to specify -g as csup doesn't appear to have a gui.

`# csup /root/supfile_20110318`

It took about 10 minutes or so to complete. Do it again, just for kicks, and you'll find that it should take about 5 seconds, if that. So we are finished syncing source. Now it's time to "rebuild".


----------



## carlton_draught (Mar 19, 2011)

Note that these are my notes to myself, I'll print them out and try and out, updating this post with what I should have done if I make a mistake. Edit: I've now finished, and fleshed out any bits that I had trouble with. In particular, the following post covers the mergemaster in detail, which was the trickiest for a newbie.

Rebuilding world


Check /usr/src/UPDATING.
`# vim /usr/src/UPDATING`
Nothing of particular note. Note that the mailing list for RELEASE appears not to exist, so checking UPDATING should be fine.

Check /etc/make.conf. Ok, did some googling, this thread was good. Seems the consensus is to not add anything to /etc/make.conf. (Perhaps the handbook needs updating, as they suggest a typical user wants to copy CFLAGS and NO_PROFILE lines.) So left as is.

Get to single user mode. I'm in gnome, so I need to get out of that first, and then get to single user mode.
`# /usr/local/etc/rc.d/gdm stop`
`# shutdown now`

Remove object files for a clean build.
`# cd /usr/obj`
`# chflags -R noschg *`
`# rm -rf *`

Make buildworld (using script to get a record of what we have done).
`# cd /usr/src`
`# script /root/make_buildworld_20110319.out`
(Note, we use -j8 to speed up the build process, I have a 4 core (8 with HT) CPU, so we use -j8.)
`# make -j8 buildworld`
This took 17 minutes.

Make and install the GENERIC kernel (don't worry, we will make our custom kernel later). Defaults to generic, so no need to put buildkernel KERNCONF=GENERIC.
`# make -j8 buildkernel` - this took 9 minutes.
`# make installkernel`

Make and install the custom kernel, in my case, WORKSTATION.
`# make -j8 buildkernel KERNCONF=WORKSTATION` - this took 6 minutes
`# make installkernel KERNCONF=WORKSTATION`

Backup /etc. Not really necessary, as we will have a snapshot to look at that if we need to. However, just to be needlessly ultra-anal,
`# cp -Rp /etc /etc.old`

Reboot the system, and at the boot prompt, select the â€œsingle userâ€ option (or option 6, "boot -s". The system will then boot single user. At the shell prompt you should then run:
`# fsck -p`
`# mount -u /`
`# mount -a -t ufs`
`# swapon -a`
Note we also need to mount all our ZFS file systems. (The handbook needs updating here.)
`# zfs mount -a`

Mergemaster. Read here for how to use. Search for "24.7.11.1".
`# mergemaster -p`

Install the system binaries.
`# cd /usr/src`
`# make installworld` - takes 1 minute

Mergemaster. Read here for how to use. Search for "24.7.11.1". Hmmm. This will be a good thing for the next post to cover. I see dcdbis has already trodden down this path before... Edit: Whoops. I really, really, really should use the options -Ui
`# mergemaster -Ui`

Reboot.
`# shutdown -r now`

Should now have an updated world and kernel.

Addendum

Check that the upgrade looks ok, and that you are running your custom kernel.
`# uname -iv`
Upgrade pools and ZFS file systems.
`# zpool upgrade -a`
`# zfs upgrade -a`
Strangely, the system crashed immediately after/during the zfs upgrade, but after a reboot it says everything is upgraded. A bit scary.
Upgrade bootcode on the root mirror. Used dmesg to determine device nodes, then issued the command that was given after the zpool upgrade. (not in my history unfortunately, due to crash).
Taken from here. Clean up and delete obsolete files, directories and libraries.
`# cd /usr/src`
`# make check-old`
`# yes|make delete-old`


----------



## carlton_draught (Mar 19, 2011)

Dealing with mergemaster -p
Ok, the first up is /etc/group.

I select "m" for merge, then "l" several times until done. Then "i" to install the merged file.

/etc/master.passwd
"m", "l" until finished, then "i".

Then it comes up and says:

```
You installed a new master.passwd file, so make sure that you run /usr/sbin/pwd_mkdb -p /etc/master.passwd to rebuild your password files. Would you like it to run now?
```
Hit "y".

Ok, that is it. It compares make variables from /etc/make.conf with /usr/src/share/examples/etc/make.conf and bails back to the command prompt.

I guess it's time to go to the next step... (previous post)

Dealing with mergemaster
Ok, now we have installed world, we are looking at...
./.cshrc
"r", "i", and, uh "l", for create a link.

.profile
"r", "l" (the "l" is for my changed path), "i", and "l"

COPYRIGHT
"i" (it's new, just changes the dates basically, so install the new one)

./boot/device.hints
"i" (it's new, just changes the dates basically, so install the new one)

./etc/amd.map
"i"

./etc/apmd.conf
"i"

./etc/auth.conf
"i"

... more files I know I have not touched, just hit "i"

./etc/crontab
"i" (thought this was the file generated by `# crontab -e`, must be wrong?

... more files I know I have not touched, just hit "i"

./etc/rc.conf
m,r,i    (I think this must be src/etc/defaults/rc.conf)


... more files I know I have not touched, just hit "i"

./etc/group
r,l,l,i

... more files I know I have not touched, just hit "i"

./etc/master.passwd
m,r,l,l,i


... more files I know I have not touched, just hit "i"


./etc/shells
m,r,l,i


... more files I know I have not touched, just hit "i", rebuild aliases database by typing "y", and several other things.

Finished.


----------



## wblock@ (Mar 19, 2011)

After a release, the version control strings have changed in many of those files.
`# mergemaster -Ui`
helps with that.


----------

