# SPARC64 network boot server and installation



## mhollis60 (Feb 5, 2018)

Just posting this in case it is useful to anyone (and for my future reference). I have access to a number of SPARC servers through my workplace and recently wanted to try setting up FreeBSD on them as Solaris is pretty much dead these days. I have SunFire V210, V240 and V245 that I'm using for basic file servers, DNS, DHCP in my lab. The kit is very old but still reliable and useful for basic tasks. I needed to setup a network boot server to get FreeBSD installed as I didn't have access to a local DVD on the clients.

Boot server host = SunFire V240
Boot clients = SunFire V210, V240, V245

On the boot server the following packages need to be installed and configured:

rarpd
dhcpd
nfs
tftpd


*TFTP setup*

Create a directory /tftpboot and make sure the tftpd server is started and running (edit /etc/inetd.conf and /etc/rc.conf to enable the service)


- unhash the following line in /etc/inetd.conf
`tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /tftpboot`

- add the following line to /etc/rc.conf
`inetd_enable="YES"`

- enable the inetd service and start
`service inetd enabled`
`service inetd restart`


The FreeBSD SPARC64 DVD is SCP'd to the boot server to a directory /tftpboot/iso. Create a /tftpboot/freebsd-installer directory and copy the contents of the FreeBSD 11.1 SPARC DVD to this directory by mounting the iso and using rsync to copy.

To mount an ISO on FreeBSD host:

`mdconfig -a -t vnode -f /tftpboot/iso/FreeBSD-11.1-RELEASE-sparc64-dvd1.iso -u 0`
`mount -t cd9660 /dev/md0 /mnt`
`cd /mnt`


Copy the DVD contents to /tftpboot/freebsd-installer directory:
`rsync -var /mnt/ /tftpboot/freebsd-installer/`


*Client hosts configuration on boot server*

Edit /etc/hosts and add entries for the clients:


```
10.1.1.46   spa     spa.example.lab
::1                     localhost localhost.my.domain
127.0.0.1               localhost localhost.my.domain

# FreeBSD clients for network booting
10.1.1.48   zolder
10.1.1.20   monaco
10.1.1.28   monza
10.1.1.36   hockenheim
```

In order for SPARC openboot to recognise the boot loader, translate the IP address of the boot client in decimal to hex and use this as a symbolic link target to freebsd-installer/boot/loader.

Example below:

0A010130 = 10.1.1.48
0A010114 = 10.1.1.20
0A01011C = 10.1.1.28
0A010124 = 10.1.1.36

Change to the /tftpboot directory and create the symlinks:

`cd /tftpboot`
`ln -s freebsd-installer/boot/loader 0A010130`
`ln -s freebsd-installer/boot/loader 0A010114`
`ln -s freebsd-installer/boot/loader 0A01011C`
`ln -s freebsd-installer/boot/loader 0A010124`


*RARPd setup*

Edit /etc/ethers and add specific lines mapping the client MAC addresses to IPs


```
root@spa:/tftpboot# cat /etc/ethers
00:14:4f:74:95:cc       spa
00:14:4f:71:04:20       zolder
00:03:ba:e9:d6:05       monaco
00:03:ba:eb:5a:11       monza
00:14:4f:7a:1c:fc       hockenheim
```

- add the following line to /etc/rc.conf
`rarpd_enable="YES"`

- enable the rarpd service and start
`service rarpd enabled`
`service rarpd restart`


*DHCP server setup*

DHCP server should be compiled from ports /usr/ports/net/dhcpd. Configuration is in /usr/local/etc/dhcpd.conf


```
# dhcpd.conf
#
# Sample configuration file for ISC dhcpd
#
authoritative;

# option definitions common to all supported networks...
option domain-name "example.lab";
option domain-name-servers 10.1.1.53;

default-lease-time 600;
max-lease-time 7200;


subnet 10.1.1.0 netmask 255.255.255.0 {
  range 10.1.1.200 10.1.1.250;
  option routers 10.1.1.1;
  option broadcast-address 10.1.1.255;
}

# Booting SPARC FreeBSD
group {
    always-reply-rfc1048 on;
    server-name "spa";
    next-server spa;
    filename "freebsd-installer/boot/loader";
    option root-path "10.1.1.46:/tftpboot/freebsd-installer/";

    host zolder { hardware ethernet 00:14:4f:71:04:20; option host-name "zolder"; fixed-address 10.1.1.48; }
    host spa { hardware ethernet 00:14:4f:74:95:cc; option host-name "spa"; fixed-address 10.1.1.46; }
    host monaco { hardware ethernet 00:03:ba:e9:d6:05; option host-name "monaco"; fixed-address 10.1.1.20; }
    host monza { hardware ethernet 00:03:ba:eb:5a:11; option host-name "monza"; fixed-address 10.1.1.28; }
    host hockenheim { hardware ethernet 00:14:4f:7a:1c:fc; option host-name "hockenheim"; fixed-address 10.1.1.36; }
}
```

- add the following line to /etc/rc.conf
`dhcpd_enable="YES"`

- enable the dhcpd service and start
`service dhcpd enabled`
`service dhcpd restart`


*NFS Server setup*

Edit /etc/exports and add the line to share the directory containing the contents of the FreeBSD DVD:
`/tftpboot/freebsd-installer -maproot=root -network 10.1.1.0 -mask 255.255.255.0`

- add the following line to /etc/rc.conf
`nfs_server_enable="YES"`
`nfs_server_flags="-u -t -n 4"`
`rpcbind_enable="YES"`
`mountd_flags="-r"`
`mountd_enable="YES"`

- enable and start the NFS server:
`service nfsd enabled`
`service nfsd restart`

Edit /tftpboot/freebsd-installer/etc/fstab and hash the line related to the DVD and add the mountpoint for the NFS. If you don't do this step the server will start to boot but will be unable to find the DVD on the NFS server (by default fstab points it to the DVD drive):


```
#/dev/iso9660/11_1_RELEASE_SPARC64_DVD / cd9660 ro 0 0
spa:/tftpboot/freebsd-installer      /mnt    nfs rw  0       0
```


*Network booting the client machine*

To network boot the SPARC client, first SSH to its service console (ALOM):

`sc> console -f`
`ok> boot net -v`

Wait until the server boots from the network. If it starts to boot from the disk rather than the network try the following commands from the SC:

`sc> bootmode bootscript="boot net"`
`sc> reset`
`sc> console -f`


----------



## mhollis60 (Feb 5, 2018)

I've also put the instructions below for network booting the client, basic installation steps and creating a root zpool with mirrored disks. 

*FreeBSD installation on SPARC*

The client should first get an IP address using RARP/DHCP. It will then pull the FreeBSD loader via TFTP and load the kernel. The remaining software is pulled from the NFS server. If successful, you get the prompt for the FreeBSD 11.1 installer:


```
Welcome to FreeBSD!

Please choose the appropriate terminal type for your system.
Common console types are:
   ansi     Standard ANSI terminal
   vt100    VT100 or compatible terminal
   xterm    xterm terminal emulator (or compatible)
   cons25w  cons25w terminal

Console type [vt100]: xterm
```


- Select shell from the menu. This will allow you to clean up the client system before installing FreeBSD 11.1 assuming a clean install.

**** WARNING THIS NEXT PART WILL DESTROY ANY EXISTING DATA ****
- Identify the disks, check for existing zpools (use zpool destroy if required), delete the exising partition table from the disk and delete the first few sectors to clear any existing bootcode from the disks (it assumes a server with 4x HDD):


```
# sysctl kern.disks
kern.disks: cd0 da3 da2 da1 da0
# zpool status
no pools available
# gpart destroy -F da0
da0 destroyed
# gpart destroy -F da1
da1 destroyed
# gpart destroy -F da2
da2 destroyed
# gpart destroy -F da3
da3 destroyed
# dd if=/dev/zero of=/dev/da0 bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes transferred in 6.075506 secs (84273 bytes/sec)
# dd if=/dev/zero of=/dev/da1 bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes transferred in 0.194581 secs (2631291 bytes/sec)
# dd if=/dev/zero of=/dev/da2 bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes transferred in 0.184106 secs (2781013 bytes/sec)
# dd if=/dev/zero of=/dev/da3 bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes transferred in 0.184583 secs (2773814 bytes/sec)
#
```

- Type exit to get back to the installer

- Type xterm to select the type of install console. Choose the option to "Install"

- Continue with default keymap

- Set the hostname for the client (should already be populated from DHCP)

- For distribution select just leave "ports" as the select option

- Partitioning menu -> Select Manual
   Select the first disk da0, click (c) to create and select VTOC8 partition table
   Repeat for the da1, da2 and da3 disks

   Go back to the first disk da0 and select (c) to create a new partition da0a. This time set the following options using tab to move around the menu:

`Type: freebsd-zfs
       Size: 60GB
       Mountpoint: /`

   Press enter when done. Select c on the da0 disk again this time making a partition for swap (da0b):

`Type: freebsd-swap
       Size: 8550MB
       Mountpoint: <leave this blank>`


   On the main partitioning menu select the second disk da1. Create a 60GB freebsd-zfs partition (da1a) as above but leave the mountpoint blank.

`Type: freebsd-zfs
       Size: 60GB
       Mountpoint: <leave this blank>`

   Create a swap partition (da1b) on the second disk da1:

`Type: freebsd-swap
       Size: 8550MB
       Mountpoint: <leave this blank>`

   On the main partitioning menu select the third disk da2. Create a freebsd-zfs partition (da2a) using all the space and leave the mountpoint blank.

`Type: freebsd-zfs
       Size: 68GB
       Mountpoint: <leave this blank>`

   On the main partition menu select the fourth disk da3. Create a freebsd-zfs partition (da3a) using all the space and leave the mountpoint blank.

`Type: freebsd-zfs
       Size: 68GB
       Mountpoint: <leave this blank>`


   Select the finish button to proceed. Select the "Commit" button to confirm the changes to the disks. The installer should write the bootcode and partition table to the disks.


- Fetching distribution, checksum and archive extraction should then proceed automatically.

- Set the root password

- Network configuration. Choose the interface to use, and confirm the IP address/DNS configurations. Most of this will be populated from the DHCP server even if your setting a static IP config.

- Choose the clock/timezone/date/time settings

- System configuration - leave defaults sshd and dumpdev to start at boot

- System hardening - leave defaults

- Add user accounts if required

- Choose exit, select no manual configurations and then choose reboot.



At this point drop back to the SC and run the following to boot off the disk rather than network:

`sc> bootmode bootscript="boot disk"
sc> reset
sc> console -f`


*Login and check the zpool*


```
root@monza:~ # zpool status
  pool: root
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        root        ONLINE       0     0     0
          da0a      ONLINE       0     0     0

errors: No known data errors
```



*Attach da1a to da0a to create a mirror of the boot disk*

Disk 0 and disk 1 are attached to make a mirror of the root filesystem. Resilvering (syncing) the disk takes a couple of minutes.


```
root@monza:~ # zpool attach -f root da0a da1a
root@monza:~ # zpool status
  pool: root
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Feb  5 14:59:09 2018
        440M scanned out of 795M at 24.5M/s, 0h0m to go
        440M resilvered, 55.38% done
config:

        NAME        STATE     READ WRITE CKSUM
        root        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0a    ONLINE       0     0     0
            da1a    ONLINE       0     0     0  (resilvering)

errors: No known data errors
```


*Expanding the pool*

Disk 2 and disk 3 can be added to the root ZFS pool to increase the overall size of the pool. The disks are mirrored and added to the pool to create a RAID-10 type setup
NOTE: when the second pair of disks have been added to the pool this can't be undone without losing data. A disk can be detached from a mirror though without data loss providing the partner disk remains part of the pool.


```
root@monza:~ # zpool add root mirror da2a da3a
root@monza:~ # zpool status
  pool: root
 state: ONLINE
  scan: resilvered 796M in 0h1m with 0 errors on Mon Feb  5 15:00:32 2018
config:

        NAME        STATE     READ WRITE CKSUM
        root        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0a    ONLINE       0     0     0
            da1a    ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            da2a    ONLINE       0     0     0
            da3a    ONLINE       0     0     0

errors: No known data errors
```


----------



## mhollis60 (Feb 5, 2018)

*Issues with booting the new system*

If the server fails to boot with an error like below the bootblock/loader may be missing from the disk.In this case I was able to successfully install FreeBSD 11.1, but the system didn't boot after exiting the installer and rebooting.


```
Boot device: /pci@1c,600000/scsi@2/disk@0,0  File and args:

>> FreeBSD/sparc64 ZFS boot block
   Boot path:   /pci@1c,600000/scsi@2/disk@0,0:a
Not an ELF file
Program terminated
```

In this case the bootblock/loader can be restored to the disks using the commands below (thanks to: https://wiki.freebsd.org/RootOnZFS/VTOC8ZFSBoot/9.x-RELEASE)
Examples below are for da0 and da1. If all four disks are members of the root pool then the bootblock and loader should be applied to all.


```
# gpart bootcode -p /boot/zfsboot da0
# gpart bootcode -p /boot/zfsboot da1
# sysctl kern.geom.debugflags=0x10
kern.geom.debugflags: 0 -> 16
# dd if=/boot/zfsloader of=/dev/da0a bs=512 oseek=1024 conv=notrunc,sync
465+1 records in
466+0 records out
238592 bytes transferred in 0.244906 secs (974219 bytes/sec)
# dd if=/boot/zfsloader of=/dev/da1a bs=512 oseek=1024 conv=notrunc,sync
465+1 records in
466+0 records out
238592 bytes transferred in 0.244906 secs (974219 bytes/sec)
```


----------

