# Altering rcorder and /etc/rc to gain faster startup



## unitrunker (Aug 6, 2020)

This is something I'd like to try this weekend. I'm putting it out here in case anyone sees an obvious flaw.

FreeBSD's RC init system manages services as a family of sh(1) scripts under /etc/rc.d/ and /usr/local/etc/rc.d/.

The /etc/rc script calls rcorder(8) to get the list of rc.d scripts to run in a safe order. The output of rcorder is one rc script name per line.

/etc/rc calls rcorder(8) here:


```
files=`rcorder ${skip} ${skip_firstboot} /etc/rc.d/* ${local_rc} 2>/dev/null`
for _rc_elem in ${files}; do
    case "$_rc_elem_done" in
    *" $_rc_elem "*)    continue ;;
    esac
    run_rc_script ${_rc_elem} ${_boot}
done
```

The stock /etc/rc script runs each RC service script one at a time. Very reliable but no concurrency.

The plan is to add very myopic parallel support by altering rc only. The altered rc will look at the PROVIDE: and REQUIRE: clauses to determine if two adjacent scripts can be run in parallel. Successive script names are added to a list called 'group' - up to a small limit is reached (say 5) - or until the next script cannot be run in parallel. This group of scripts is run in parallel. The 'wait' built-in waits for this script group to complete before going on to the next group. I'm looking to see what if any gain in start up time can be had with this lightweight amount of concurrency.


```
for _rc_elem in $group; do
    case "$_rc_elem_done" in
    *" $_rc_elem "*)    continue ;;
    esac
    run_rc_script $_rc_elem resume &
done
wait
```

If this works, I'd look at modifying rcorder (a new flag like '-p 5' to keep backward compatibility) to return multiple rc script names on the same line to indicate they can be run in parallel.

My sh(1) scripting is horrid so this will be interesting.

Edit: updated to say working on /etc/rc (and not /etc/rc.resume).


----------



## unitrunker (Aug 6, 2020)

Here's the output showing which rc scripts are eligible to run concurrently (as run on a Digital Ocean droplet):


```
Group:  /usr/local/etc/rc.d/digitaloceanpre /etc/rc.d/growfs /etc/rc.d/sysctl
Group:  /etc/rc.d/hostid
Group:  /etc/rc.d/zvol /etc/rc.d/dumpon
Group:  /etc/rc.d/ddb /etc/rc.d/geli /etc/rc.d/gbde /etc/rc.d/ccd
Group:  /etc/rc.d/swap
Group:  /etc/rc.d/fsck
Group:  /etc/rc.d/root
Group:  /etc/rc.d/mdconfig /etc/rc.d/hostid_save
Group:  /etc/rc.d/mountcritlocal
Group:  /etc/rc.d/zfsbe
Group:  /etc/rc.d/zfs /etc/rc.d/var
Group:  /etc/rc.d/cleanvar
Group:  /etc/rc.d/FILESYSTEMS
Group:  /etc/rc.d/ldconfig /etc/rc.d/kldxref
Group:  /etc/rc.d/kld
Group:  /etc/rc.d/addswap /etc/rc.d/adjkerntz /etc/rc.d/hostname /etc/rc.d/ip6addrctl /etc/rc.d/netoptions
Group:  /etc/rc.d/opensm /etc/rc.d/random /etc/rc.d/sppp /etc/rc.d/ippool /etc/rc.d/ipfilter
Group:  /etc/rc.d/ipnat
Group:  /etc/rc.d/ipfs /etc/rc.d/serial /etc/rc.d/iovctl
Group:  /etc/rc.d/netif
Group:  /etc/rc.d/devd /etc/rc.d/ipsec /etc/rc.d/pfsync /etc/rc.d/pflog
Group:  /etc/rc.d/pf /etc/rc.d/stf /etc/rc.d/ppp
Group:  /etc/rc.d/routing /etc/rc.d/ipfw
Group:  /etc/rc.d/netwait /etc/rc.d/resolv /etc/rc.d/defaultroute
Group:  /etc/rc.d/local_unbound /etc/rc.d/nsswitch /etc/rc.d/routed /etc/rc.d/rtsold /etc/rc.d/static_ndp
Group:  /etc/rc.d/static_arp /etc/rc.d/bridge /etc/rc.d/route6d
Group:  /etc/rc.d/NETWORKING
Group:  /etc/rc.d/mountcritremote
Group:  /etc/rc.d/accounting /etc/rc.d/newsyslog
Group:  /etc/rc.d/syslogd
Group:  /etc/rc.d/ntpdate
Group:  /etc/rc.d/rpcbind
Group:  /etc/rc.d/nfsclient /etc/rc.d/devfs /etc/rc.d/ipmon /etc/rc.d/kdc /etc/rc.d/mdconfig2
Group:  /etc/rc.d/watchdogd /etc/rc.d/savecore /etc/rc.d/archdep
Group:  /etc/rc.d/abi
Group:  /etc/rc.d/SERVERS
Group:  /etc/rc.d/nisdomain /etc/rc.d/ypserv
Group:  /etc/rc.d/ypbind
Group:  /etc/rc.d/ypset
Group:  /etc/rc.d/amd /etc/rc.d/auditd
Group:  /etc/rc.d/auditdistd /etc/rc.d/automountd
Group:  /etc/rc.d/automount /etc/rc.d/autounmountd /etc/rc.d/tmp
Group:  /etc/rc.d/cleartmp /etc/rc.d/ctld /etc/rc.d/dmesg /etc/rc.d/hastd /etc/rc.d/iscsid
Group:  /etc/rc.d/iscsictl /etc/rc.d/keyserv /etc/rc.d/nfsuserd /etc/rc.d/gssd /etc/rc.d/quota
Group:  /etc/rc.d/mountd
Group:  /etc/rc.d/nfsd
Group:  /etc/rc.d/statd
Group:  /etc/rc.d/lockd /etc/rc.d/pppoed /etc/rc.d/pwcheck /etc/rc.d/virecover /etc/rc.d/ypldap
Group:  /usr/local/etc/rc.d/uuidd /etc/rc.d/DAEMON
Group:  /etc/rc.d/apm /etc/rc.d/bootparams /etc/rc.d/hcsecd
Group:  /etc/rc.d/bthidd /etc/rc.d/local /etc/rc.d/lpd /etc/rc.d/motd /etc/rc.d/mountlate
Group:  /etc/rc.d/nscd /etc/rc.d/ntpd /etc/rc.d/powerd /etc/rc.d/rarpd /etc/rc.d/rctl
Group:  /etc/rc.d/sdpd
Group:  /etc/rc.d/rfcomm_pppd_server /etc/rc.d/rtadvd /etc/rc.d/rwho /etc/rc.d/timed /etc/rc.d/ugidfw
Group:  /etc/rc.d/utx /etc/rc.d/yppasswdd /etc/rc.d/LOGIN
Group:  /usr/local/etc/rc.d/znc /usr/local/etc/rc.d/rsyncd /usr/local/etc/rc.d/nginx /usr/local/etc/rc.d/digitalocean /etc/rc.d/zfsd
Group:  /etc/rc.d/ypxfrd /etc/rc.d/ypupdated /etc/rc.d/wpa_supplicant /etc/rc.d/ubthidhci /etc/rc.d/syscons
Group:  /etc/rc.d/swaplate /etc/rc.d/sshd /etc/rc.d/sendmail /etc/rc.d/cron /etc/rc.d/jail
Group:  /etc/rc.d/localpkg /etc/rc.d/securelevel /etc/rc.d/power_profile /etc/rc.d/othermta /etc/rc.d/nfscbd
Group:  /etc/rc.d/natd /etc/rc.d/msgs /etc/rc.d/moused /etc/rc.d/mixer /etc/rc.d/kpasswdd
Group:  /etc/rc.d/kfd /etc/rc.d/kadmind /etc/rc.d/ipropd_slave /etc/rc.d/ipropd_master /etc/rc.d/ipfw_netflow
Group:  /etc/rc.d/inetd /etc/rc.d/hostapd /etc/rc.d/gptboot /etc/rc.d/geli2 /etc/rc.d/ftpd
Group:  /etc/rc.d/ftp-proxy /etc/rc.d/dhclient /etc/rc.d/devmatch /etc/rc.d/cfumass /etc/rc.d/bsnmpd
Group:  /etc/rc.d/bluetooth /etc/rc.d/blacklistd /etc/rc.d/bgfsck
```


----------



## unitrunker (Aug 6, 2020)

... and here is the script used to generate that run ...


```
#!/bin/sh -x

# returns true if the two lists intersect.
# $1 is the name of the left list.
# $2 is the name of the right list.
intersect()
{
    _left_list=$1
    _right_list=$2
#    printf "Left: $_left_list\n"
#    printf "Right: $_right_list\n"
    for _left in $_left_list; do
        if [ "$_left" ]; then
            for _right in $_right_list; do
                if [ "$_right" ]; then
#                    printf "comparing $_left to $_right\n"
                    if [ "$_left" = "$_right" ]; then
#                        printf "match!\n"
                        return 0
                    fi
                fi
            done
        fi
    done
    return 1
}

print_group()
{
    printf "Group: "
    for _local in $1; do
        if [ "$_local" ]; then
            printf " $_local"
        fi
    done
    printf "\n"
    return 0
}

provides_list=""
span=0
group=
files=`rcorder /etc/rc.d/* /usr/local/etc/rc.d/* 2>/dev/null`
for _rc_elem in $files; do
       _requires=`grep '# REQUIRE: ' $_rc_elem | cut -c 12-`
       _provides=`grep '# PROVIDE: ' $_rc_elem | cut -c 12-`
    if intersect "$_requires" "$provides_list"; then
#        printf "Provides: $provides_list\n"
           print_group "$group"
           group="$_rc_elem"
           provides_list="$_provides"
           span=1
    elif [ $span = 5 ]; then
#        printf "Provides: $provides_list\n"
        print_group "$group"
        group="$_rc_elem"
        provides_list="$_provides"
        span=1
       else
           if [ ! "$provides_list" ]; then
               provides_list="$_provides"
        else
            provides_list="$provides_list $_provides"
        fi
        group="$group $_rc_elem"
           span=$(expr $span + 1)
       fi
done
print_group "$group"
```

There's not much to it. Next step would be to shoehorn this into /etc/rc and watch it totally break my system.


----------



## leebrown66 (Aug 6, 2020)

from rc(8):


```
Each script should contain rcorder(8) keywords, especially an appropriate
     "PROVIDE" entry, and if necessary "REQUIRE" and "BEFORE" keywords.
```

Looking at nisdomain, it contains:
# BEFORE: ypset ypbind ypserv ypxfrd

So it looks like your method needs to take BEFORE into account also.
Looks interesting and simple.


----------



## Mjölnir (Aug 7, 2020)

I'd like to suggest not to mimic in shell what make(1) already does well: calculate a tree of dependencies.
EDIT: make is _very_ mature & does this much faster because it's not interpreted.  So writing out the depencies _once_ can be done by a shell script, and a stripped down make(1) can process that, then file the resulting _groups_ back to your rc.resume script.
next EDIT: In your shell script, do not use printf(), but source in rc.subr(8) and use err(), warn(), info(), debug()


----------



## unitrunker (Aug 7, 2020)

Here is updated output that takes the BEFORE clause into account:


```
1 /usr/local/etc/rc.d/digitaloceanpre | hostid cloudinit digitalocean | FILESYSTEMS | digitaloceanpre
2 /etc/rc.d/growfs | sysctl |  | growfs
Group:  /usr/local/etc/rc.d/digitaloceanpre /etc/rc.d/growfs
1 /etc/rc.d/sysctl |  |  | sysctl
Group:  /etc/rc.d/sysctl
1 /etc/rc.d/hostid |  | sysctl | hostid
Group:  /etc/rc.d/hostid
1 /etc/rc.d/zvol | dumpon | hostid | zvol
Group:  /etc/rc.d/zvol
1 /etc/rc.d/dumpon | disks |  | dumpon
Group:  /etc/rc.d/dumpon
1 /etc/rc.d/ddb | disks | dumpon | ddb
Group:  /etc/rc.d/ddb
1 /etc/rc.d/geli |  |  | disks
2 /etc/rc.d/gbde |  |  | disks
3 /etc/rc.d/ccd |  |  | disks
Group:  /etc/rc.d/geli /etc/rc.d/gbde /etc/rc.d/ccd
1 /etc/rc.d/swap |  | disks | swap
Group:  /etc/rc.d/swap
1 /etc/rc.d/fsck |  | swap | fsck
Group:  /etc/rc.d/fsck
1 /etc/rc.d/root |  | fsck | root
Group:  /etc/rc.d/root
1 /etc/rc.d/mdconfig |  | swap root | mdconfig
2 /etc/rc.d/hostid_save |  | hostid root | hostid_save
Group:  /etc/rc.d/mdconfig /etc/rc.d/hostid_save
1 /etc/rc.d/mountcritlocal |  | root hostid_save mdconfig | mountcritlocal
Group:  /etc/rc.d/mountcritlocal
1 /etc/rc.d/zfsbe |  | mountcritlocal | zfsbe
Group:  /etc/rc.d/zfsbe
1 /etc/rc.d/zfs | FILESYSTEMS var | zfsbe | zfs
Group:  /etc/rc.d/zfs
1 /etc/rc.d/var |  | mountcritlocal | var
Group:  /etc/rc.d/var
1 /etc/rc.d/cleanvar |  | var | cleanvar
Group:  /etc/rc.d/cleanvar
1 /etc/rc.d/FILESYSTEMS |  | root mountcritlocal cleanvar | FILESYSTEMS
Group:  /etc/rc.d/FILESYSTEMS
1 /etc/rc.d/ldconfig | DAEMON | FILESYSTEMS | ldconfig
2 /etc/rc.d/kldxref | netif | FILESYSTEMS | kldxref
Group:  /etc/rc.d/ldconfig /etc/rc.d/kldxref
1 /etc/rc.d/kld |  | kldxref | kld
Group:  /etc/rc.d/kld
1 /etc/rc.d/addswap | netif | FILESYSTEMS kld | addswap
2 /etc/rc.d/adjkerntz | netif | FILESYSTEMS | adjkerntz
3 /etc/rc.d/hostname | netif | FILESYSTEMS | hostname
4 /etc/rc.d/ip6addrctl | netif | FILESYSTEMS | ip6addrctl
5 /etc/rc.d/netoptions | netif | FILESYSTEMS | netoptions
Group:  /etc/rc.d/addswap /etc/rc.d/adjkerntz /etc/rc.d/hostname /etc/rc.d/ip6addrctl /etc/rc.d/netoptions
1 /etc/rc.d/opensm | netif | FILESYSTEMS | opensm
2 /etc/rc.d/random | netif | FILESYSTEMS | random
3 /etc/rc.d/sppp | netif | root | sppp
4 /etc/rc.d/ippool | ipfilter | FILESYSTEMS | ippool
Group:  /etc/rc.d/opensm /etc/rc.d/random /etc/rc.d/sppp /etc/rc.d/ippool
1 /etc/rc.d/ipfilter |  | FILESYSTEMS | ipfilter
Group:  /etc/rc.d/ipfilter
1 /etc/rc.d/ipnat |  | ipfilter | ipnat
Group:  /etc/rc.d/ipnat
1 /etc/rc.d/ipfs |  | ipnat | ipfs
2 /etc/rc.d/serial |  | root | serial
3 /etc/rc.d/iovctl |  | FILESYSTEMS sysctl | iovctl
Group:  /etc/rc.d/ipfs /etc/rc.d/serial /etc/rc.d/iovctl
1 /etc/rc.d/netif |  | FILESYSTEMS iovctl serial sppp sysctl
hostid ipfilter ipfs | netif
Group:  /etc/rc.d/netif
1 /etc/rc.d/devd | NETWORKING mountcritremote | netif ldconfig | devd
2 /etc/rc.d/ipsec | DAEMON mountcritremote | FILESYSTEMS | ipsec
3 /etc/rc.d/pfsync |  | FILESYSTEMS netif | pfsync
4 /etc/rc.d/pflog |  | FILESYSTEMS netif | pflog
Group:  /etc/rc.d/devd /etc/rc.d/ipsec /etc/rc.d/pfsync /etc/rc.d/pflog
1 /etc/rc.d/pf | routing | FILESYSTEMS netif pflog pfsync | pf
2 /etc/rc.d/stf |  | netif | stf
3 /etc/rc.d/ppp |  | netif | ppp
Group:  /etc/rc.d/pf /etc/rc.d/stf /etc/rc.d/ppp
1 /etc/rc.d/routing |  | netif ppp stf | routing
2 /etc/rc.d/ipfw |  | ppp | ipfw
Group:  /etc/rc.d/routing /etc/rc.d/ipfw
1 /etc/rc.d/netwait |  | devd ipfilter ipfw pf routing | netwait
2 /etc/rc.d/resolv |  | netif FILESYSTEMS | resolv
3 /etc/rc.d/defaultroute |  | devd netif stf | defaultroute
Group:  /etc/rc.d/netwait /etc/rc.d/resolv /etc/rc.d/defaultroute
1 /etc/rc.d/local_unbound | NETWORKING | FILESYSTEMS defaultroute netwait resolv | local_unbound
2 /etc/rc.d/nsswitch | NETWORK | root | nsswitch
3 /etc/rc.d/routed | NETWORK | netif routing | routed
4 /etc/rc.d/rtsold | NETWORKING | netif | rtsold
5 /etc/rc.d/static_ndp |  | netif | static_ndp
Group:  /etc/rc.d/local_unbound /etc/rc.d/nsswitch /etc/rc.d/routed /etc/rc.d/rtsold /etc/rc.d/static_ndp
1 /etc/rc.d/static_arp |  | netif | static_arp
2 /etc/rc.d/bridge |  | netif ppp stf | bridge
3 /etc/rc.d/route6d |  | netif routing | route6d
Group:  /etc/rc.d/static_arp /etc/rc.d/bridge /etc/rc.d/route6d
1 /etc/rc.d/NETWORKING |  | netif netwait netoptions routing ppp ipfw stf
defaultroute route6d resolv bridge
static_arp static_ndp | NETWORKING NETWORK
Group:  /etc/rc.d/NETWORKING
1 /etc/rc.d/mountcritremote |  | NETWORKING FILESYSTEMS ipsec netwait | mountcritremote
Group:  /etc/rc.d/mountcritremote
1 /etc/rc.d/accounting | DAEMON | mountcritremote | accounting
2 /etc/rc.d/newsyslog |  | FILESYSTEMS mountcritremote | newsyslog
Group:  /etc/rc.d/accounting /etc/rc.d/newsyslog
1 /etc/rc.d/syslogd | SERVERS | mountcritremote FILESYSTEMS newsyslog netif | syslogd
Group:  /etc/rc.d/syslogd
1 /etc/rc.d/ntpdate |  | NETWORKING syslogd | ntpdate
Group:  /etc/rc.d/ntpdate
1 /etc/rc.d/rpcbind |  | NETWORKING ntpdate syslogd | rpcbind
Group:  /etc/rc.d/rpcbind
1 /etc/rc.d/nfsclient |  | NETWORKING mountcritremote rpcbind | nfsclient
2 /etc/rc.d/devfs | SERVERS securelevel | mountcritremote | devfs
3 /etc/rc.d/ipmon | SERVERS | FILESYSTEMS hostname sysctl ipfilter | ipmon
4 /etc/rc.d/kdc | SERVERS | NETWORKING | kdc
5 /etc/rc.d/mdconfig2 | SERVERS | mountcritremote | mdconfig2
Group:  /etc/rc.d/nfsclient /etc/rc.d/devfs /etc/rc.d/ipmon /etc/rc.d/kdc /etc/rc.d/mdconfig2
1 /etc/rc.d/watchdogd |  | FILESYSTEMS syslogd | watchdogd
2 /etc/rc.d/savecore |  | dumpon ddb syslogd | savecore
3 /etc/rc.d/archdep |  | mountcritremote | archdep
Group:  /etc/rc.d/watchdogd /etc/rc.d/savecore /etc/rc.d/archdep
1 /etc/rc.d/abi |  | archdep | abi
Group:  /etc/rc.d/abi
1 /etc/rc.d/SERVERS |  | mountcritremote abi ldconfig savecore watchdogd | SERVERS
Group:  /etc/rc.d/SERVERS
1 /etc/rc.d/nisdomain | ypset ypbind ypserv ypxfrd | SERVERS rpcbind | nisdomain
Group:  /etc/rc.d/nisdomain
1 /etc/rc.d/ypserv |  | rpcbind | ypserv
Group:  /etc/rc.d/ypserv
1 /etc/rc.d/ypbind | DAEMON | ypserv | ypbind
Group:  /etc/rc.d/ypbind
1 /etc/rc.d/ypset |  | ypbind | ypset
Group:  /etc/rc.d/ypset
1 /etc/rc.d/amd | DAEMON | rpcbind ypset nfsclient FILESYSTEMS ldconfig | amd
2 /etc/rc.d/auditd | DAEMON | syslogd | auditd
Group:  /etc/rc.d/amd /etc/rc.d/auditd
1 /etc/rc.d/auditdistd | DAEMON | auditd | auditdistd
2 /etc/rc.d/automountd | DAEMON | rpcbind ypset nfsclient FILESYSTEMS ldconfig | automountd
Group:  /etc/rc.d/auditdistd /etc/rc.d/automountd
1 /etc/rc.d/automount | DAEMON | nfsclient automountd | automount
2 /etc/rc.d/autounmountd | DAEMON | FILESYSTEMS | autounmountd
3 /etc/rc.d/tmp |  | mountcritremote | tmp
Group:  /etc/rc.d/automount /etc/rc.d/autounmountd /etc/rc.d/tmp
1 /etc/rc.d/cleartmp | DAEMON | mountcritremote tmp | cleartmp
2 /etc/rc.d/ctld | DAEMON | FILESYSTEMS | ctld
3 /etc/rc.d/dmesg | DAEMON | mountcritremote FILESYSTEMS | dmesg
4 /etc/rc.d/hastd | DAEMON | NETWORKING syslogd | hastd
5 /etc/rc.d/iscsid | DAEMON | NETWORK | iscsid
Group:  /etc/rc.d/cleartmp /etc/rc.d/ctld /etc/rc.d/dmesg /etc/rc.d/hastd /etc/rc.d/iscsid
1 /etc/rc.d/iscsictl | DAEMON | NETWORK iscsid | iscsictl
2 /etc/rc.d/keyserv | DAEMON | ypset | keyserv
3 /etc/rc.d/nfsuserd |  | NETWORKING | nfsuserd
4 /etc/rc.d/gssd |  | root | gssd
5 /etc/rc.d/quota | DAEMON | mountcritremote ypset | quota
Group:  /etc/rc.d/iscsictl /etc/rc.d/keyserv /etc/rc.d/nfsuserd /etc/rc.d/gssd /etc/rc.d/quota
1 /etc/rc.d/mountd |  | NETWORKING rpcbind quota | mountd
Group:  /etc/rc.d/mountd
1 /etc/rc.d/nfsd |  | mountcritremote mountd hostname gssd nfsuserd | nfsd
Group:  /etc/rc.d/nfsd
1 /etc/rc.d/statd | DAEMON | nfsclient nfsd rpcbind | statd
Group:  /etc/rc.d/statd
1 /etc/rc.d/lockd | DAEMON | nfsclient nfsd rpcbind statd | lockd
2 /etc/rc.d/pppoed | DAEMON | NETWORKING | pppoed
3 /etc/rc.d/pwcheck | DAEMON | mountcritremote syslogd | pwcheck
4 /etc/rc.d/virecover | DAEMON | mountcritremote ldconfig | virecover
5 /etc/rc.d/ypldap | DAEMON | ypserv | ypldap
Group:  /etc/rc.d/lockd /etc/rc.d/pppoed /etc/rc.d/pwcheck /etc/rc.d/virecover /etc/rc.d/ypldap
1 /usr/local/etc/rc.d/uuidd | DAEMON | FILESYSTEMS ldconfig | uuidd
Group:  /usr/local/etc/rc.d/uuidd
1 /etc/rc.d/DAEMON |  | NETWORKING SERVERS | DAEMON
Group:  /etc/rc.d/DAEMON
1 /etc/rc.d/apm | LOGIN | DAEMON | apm
2 /etc/rc.d/bootparams | LOGIN | rpcbind DAEMON | bootparams
3 /etc/rc.d/hcsecd | LOGIN | DAEMON | hcsecd
Group:  /etc/rc.d/apm /etc/rc.d/bootparams /etc/rc.d/hcsecd
1 /etc/rc.d/bthidd | LOGIN | DAEMON hcsecd | bthidd
2 /etc/rc.d/local | LOGIN | DAEMON | local
3 /etc/rc.d/lpd | LOGIN | DAEMON | lpd
4 /etc/rc.d/motd | LOGIN | mountcritremote | motd
5 /etc/rc.d/mountlate | LOGIN | DAEMON | mountlate
Group:  /etc/rc.d/bthidd /etc/rc.d/local /etc/rc.d/lpd /etc/rc.d/motd /etc/rc.d/mountlate
1 /etc/rc.d/nscd | LOGIN | DAEMON | nscd
2 /etc/rc.d/ntpd | LOGIN | DAEMON ntpdate FILESYSTEMS devfs | ntpd
3 /etc/rc.d/powerd | LOGIN | DAEMON | powerd
4 /etc/rc.d/rarpd | LOGIN | DAEMON FILESYSTEMS | rarpd
5 /etc/rc.d/rctl | LOGIN |  | rctl
Group:  /etc/rc.d/nscd /etc/rc.d/ntpd /etc/rc.d/powerd /etc/rc.d/rarpd /etc/rc.d/rctl
1 /etc/rc.d/sdpd | LOGIN | DAEMON | sdpd
Group:  /etc/rc.d/sdpd
1 /etc/rc.d/rfcomm_pppd_server | LOGIN | DAEMON sdpd | rfcomm_pppd_server
2 /etc/rc.d/rtadvd | LOGIN | DAEMON | rtadvd
3 /etc/rc.d/rwho | LOGIN | DAEMON | rwho
4 /etc/rc.d/timed | LOGIN | DAEMON | timed
5 /etc/rc.d/ugidfw | LOGIN | FILESYSTEMS | ugidfw
Group:  /etc/rc.d/rfcomm_pppd_server /etc/rc.d/rtadvd /etc/rc.d/rwho /etc/rc.d/timed /etc/rc.d/ugidfw
1 /etc/rc.d/utx | LOGIN | DAEMON FILESYSTEMS | utx
2 /etc/rc.d/yppasswdd | LOGIN | ypserv ypset | yppasswdd
Group:  /etc/rc.d/utx /etc/rc.d/yppasswdd
1 /etc/rc.d/LOGIN |  | DAEMON | LOGIN
Group:  /etc/rc.d/LOGIN
1 /usr/local/etc/rc.d/znc |  | LOGIN DAEMON | znc
2 /usr/local/etc/rc.d/rsyncd | securelevel | LOGIN | rsyncd
3 /usr/local/etc/rc.d/nginx |  | LOGIN cleanvar | nginx
4 /usr/local/etc/rc.d/digitalocean |  | FILESYSTEMS cloudinit digitaloceanpre | digitalocean
5 /etc/rc.d/zfsd |  | devd zfs | zfsd
Group:  /usr/local/etc/rc.d/znc /usr/local/etc/rc.d/rsyncd /usr/local/etc/rc.d/nginx /usr/local/etc/rc.d/digitalocean /etc/rc.d/zfsd
1 /etc/rc.d/ypxfrd |  | rpcbind ypserv | ypxfrd
2 /etc/rc.d/ypupdated |  | rpcbind ypserv | ypupdated
3 /etc/rc.d/wpa_supplicant |  | mountcritremote | wpa_supplicant
4 /etc/rc.d/ubthidhci | bluetooth | DAEMON | ubthidhci
5 /etc/rc.d/syscons |  | LOGIN | syscons
Group:  /etc/rc.d/ypxfrd /etc/rc.d/ypupdated /etc/rc.d/wpa_supplicant /etc/rc.d/ubthidhci /etc/rc.d/syscons
1 /etc/rc.d/swaplate |  | mountlate | swaplate
2 /etc/rc.d/sshd |  | LOGIN FILESYSTEMS | sshd
3 /etc/rc.d/sendmail |  | LOGIN FILESYSTEMS | mail
4 /etc/rc.d/cron | securelevel | LOGIN FILESYSTEMS | cron
5 /etc/rc.d/jail | securelevel | LOGIN FILESYSTEMS | jail
Group:  /etc/rc.d/swaplate /etc/rc.d/sshd /etc/rc.d/sendmail /etc/rc.d/cron /etc/rc.d/jail
1 /etc/rc.d/localpkg | securelevel | abi | localpkg
Group:  /etc/rc.d/localpkg
1 /etc/rc.d/securelevel |  | adjkerntz ipfw ipfilter pf | securelevel
2 /etc/rc.d/power_profile |  | FILESYSTEMS syslogd | power_profile
3 /etc/rc.d/othermta |  | LOGIN | mail
4 /etc/rc.d/nfscbd |  | NETWORKING nfsuserd | nfscbd
5 /etc/rc.d/natd |  |  | natd
Group:  /etc/rc.d/securelevel /etc/rc.d/power_profile /etc/rc.d/othermta /etc/rc.d/nfscbd /etc/rc.d/natd
1 /etc/rc.d/msgs |  | LOGIN | msgs
2 /etc/rc.d/moused |  | DAEMON FILESYSTEMS | moused
3 /etc/rc.d/mixer |  | FILESYSTEMS | mixer
4 /etc/rc.d/kpasswdd |  | kdc | kpasswdd
5 /etc/rc.d/kfd |  | NETWORK | kfd
Group:  /etc/rc.d/msgs /etc/rc.d/moused /etc/rc.d/mixer /etc/rc.d/kpasswdd /etc/rc.d/kfd
1 /etc/rc.d/kadmind |  | kdc | kadmind
2 /etc/rc.d/ipropd_slave |  | kdc | ipropd_slave
3 /etc/rc.d/ipropd_master |  | kdc | ipropd_master
4 /etc/rc.d/ipfw_netflow |  | ipfw | ipfw_netflow
5 /etc/rc.d/inetd |  | DAEMON LOGIN FILESYSTEMS | inetd
Group:  /etc/rc.d/kadmind /etc/rc.d/ipropd_slave /etc/rc.d/ipropd_master /etc/rc.d/ipfw_netflow /etc/rc.d/inetd
1 /etc/rc.d/hostapd |  | mountcritremote | hostapd
2 /etc/rc.d/gptboot |  | mountcritremote | gptboot
3 /etc/rc.d/geli2 |  | FILESYSTEMS | geli2
4 /etc/rc.d/ftpd |  | LOGIN FILESYSTEMS | ftpd
5 /etc/rc.d/ftp-proxy |  | DAEMON pf | ftp-proxy
Group:  /etc/rc.d/hostapd /etc/rc.d/gptboot /etc/rc.d/geli2 /etc/rc.d/ftpd /etc/rc.d/ftp-proxy
1 /etc/rc.d/dhclient |  |  | dhclient
2 /etc/rc.d/devmatch |  | kldxref | devmatch
3 /etc/rc.d/cfumass |  | var | cfumass
4 /etc/rc.d/bsnmpd |  | NETWORKING syslogd | bsnmpd
5 /etc/rc.d/bluetooth |  | DAEMON | bluetooth
Group:  /etc/rc.d/dhclient /etc/rc.d/devmatch /etc/rc.d/cfumass /etc/rc.d/bsnmpd /etc/rc.d/bluetooth
1 /etc/rc.d/blacklistd |  | netif pf | blacklistd
2 /etc/rc.d/bgfsck |  | cron devfs syslogd | bgfsck
Group:  /etc/rc.d/blacklistd /etc/rc.d/bgfsck
```

The "Group:" lines represent what rc scripts would run concurrently. The rest is just for debugging my weak sh(1) skills.


----------



## unitrunker (Aug 7, 2020)

Here's the updated script that extracts concurrent groups from the annotated RC scripts:


```
#!/bin/sh -x

# returns true if the two lists intersect.
# $1 is the name of the left list.
# $2 is the name of the right list.
intersect()
{
    _left_list=$1
    _right_list=$2
#    printf "Left: $_left_list\n"
#    printf "Right: $_right_list\n"
    for _left in $_left_list; do
        if [ "$_left" ]; then
            for _right in $_right_list; do
                if [ "$_right" ]; then
#                    printf "comparing $_left to $_right\n"
                    if [ "$_left" = "$_right" ]; then
#                        printf "match!\n"
                        return 0
                    fi
                fi
            done
        fi
    done
    return 1
}

print_group()
{
    printf "Group: "
    for _local in $1; do
        if [ "$_local" ]; then
            printf " $_local"
        fi
    done
    printf "\n"
    return 0
}

provides_list=""
before_list=""
group=""

grow_provides_list()
{
    if [ "$1" ]; then
        if [ "$provides_list" ]; then
            provides_list="$provides_list $1"
        else
            provides_list="$1"
        fi
    fi
#    printf "Provides: $provides_list\n"
}

grow_before_list()
{
    if [ "$1" ]; then
        if [ "$before_list" ]; then
            before_list="$before_list $1"
        else
            before_list="$1"
        fi
    fi
#    printf "Before: $before_list\n"
}

grow_group_list()
{
    if [ "$1" ]; then
        if [ "$group" ]; then
            group="$group $1"
        else
            group="$1"
        fi
    fi
#    printf "Group grow: $group\n"
}

span=0
files=`rcorder /etc/rc.d/* /usr/local/etc/rc.d/* 2>/dev/null`
for _rc_elem in $files; do
       _requires=`grep '# REQUIRE: ' $_rc_elem | cut -c 12- | sed 's/  */ /g' | sed 's/^ //g' `
       _provides=`grep '# PROVIDE: ' $_rc_elem | cut -c 12- | sed 's/  */ /g' | sed 's/^ //g' `
       _before=`grep   '# BEFORE: ' $_rc_elem  | cut -c 10- | sed 's/  */ /g' | sed 's/^ //g' `
    if intersect "$_requires" "$provides_list" || intersect "$before_list" $_provides || [ $span = 5 ]; then
#        printf "Provides: $provides_list\n"
           print_group "$group"
           group="$_rc_elem"
           provides_list="$_provides"
           before_list="$_before"
           span=1
       else
           grow_provides_list "$_provides"
           grow_before_list "$_before"
        grow_group_list $_rc_elem
           span=$(expr $span + 1)
       fi
       printf "$span $_rc_elem | $_before | $_requires | $_provides\n"
done
print_group "$group"
```

Again, this is not the replacement rc.resume script but represents progress towards creating one. Ultimately an enhanced rcorder would make most of this go away. Baby steps for now.


----------



## Mjölnir (Aug 7, 2020)

Now again: make(1) would not calculate the depency graph over and over again when none of the input (files) changed.  Given that this seldomly happens, you can save that output to /var/db and use it in rc.resume.  During development, set the shell flags _noclobber, errexit_ & _nounset_, comment these out when you're done.


----------



## unitrunker (Aug 7, 2020)

mjollnir said:


> Now again: make(1) would not calculate the dependency graph over and over again when none of the input (files) changed.


If you could, please provide an example Makefile that shows how to do this. I assume "make -j 5" to tell make to run multiple jobs at once. rc.resume already has a number of script variables. It's not clear to me how those variables would pass through the make command to the individual RC scripts (as opposed to one script directly calling another).

I like your suggestion but you need to spell it out.


----------



## unitrunker (Aug 7, 2020)

Taking your idea of a generated file - I can skip Makefile and generate a flat sh(1) file. The sh(1) file - calls out one RC script per line trailed by an ampersand. Occasional 'wait' lines to pause for the previous group to complete. This would not offer as high a degree of concurrency as make(1) but avoids some make related uncertainties.

Please keep the ideas coming!


----------



## Mjölnir (Aug 7, 2020)

When I was researching regarding this _nosh/OpenRC/runit_ thing, I stumbled over a FreeBSD webpage (wiki or projects page?), telling that there have been some successful _proof of concept_ attempts to parallelize _init/rc scripts_ by using make(1).  A Makefile describing service dependencies would look like

```
# /var/db/service.makefile
#
# Run this from e.g. /var/run/rc.d
# Each service started successfully is represented by a file of the service's name containing it's startup time.
# The PIDs are already in /var/run/service_name written by service(8).
# E.g.
# PROVIDE: ypbind
# REQUIRE: ypserv
# BEFORE:  DAEMON
# KEYWORD: shutdown
# translates to
ypbind.svc: _daemon.svc ypserv.svc
    service $@ start
    touch $@
    date -u +%F_%T_%Z > $@
```
Disclaimer: this is a quick & dirty hack / brainstorming; it can be total crap & not work at all; general rules like in the /usr/share/mk/* might be more elegant.  You might be right that these dependencies are so simple that it's perfectly ok to write them out once into a shell script that is beeing executed.  BUT: as always, 90% is done within 10% effort, the important parts are error checking and to define what to do if something fails...  and all this has already been solved by e.g. sysutils/runit (sysutils/runit-faster; also see `pkg search daemontools`) 

Unfortunately, the sh(1) manpage is somewhat oldschool/technical/not so easy to read.  You can set these knobs in a shell script like this:

```
#! /bin/sh

set -o errexit
set -o noclobber
set -o nounset
set -o xtrace
[...]
```
To pass variables from a shell script to a Makefile, you would use variables exported to the environ(7)ment (see also env(1)), or `make -Dvar=value`, or source in from a file like is done by build(7) via e.g. make.conf(5).


----------



## unitrunker (Aug 7, 2020)

Here's the flattened sh(1) script. This would replace the for-loop in rc.resume:


```
run_rc_script /usr/local/etc/rc.d/digitaloceanpre resume &
run_rc_script /etc/rc.d/growfs resume &
wait
run_rc_script /etc/rc.d/sysctl resume &
wait
run_rc_script /etc/rc.d/hostid resume &
wait
run_rc_script /etc/rc.d/zvol resume &
wait
run_rc_script /etc/rc.d/dumpon resume &
wait
run_rc_script /etc/rc.d/ddb resume &
wait
run_rc_script /etc/rc.d/geli resume &
run_rc_script /etc/rc.d/gbde resume &
run_rc_script /etc/rc.d/ccd resume &
wait
run_rc_script /etc/rc.d/swap resume &
wait
run_rc_script /etc/rc.d/fsck resume &
wait
run_rc_script /etc/rc.d/root resume &
wait
run_rc_script /etc/rc.d/mdconfig resume &
run_rc_script /etc/rc.d/hostid_save resume &
wait
run_rc_script /etc/rc.d/mountcritlocal resume &
wait
run_rc_script /etc/rc.d/zfsbe resume &
wait
run_rc_script /etc/rc.d/zfs resume &
wait
run_rc_script /etc/rc.d/var resume &
wait
run_rc_script /etc/rc.d/cleanvar resume &
wait
run_rc_script /etc/rc.d/FILESYSTEMS resume &
wait
run_rc_script /etc/rc.d/ldconfig resume &
run_rc_script /etc/rc.d/kldxref resume &
wait
run_rc_script /etc/rc.d/kld resume &
wait
run_rc_script /etc/rc.d/addswap resume &
run_rc_script /etc/rc.d/adjkerntz resume &
run_rc_script /etc/rc.d/hostname resume &
run_rc_script /etc/rc.d/ip6addrctl resume &
run_rc_script /etc/rc.d/netoptions resume &
wait
run_rc_script /etc/rc.d/opensm resume &
run_rc_script /etc/rc.d/random resume &
run_rc_script /etc/rc.d/sppp resume &
run_rc_script /etc/rc.d/ippool resume &
wait
run_rc_script /etc/rc.d/ipfilter resume &
wait
run_rc_script /etc/rc.d/ipnat resume &
wait
run_rc_script /etc/rc.d/ipfs resume &
run_rc_script /etc/rc.d/serial resume &
run_rc_script /etc/rc.d/iovctl resume &
wait
run_rc_script /etc/rc.d/netif resume &
wait
run_rc_script /etc/rc.d/devd resume &
run_rc_script /etc/rc.d/ipsec resume &
run_rc_script /etc/rc.d/pfsync resume &
run_rc_script /etc/rc.d/pflog resume &
wait
run_rc_script /etc/rc.d/pf resume &
run_rc_script /etc/rc.d/stf resume &
run_rc_script /etc/rc.d/ppp resume &
wait
run_rc_script /etc/rc.d/routing resume &
run_rc_script /etc/rc.d/ipfw resume &
wait
run_rc_script /etc/rc.d/netwait resume &
run_rc_script /etc/rc.d/resolv resume &
run_rc_script /etc/rc.d/defaultroute resume &
wait
run_rc_script /etc/rc.d/local_unbound resume &
run_rc_script /etc/rc.d/nsswitch resume &
run_rc_script /etc/rc.d/routed resume &
run_rc_script /etc/rc.d/rtsold resume &
run_rc_script /etc/rc.d/static_ndp resume &
wait
run_rc_script /etc/rc.d/static_arp resume &
run_rc_script /etc/rc.d/bridge resume &
run_rc_script /etc/rc.d/route6d resume &
wait
run_rc_script /etc/rc.d/NETWORKING resume &
wait
run_rc_script /etc/rc.d/mountcritremote resume &
wait
run_rc_script /etc/rc.d/accounting resume &
run_rc_script /etc/rc.d/newsyslog resume &
wait
run_rc_script /etc/rc.d/syslogd resume &
wait
run_rc_script /etc/rc.d/ntpdate resume &
wait
run_rc_script /etc/rc.d/rpcbind resume &
wait
run_rc_script /etc/rc.d/nfsclient resume &
run_rc_script /etc/rc.d/devfs resume &
run_rc_script /etc/rc.d/ipmon resume &
run_rc_script /etc/rc.d/kdc resume &
run_rc_script /etc/rc.d/mdconfig2 resume &
wait
run_rc_script /etc/rc.d/watchdogd resume &
run_rc_script /etc/rc.d/savecore resume &
run_rc_script /etc/rc.d/archdep resume &
wait
run_rc_script /etc/rc.d/abi resume &
wait
run_rc_script /etc/rc.d/SERVERS resume &
wait
run_rc_script /etc/rc.d/nisdomain resume &
wait
run_rc_script /etc/rc.d/ypserv resume &
wait
run_rc_script /etc/rc.d/ypbind resume &
wait
run_rc_script /etc/rc.d/ypset resume &
wait
run_rc_script /etc/rc.d/amd resume &
run_rc_script /etc/rc.d/auditd resume &
wait
run_rc_script /etc/rc.d/auditdistd resume &
run_rc_script /etc/rc.d/automountd resume &
wait
run_rc_script /etc/rc.d/automount resume &
run_rc_script /etc/rc.d/autounmountd resume &
run_rc_script /etc/rc.d/tmp resume &
wait
run_rc_script /etc/rc.d/cleartmp resume &
run_rc_script /etc/rc.d/ctld resume &
run_rc_script /etc/rc.d/dmesg resume &
run_rc_script /etc/rc.d/hastd resume &
run_rc_script /etc/rc.d/iscsid resume &
wait
run_rc_script /etc/rc.d/iscsictl resume &
run_rc_script /etc/rc.d/keyserv resume &
run_rc_script /etc/rc.d/nfsuserd resume &
run_rc_script /etc/rc.d/gssd resume &
run_rc_script /etc/rc.d/quota resume &
wait
run_rc_script /etc/rc.d/mountd resume &
wait
run_rc_script /etc/rc.d/nfsd resume &
wait
run_rc_script /etc/rc.d/statd resume &
wait
run_rc_script /etc/rc.d/lockd resume &
run_rc_script /etc/rc.d/pppoed resume &
run_rc_script /etc/rc.d/pwcheck resume &
run_rc_script /etc/rc.d/virecover resume &
run_rc_script /etc/rc.d/ypldap resume &
wait
run_rc_script /usr/local/etc/rc.d/uuidd resume &
wait
run_rc_script /etc/rc.d/DAEMON resume &
wait
run_rc_script /etc/rc.d/apm resume &
run_rc_script /etc/rc.d/bootparams resume &
run_rc_script /etc/rc.d/hcsecd resume &
wait
run_rc_script /etc/rc.d/bthidd resume &
run_rc_script /etc/rc.d/local resume &
run_rc_script /etc/rc.d/lpd resume &
run_rc_script /etc/rc.d/motd resume &
run_rc_script /etc/rc.d/mountlate resume &
wait
run_rc_script /etc/rc.d/nscd resume &
run_rc_script /etc/rc.d/ntpd resume &
run_rc_script /etc/rc.d/powerd resume &
run_rc_script /etc/rc.d/rarpd resume &
run_rc_script /etc/rc.d/rctl resume &
wait
run_rc_script /etc/rc.d/sdpd resume &
wait
run_rc_script /etc/rc.d/rfcomm_pppd_server resume &
run_rc_script /etc/rc.d/rtadvd resume &
run_rc_script /etc/rc.d/rwho resume &
run_rc_script /etc/rc.d/timed resume &
run_rc_script /etc/rc.d/ugidfw resume &
wait
run_rc_script /etc/rc.d/utx resume &
run_rc_script /etc/rc.d/yppasswdd resume &
wait
run_rc_script /etc/rc.d/LOGIN resume &
wait
run_rc_script /usr/local/etc/rc.d/znc resume &
run_rc_script /usr/local/etc/rc.d/rsyncd resume &
run_rc_script /usr/local/etc/rc.d/nginx resume &
run_rc_script /usr/local/etc/rc.d/digitalocean resume &
run_rc_script /etc/rc.d/zfsd resume &
wait
run_rc_script /etc/rc.d/ypxfrd resume &
run_rc_script /etc/rc.d/ypupdated resume &
run_rc_script /etc/rc.d/wpa_supplicant resume &
run_rc_script /etc/rc.d/ubthidhci resume &
run_rc_script /etc/rc.d/syscons resume &
wait
run_rc_script /etc/rc.d/swaplate resume &
run_rc_script /etc/rc.d/sshd resume &
run_rc_script /etc/rc.d/sendmail resume &
run_rc_script /etc/rc.d/cron resume &
run_rc_script /etc/rc.d/jail resume &
wait
run_rc_script /etc/rc.d/localpkg resume &
wait
run_rc_script /etc/rc.d/securelevel resume &
run_rc_script /etc/rc.d/power_profile resume &
run_rc_script /etc/rc.d/othermta resume &
run_rc_script /etc/rc.d/nfscbd resume &
run_rc_script /etc/rc.d/natd resume &
wait
run_rc_script /etc/rc.d/msgs resume &
run_rc_script /etc/rc.d/moused resume &
run_rc_script /etc/rc.d/mixer resume &
run_rc_script /etc/rc.d/kpasswdd resume &
run_rc_script /etc/rc.d/kfd resume &
wait
run_rc_script /etc/rc.d/kadmind resume &
run_rc_script /etc/rc.d/ipropd_slave resume &
run_rc_script /etc/rc.d/ipropd_master resume &
run_rc_script /etc/rc.d/ipfw_netflow resume &
run_rc_script /etc/rc.d/inetd resume &
wait
run_rc_script /etc/rc.d/hostapd resume &
run_rc_script /etc/rc.d/gptboot resume &
run_rc_script /etc/rc.d/geli2 resume &
run_rc_script /etc/rc.d/ftpd resume &
run_rc_script /etc/rc.d/ftp-proxy resume &
wait
run_rc_script /etc/rc.d/dhclient resume &
run_rc_script /etc/rc.d/devmatch resume &
run_rc_script /etc/rc.d/cfumass resume &
run_rc_script /etc/rc.d/bsnmpd resume &
run_rc_script /etc/rc.d/bluetooth resume &
wait
run_rc_script /etc/rc.d/blacklistd resume &
run_rc_script /etc/rc.d/bgfsck resume &
wait
```

This is starting to look brain-dead simple.


----------



## unitrunker (Aug 7, 2020)

mjollnir said:


> When I was researching regarding this _nosh/OpenRC/runit_ thing, I stumbled over a FreeBSD webpage (wiki or projects page?), telling that there have been some successful _proof of concept_ attempts to parallelize _init/rc scripts_ by using make(1).  A Makefile describing service dependencies would look like



Awesome! Very helpful. Thank you!


----------



## Mjölnir (Aug 7, 2020)

unitrunker said:


> Awesome! Very helpful. Thank you!


Is this ironic?  There's a serious bug in the Makefile I posted above...


----------



## unitrunker (Aug 7, 2020)

No irony. I've not had a chance to try this out.


----------



## unitrunker (Aug 8, 2020)

I was looking at the wrong rc script. '/etc/rc' is where I should be doing this.


----------



## unitrunker (Aug 8, 2020)

I could not figure out a way to get a Makefile to work inside /etc/rc. I also could not figure out how to use a generated file as /etc/rc has multiple use cases that make the list of scripts to run dynamic. So ... cut to the chase ... it works. 

Now I'm adding some logging to see if I've gotten a reduction on boot time.


----------



## Mjölnir (Aug 8, 2020)

Well, as I wrote above: All this has been tried & evaluated multiple times by others.  Issues to solve:

To enshure a service is _up_, it's not sufficient that a service is _running_ (PID exists):  It might need some additional time to do some housework, or a master task/thread might fire up worker tasks/threads.  The shell's _wait_ only checks the PID of the service's rc script to die.  This does not mean in all cases that the service is _up_?  To solve that reliably, a standardized communication between a service & it's rc script has to be introduced.  The service needs to tell the service manager it's status (_"I'm up & ready to serve"_).  MAYBE this is solved by convention: the service rc script does not return before the service is ready (_up_).  I just don't know.
I'm curious to read about your experience.


----------



## unitrunker (Aug 8, 2020)

I have a very surgical goal which as stated above - is a reduction in boot time by a modest amount of concurrency. I'm not trying to rewrite how RC works. In other words, make a small change that is easy to verify. Don't force the world to rewrite their init scripts. A year ago I read up on other init systems. DJB had some very good ideas - and others have built on those ideas. But that is a whole other discussion.

The focus of my attention is this for-loop in the /etc/rc script (similar to one in /etc/rc.resume):


```
files=`rcorder ${skip} ${skip_firstboot} /etc/rc.d/* ${local_rc} 2>/dev/null`
for _rc_elem in ${files}; do
    case "$_rc_elem_done" in
    *" $_rc_elem "*)    continue ;;
    esac
    run_rc_script ${_rc_elem} ${_boot}
done
```

The script may make multiple passes. The _rc_elem_done variable tracks which services are already started. The "run_rc_script" statement is what actually starts the service. All I'm trying to do is add some concurrency around that one line. 

Side note: I'm using vi(1) to make small edits. I used bectl(8) to create a boot environment as my safety net. So far I've not needed to restore my boot environment.

Here are my test results so far where 'level' is the number of tasks:

sequential /etc/rc: 8 seconds
concurrent /etc/rc (level 1): 11 seconds
concurrent /etc/rc (level 5): 11 seconds
concurrent /etc/rc (level 15): 10 seconds
concurrent /etc/rc (level 30): 10 seconds

I'm measuring the time to execute the aforementioned for-loop. Not total boot time - just the portion of time that I'm focused on. The numbers are so small because my test box has very little running on it outside of a stock 12.1-RELEASE. 

In short, I have a *3 second penalty* for adding this crufty script. There is one ray of sunshine - a one second drop when going from 5 to 15 tasks.

I'm going to create an empty service that does nothing but sleep for some # of seconds and add multiple instances of it in /etc/rc.conf. This will simulate running large services with long start-up latency. This should verify whether this will actually scale on a real host running real-world services.

I can address the 3 second penalty by moving the script logic into rcorder so I'm not too worried about that.

On this same box I've got separate boot environments for Mate, Xfce, Gnome3, and KDE. Will be fun to see how well they perform - but I'm getting ahead.


----------



## Mjölnir (Aug 8, 2020)

unitrunker said:


> Here are my test results so far where 'level' is the number of tasks:
> sequential /etc/rc: 8 seconds
> concurrent /etc/rc (level 1): 11 seconds
> concurrent /etc/rc (level 5): 11 seconds
> ...


That's likely because of the additional logic you introduced.  Check if rc(8) uses the static /rescue/sh, that's much faster for this use-case.


> On this same box I've got separate boot environments for Mate, Xfce, Gnome3, and KDE. Will be fun to see how well they perform - but I'm getting ahead.


Boot environments are for managing the base OS.  I would suggest to mount different /usr/local-_GUI_ ZFS datasets to /usr/local instead, and all use the same base (boot env).


----------



## unitrunker (Aug 8, 2020)

No idea which sh(1) is used by rc.

The GUI related BEs are for a different project. Outside the initial 12.1-RELEASE install, they do not share by design (desktop-installer is awesome).

Here's the RC script that I cloned and dropped into /usr/local/etc/rc.d/ ...


```
#!/bin/sh

. /etc/rc.subr

name=placebo
rcvar=placebo_enable
start_cmd="${name}_start"
stop_cmd="${name}_stop"

placebo_start()
{
  sleep 5
}

placebo_stop()
{
  sleep 5
}

load_rc_config $name
run_rc_command "$1"
```

The cloned copies are called placebo1, placebo2 and placebo3. Each was verified working by using 'service placebo[123] onestart'. I get a five second pause as expected.

I added lines to /etc/rc.conf to run them a start up:


```
placebo1_enable="YES"
placebo2_enable="YES"
placebo3_enable="YES"
```

On reboot - with a stock /etc/rc - I'd see an extra 15 seconds of boot time (23 seconds total). With the lightweight concurrency added - the startup time went up from 10 to 11 seconds.

So ... win!

It will be better to put the group logic into rcorder. Save that for another day ...


----------



## unitrunker (Aug 20, 2020)

Small wiki page documenting a new rcorder(8) with altered /etc/rc scripts.

Works for me but please consider this very experimental. Brave souls are welcome to test this out.





__





						unitrunker/rcorder - FreeBSD Wiki
					

rcorder



					wiki.freebsd.org


----------



## Mjölnir (Aug 20, 2020)

I'd like to suggest the following default behaviour

no `-j` parameter given: concurrency = 1
`-j` given without number sets a default concurrency level:
concurrency = `$(( `syctl -n hw.ncpu` + 1 ))`
`-j [number]`: concurrency = number
Final assertions:
max_concurrency = `$(( (2 * `syctl -n hw.ncpu`) + 1 ))`
concurrency = MIN( concurrency, max_concurrency )
Rationale: The trend for the #cores on multi-core CPUs is going up.  Consider e.g. the high-end SPARC-M series currently has 8 threads x 32 cores = 256 concurrent threads.
Iff the shell getopt(1) does not allow for an _optional_ option argument, add another parameter `-J`  that takes no argument, to set the concurrency to default like above (2nd topic).


----------



## unitrunker (Aug 20, 2020)

-j is the number of tasks, not number of threads or cores. These tasks are largely I/O bound. On an ancient Core 2 Duo I found 99 worked better than 30.

-j defaults to 1 (please see the wiki page above). Enabling concurrency is opt-in.

I had not considered more than 99 tasks so that is a very good point.


----------



## olli@ (Aug 20, 2020)

I’m afraid you can’t use make(1) for this purpose because it doesn’t live in the root file system, as rcorder(8) does. Therefore, such an extension that creates groups for parallel execution would have to be implemented in rcorder(8). But that shouldn’t be much of a problem, because it already contains most of the code necessary to implement it.

On the other hand, I’m not sure if all of that effort is really worth it. Many (most?) of the startup actions are I/O-bound. The thing that takes longest on my private workstation is the `netwait` script; it waits for the interface link to come up and for the default router to be pingable. This takes several seconds, and no amout of parallelism is going to make that any faster. All of the other scripts together only take a few seconds, too. So it’s really not a big deal. And all of the other stuff that happens during boot takes considerably longer: hardware initialization, EFI BIOS initialization, loading the kernel and the modules, initializing the kernel, hardware probing.

Running the rc scripts on large servers may take somewhat longer, especially when several jails have to be started. But again, starting a jail is I/O-bound. And it should also be noted that – unlike desktop machines – servers are not booted every day in the morning, so the it doesn’t matter that much. Just my opinion, of course.


----------



## unitrunker (Aug 20, 2020)

olli@ said:


> Running the rc scripts on large servers may take somewhat longer, especially when several jails have to be started. But again, starting a jail is I/O-bound. And it should also be noted that – unlike desktop machines – servers are not booted every day in the morning, so then it doesn’t matter that much. Just my opinion, of course.


I think this will help for servers with a large number of jails. The time savings is small for a minimal install - 2 seconds. For larger systems, it will be more.

The nice thing is the change is actually quite small for sh(1) changes - about two paragraphs of RC script edits.

I rewrote the C code for rcorder(8) because I wanted unit tests. The resulting binary is about 5kb larger.

Another nice thing is backwards compatibility. You can run new rc script with old rcorder or old rc script with new rcorder. Everything still works. No speed improvement but you can boot.

I'm curious how much faster to see the login screen for mate or slim.

About netwait ... other rc scripts that don't require it can continue to run.


----------



## Mjölnir (Aug 20, 2020)

unitrunker said:


> -j is the number of tasks, not number of threads or cores. These tasks are largely I/O bound. On an ancient Core 2 Duo I found 99 worked better than 30.


This surprises me.  This looks like an effect that on a spinning disk, it's more likely your data comes under the moving spindle the more data you demand (in the request-queue).  So then, the _#concurrency_ should not be derived from _#threads_, but from I/O capabilities of the system?  Did you test on spinning disks or SSD?  At least on UFS, inserting an I/O scheduler should give a reasonable speedup.  Still I find it more natural to derive _#concurrency_ from the threading capabilities of the system.


> -j defaults to 1 (please see the wiki page above). Enabling concurrency is opt-in.


I mean: without `-j` default = 1. With `-j` without argument: default as above, else take the argument the user/admin gives to `-j`, then apply the limitation to a reasonable maximum.


> I had not considered more than 99 tasks so that is a very good point.


Yes, very large systems might as well have enormous I/O beyond our wildest imagination.  They might be able to start more than 100 jails in parallel.

olli@ as I already noted in another thread (nosh(8)?), there have been several PoCs to parallelize init/rc with the help of make(1) (on some FreeBSD projects web page?).  Would be no problem to move make(1) to /bin & symlink in /usr/bin if it's needed on boot.


----------



## olli@ (Aug 20, 2020)

mjollnir said:


> olli@ as I already noted in another thread (nosh(8)?), there have been several PoCs to parallelize init/rc with the help of make(1) (on some FreeBSD projects web page?).  Would be no problem to move make(1) to /bin & symlink in /usr/bin if it's needed on boot.


I’m pretty sure that make(1) is not going to be moved to the root file system. The binary is almost 1 MB (on my stable/12 amd64 machine). The whole size of the /bin directory is just 2.2 MB, so moving make(1) there would increase it by almost 50 %. rcorder(8) is just 13 KB and would need only very little changes. This is also more efficient.


----------



## olli@ (Aug 20, 2020)

unitrunker said:


> I think this will help for servers with a large number of jails. The time savings is small for a minimal install - 2 seconds. For larger systems, it will be more.


No, there will be no extra time saving for jails, because the jails framework already has a `jail_parallel_start` setting, so they are already started in parallel. See the appropriate section in the rc.conf(5) manual page.


> About netwait ... other rc scripts that don't require it can continue to run.


Well, the stuff that comes afterwards requires a working network. That’s the purpose of `netwait`.


----------



## unitrunker (Aug 20, 2020)

olli@ said:


> No, there will be no extra time saving for jails, because the jails framework already has a `jail_parallel_start` setting, so they are already started in parallel. See the appropriate section in the rc.conf(5) manual page.



I see I've been missing out as I never knew about this.



> Well, the stuff that comes afterwards requires a working network. That’s the purpose of `netwait`.



I should clarify. Only RC scripts that require netwait (directly or indirectly) need to actually wait. Others can go concurrently. The netwait task can begin sooner while allowing some tasks to continue.

2 scripts require netwait. 8 scripts require NETWORKING. This won't stop the rest of the RC ecosystem.


----------



## olli@ (Aug 20, 2020)

unitrunker said:


> I should clarify. Only RC scripts that require netwait (directly or indirectly) need to actually wait. Others can go concurrently. The netwait task can begin sooner while allowing some tasks to continue.
> 
> 2 scripts require netwait. 8 scripts require NETWORKING. This won't stop the rest of the RC ecosystem.


Practically _everything_ that comes afterwards requires networking, directly or indirectly. When the `netwait` script is running, everything else has to wait. Keep in mind that things like NFS or NIS might be required to proceed, for example. Of course, if the machine is set up in way that no network is required for _anything_, then there wouldn’t be such a problem (and the user wouldn’t have `netwait` enabled anyway).


----------



## unitrunker (Aug 20, 2020)

olli@ said:


> Practically _everything_ that comes afterwards requires networking, directly or indirectly.


I get that. The point is netwait can start waiting much sooner which means the tasks that follow also run sooner.

Anyone can see the ordering:

$ rcorder /etc/rc.d/* /usr/local/etc/rc.d/*

Edit: On a simple install, there are 19 tasks ahead of FILESYSTEMS, 59 ahead of NETWORKING, 109 ahead of DAEMON, and 131 ahead of LOGIN.

When stuck in a traffic jam, it's all the cars in front of you that slow your progress.

olli@ and mjollnir - thank you both for your feedback. Your input has been very helpful.


----------



## mark_j (Aug 20, 2020)

mjollnir said:


> Well, as I wrote above: All this has been tried & evaluated multiple times by others.  Issues to solve:
> 
> To enshure a service is _up_, it's not sufficient that a service is _running_ (PID exists):  It might need some additional time to do some housework, or a master task/thread might fire up worker tasks/threads.  The shell's _wait_ only checks the PID of the service's rc script to die.  This does not mean in all cases that the service is _up_?  To solve that reliably, a standardized communication between a service & it's rc script has to be introduced.  The service needs to tell the service manager it's status (_"I'm up & ready to serve"_).  MAYBE this is solved by convention: the service rc script does not return before the service is ready (_up_).  I just don't know.
> I'm curious to read about your experience.


This is heading into the domain of launchd (and therefore systemd) where IPC is used to co-ordinate process startup.
That's new init system territory.


----------



## mark_j (Aug 20, 2020)

olli@ said:


> Practically _everything_ that comes afterwards requires networking, directly or indirectly. When the `netwait` script is running, everything else has to wait. Keep in mind that things like NFS or NIS might be required to proceed, for example. Of course, if the machine is set up in way that no network is required for _anything_, then there wouldn’t be such a problem (and the user wouldn’t have `netwait` enabled anyway).


Could there be a case to create a memory disk and gobble up as many scripts as possible during netwait? This might give you a second or two.
I think the slowness is down to interpreted sh at least in part; this is why launchd dispensed with it.


----------



## unitrunker (Aug 21, 2020)

I found these earlier attempts.









						GitHub - buganini/rcexecr: Parallel rc.d scripts executer for FreeBSD
					

Parallel rc.d scripts executer for FreeBSD. Contribute to buganini/rcexecr development by creating an account on GitHub.




					github.com
				








__





						⚙ D3715 Add "rcorder -p".
					






					reviews.freebsd.org


----------



## olli@ (Aug 21, 2020)

mark_j said:


> Could there be a case to create a memory disk and gobble up as many scripts as possible during netwait? This might give you a second or two.
> I think the slowness is down to interpreted sh at least in part; this is why launchd dispensed with it.


No, the overhead of parsing the shell scripts is negligible:

```
$ cd /etc/rc.d
$ /usr/bin/time sh -c 'echo * | xargs -n1 sh -nx'
        0.18 real         0.10 user         0.07 sys
```
And even _if_ it would save a second or two, that would be rather meaningless and not worth the efforts at all.

When I booted up my workstation today, I noted the time stamps as follows, relative to switching it on:

```
T+00:00   power button pressed
T+00:47   rc.d begins
T+00:53   rc.d finished
T+00:57   xdm login window appears
```
So, all together it took almost a minute from switching the machine on to being able to log in.
Only 6 seconds of that is caused by the rc(8) framework.

Spending hours (or even days) trying to fix a “problem” that doesn’t really exist appears to be a terrible waste of time that could be spent for useful things instead.


----------



## Mjölnir (Aug 21, 2020)

mark_j said:


> [...]  I think the slowness is down to interpreted sh at least in part; this is why launchd dispensed with it.


The statically linked /rescue/sh starts up faster than the dynamic default one.  When it's fired up often, these differences add up to a significant time saving.  IIRC this trick is/can be used in the build(7) system.  In make.conf(5): `MAKE_SHELL?=/rescue/sh`.  Would be a good idea to do that in rc(8), too.


----------



## olli@ (Aug 21, 2020)

mjollnir said:


> The statically linked /rescue/sh starts up faster than the dynamic default one.  When it's fired up often, these differences add up to a significant time saving.  IIRC this trick is/can be used in the build(7) system.  In make.conf(5): `MAKE_SHELL?=/rescue/sh`.  Would be a good idea to do that in rc(8), too.


No, it would not.
I invite you to do a little more research before making such suggestions, or even try it out in practice. It’s not difficult to replace /bin/sh with /rescue/sh if you’re curious to give it a try.

The difference in startup time between /bin/sh and /rescue/sh is rather small, and it only occurs when they are explicitly invoked for execution via the execve(2) system call, which happens when you enter the command at the shell prompt, for example. In this case, static binaries start up a little faster because the runtime linker has less work to do.

But the rc(8) framework does _not_ do that. It opens a subshell (using the fork(2) system call) and then sources the rc script within that subshell. The runtime linker is not involved at all, so it does _not_ make a difference whether the binary is static or dynamic.

By the way, the build system doesn’t use /rescue/sh, as far as I can see. The only purpose of /rescue is to be used for disaster recovery, when the root file system got damaged (so it saves you from having to use a bootable USB stick for recovery). Also, it was once meant to be used on very small, space-limited embedded systems, but that purpose isn’t meaningful anymore today. It should also be noted that /rescue is completely optional and not required for normal operation; it can be enabled or disabled via /etc/src.conf.


----------



## leebrown66 (Aug 21, 2020)

olli@ said:


> So, all together it took almost a minute from switching the machine on to being able to log in.



<rant on>
The last time I booted one of our Linux servers which is systemd, I got a login prompt almost immediately after the kernel was loaded, impressively fast to be honest.

However I then had to wait several minutes, before networking was available (lots of VLANS with CARP starting services like DHCP, etc), so I sat there and literally twiddled my thumbs.

Not being so familiar with the new linux stuff, I did a `tail -f /var/log/messages`, so at least I would be entertained by the startup information, but nothing appeared until the entire boot sequence was completed, then many thousands of lines were appended at once.

So what;s the point of being able to login, when the system is still booting, I just don't understand.
<rant off>

_I'm still waiting for the box to die so I can replace it with FreeBSD._


----------



## unitrunker (Aug 22, 2020)

Here's an update. Fixed a few bugs which invalidated my earlier test runs. I now have better numbers - both with and without concurrency enabled. Start up time is 8 seconds (in the area managed by /etc/rc). Increasing concurrency did not change that. 8 seconds is pretty quick so that's fine.

Concurrency is controlled by /etc/rc.conf(5) variable 'rc_task_limit'.

I started adding five second "placebo" services to /usr/local/etc/rc.d/ and enabling them in /etc/rc.conf(5).
With one service enabled - startup time was 13 seconds - with or without concurrency.

With two services enabled - startup time was 18 seconds for rc_task_limit=1 and 13 seconds for rc_task_limit=15.

With three services enabled - startup time was 23 seconds for rc_task_limit=1 and 13 seconds for rc_task_limit=15.

Setting rc_task_limit higher than 15 saw no improvement.

There's definitely some savings there but only if you have lots of extra services.





__





						unitrunker/rcorder - FreeBSD Wiki
					

rcorder



					wiki.freebsd.org


----------



## Mjölnir (Sep 15, 2020)

TWIMC: from today's FreeBSD's <svn-src-stable@freebsd.org> Digest: MFC r365449: Add a few features to rcorder(8):
o Add -p option that enables grouping items that can be processed in parallel. [...]
Thx!


----------



## Alain De Vos (Sep 15, 2020)

When is it foreseen that the man page will be updated ?


----------



## Mjölnir (Sep 15, 2020)

Alain De Vos said:


> When is it foreseen that the man page will be updated ?


It was updated today...  Since 12.2 was already branched off 12-STABLE, it will be in a -RELEASE in 12.3-REL I guess.  If you sync your 12-STABLE source tree _now_, build & install, you have it _today_...


----------



## unitrunker (Sep 15, 2020)

That does not include the changes to /etc/rc to take advantage of rcorder -p.


----------

