# RPC: Port mapper failure



## Grazzy (Oct 7, 2015)

We have a physical machine running FreeBSD 10.1. It is setup as an nfs server for multiple VM servers acting as nfs clients. The machine has been stable for a long time.

I was ssh logged into nfs machine. I did the following.
`freebsd-update fetch`
_Ctrl+C _(after a few sec because I realized I didn't want to sit through this process)
`shutdown -r now`

The machine did not reboot.
I went to the physical machine and did a hard reboot. It hangs in a loop.

```
(tcp) nfs.xxx.xxx:/zraid10/FreeBSD: RPCPROG_NFS: RPC: Port mapper failure - RPC: Timed out
```

Q. Did my bailing out of the FreeBSD update fetch cause this?

Q. What is my path forward to fixing this?

Thanks for any input.


----------



## junovitch@ (Oct 8, 2015)

Grazzy said:


> Q. Did my bailing out of the FreeBSD update fetch cause this?



A `freebsd-update fetch` never got the chance to modify any in use files. Only an interrupted `freebsd-update install` would cause a potential issue.



Grazzy said:


> Q. What is my path forward to fixing this?



Boot in single user mode and comment out all NFS mounts from /etc/fstab.  Do `return` to let it boot fully.  Then troubleshoot why it's having issues mounting.


----------



## Grazzy (Oct 8, 2015)

Thanks junovitch@ for the response!



junovitch@ said:


> A  freebsd-update fetch never got the chance to modify any in use files. Only an interrupted  freebsd-update install would cause a potential issue.


I guess I kind of knew that but it helps to get confirmation. Thanks.



junovitch@ said:


> Boot in single user mode and comment out all NFS mounts from /etc/fstab. Do return to let it boot fully. Then troubleshoot why it's having issues mounting.



I booted to Single-user Mode and attempted to modify fstab.
Unfortunately I am unable to modify the  file.
`vim /etc/fstab`

When I insert the # char I get:

```
E303: Unable to open swap file for "etc/fstab", recovery impossible
```
The line does get commented out though:

```
:wq!

  "/etc/fstab" E212: Can't open file for writing

:q!

  E138: Can't write viminfo file /root/.viminfo
```

Here is the contents of my fstab file.

```
Device    Mountpoint  FStype  Options  Dump  Pass#
/dev/gpt/root  /  usf  rw  1  1
/dev/gpt/swap  none  swap  sw  0  0
nfs.xxx.xxx:/zraid10/freebsd  /freebsd  nfs  ro,intr,nosuid  0  0
```

I appreciate any continuing suggestions.

Update: I was able to edit fstab by first entering `mount -uw /`. I'll see what happens from this point.


----------



## Grazzy (Oct 8, 2015)

OK my NFS machine is back up and running.
In Single-user Mode I commented out the NFS mount line and continued the boot.
It took me to the login prompt and everything appears to be back to normal.
I checked fstab and the line was restored at some point during the final boot process.
So now when I restart the machine I have to comment out the line first and then the boot proceeds normally.

I believe the root cause is an issue with the order in the boot init routine. That will have to be investigated. But for now at least I know how to work around it.

Thank you very much junovitch@. You rock!


----------



## junovitch@ (Oct 9, 2015)

Does adding the 'late' option to /etc/fstab for that mount help?  I don't see that as a recommendation in the FreeBSD Handbook: NFS section but it may be an option.  There may be some local network configuration to consider that is preventing this from working at boot.  What about name resolution not being available when /etc/rc.d/mountcritremote runs?


----------



## Grazzy (Oct 12, 2015)

junovitch@ said:


> Does adding the 'late' option to /etc/fstab for that mount help? I don't see that as a recommendation in the FreeBSD Handbook: NFS section but it may be an option. There may be some local network configuration to consider that is preventing this from working at boot. What about name resolution not being available when /etc/rc.d/mountcritremote runs?



Thanks for the additional suggestions. It may be a while before I can this investigate further. With the machine running and a work-around in place to allow a server restart, root cause gets bumped below other higher priorities.


----------



## junovitch@ (Oct 14, 2015)

Since you are using DNS names it seems like a logical possibility.  What DNS software are you using?  Is it listed before /etc/rc.d/mountcritremote in `rcorder /etc/rc.d/* /usr/local/etc/rc.d/*`.


----------



## Grazzy (Oct 14, 2015)

junovitch@ said:


> What DNS software are you using?


I believe that would be BIND on a FreeBSD name server VM.



junovitch@ said:


> Is it listed before /etc/rc.d/mountcritremote in `rcorder /etc/rc.d/* /usr/local/etc/rc.d/*`.


Here are the results of running
`root@nfs:/ # rcorder /etc/rc.d/* /usr/local/etc/rc.d/*`
mountcritlocal is listed prior to anything with "bind" in it. I see rpcbind and ybind.

```
rcorder: file `/usr/local/etc/rc.d/tcsd' is before unknown provision `kerberos'
rcorder: file `/usr/local/etc/rc.d/tcsd' is before unknown provision `named'
rcorder: requirement `tpmd' in file `/usr/local/etc/rc.d/tcsd' has no providers.
/etc/rc.d/sysctl
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/dumpon
/etc/rc.d/ddb
/etc/rc.d/initrandom
/etc/rc.d/geli
/etc/rc.d/gbde
/etc/rc.d/ccd
/etc/rc.d/swap
/etc/rc.d/fsck
/etc/rc.d/root
/etc/rc.d/mdconfig
/etc/rc.d/hostid_save
/etc/rc.d/mountcritlocal
/etc/rc.d/zfs
/etc/rc.d/var
/etc/rc.d/cleanvar
/etc/rc.d/FILESYSTEMS
/etc/rc.d/kldxref
/etc/rc.d/kld
/etc/rc.d/addswap
/etc/rc.d/random
/etc/rc.d/postrandom
/etc/rc.d/adjkerntz
/etc/rc.d/atm1
/etc/rc.d/hostname
/etc/rc.d/ip6addrctl
/etc/rc.d/netoptions
/etc/rc.d/sppp
/etc/rc.d/ipfilter
/etc/rc.d/ipnat
/etc/rc.d/ipfs
/etc/rc.d/serial
/etc/rc.d/netif
/etc/rc.d/devd
/etc/rc.d/ipsec
/etc/rc.d/atm2
/etc/rc.d/pfsync
/etc/rc.d/pflog
/etc/rc.d/pf
/etc/rc.d/stf
/etc/rc.d/ppp
/etc/rc.d/faith
/etc/rc.d/routing
/etc/rc.d/mroute6d
/etc/rc.d/nsswitch
/etc/rc.d/rtsold
/etc/rc.d/resolv
/etc/rc.d/local_unbound
/etc/rc.d/static_ndp
/etc/rc.d/static_arp
/etc/rc.d/bridge
/etc/rc.d/route6d
/etc/rc.d/mrouted
/etc/rc.d/routed
/etc/rc.d/defaultroute
/etc/rc.d/ipfw
/etc/rc.d/NETWORKING
/etc/rc.d/netwait
/etc/rc.d/mountcritremote
/etc/rc.d/devfs
/etc/rc.d/ipmon
/etc/rc.d/mdconfig2
/etc/rc.d/ldconfig
/usr/local/etc/rc.d/nslcd
/etc/rc.d/newsyslog
/etc/rc.d/syslogd
/etc/rc.d/kdc
/etc/rc.d/watchdogd
/etc/rc.d/savecore
/etc/rc.d/archdep
/etc/rc.d/abi
/etc/rc.d/SERVERS
/usr/local/etc/rc.d/tcsd
rcorder: requirement `cupsd' in file `/usr/local/etc/rc.d/samba' has no providers.
/etc/rc.d/accounting
/etc/rc.d/ntpdate
/etc/rc.d/rpcbind
/etc/rc.d/nfsclient
/etc/rc.d/nisdomain
/etc/rc.d/ypserv
/etc/rc.d/ypbind
/etc/rc.d/ypset
/etc/rc.d/amd
/etc/rc.d/atm3
/etc/rc.d/audit
/etc/rc.d/auditdistd
/etc/rc.d/tmp
/etc/rc.d/cleartmp
/etc/rc.d/ctld
/etc/rc.d/dmesg
/etc/rc.d/hastd
/etc/rc.d/ipxrouted
/etc/rc.d/iscsid
/etc/rc.d/iscsictl
/etc/rc.d/keyserv
/etc/rc.d/nfsuserd
/etc/rc.d/gssd
/etc/rc.d/quota
/etc/rc.d/mountd
/etc/rc.d/nfsd
/etc/rc.d/statd
/etc/rc.d/lockd
/etc/rc.d/pppoed
/etc/rc.d/pwcheck
/etc/rc.d/virecover
/etc/rc.d/DAEMON
/usr/local/etc/rc.d/samba
/etc/rc.d/apm
/etc/rc.d/apmd
/etc/rc.d/bootparams
/etc/rc.d/hcsecd
/etc/rc.d/bthidd
/etc/rc.d/local
/etc/rc.d/lpd
/etc/rc.d/motd
/etc/rc.d/mountlate
/etc/rc.d/nscd
/etc/rc.d/ntpd
/etc/rc.d/powerd
/etc/rc.d/rarpd
/etc/rc.d/rctl
/etc/rc.d/sdpd
/etc/rc.d/rfcomm_pppd_server
/etc/rc.d/rtadvd
/etc/rc.d/rwho
/etc/rc.d/timed
/etc/rc.d/ugidfw
/etc/rc.d/yppasswdd
/etc/rc.d/utx
/etc/rc.d/LOGIN
/usr/local/etc/rc.d/salt_syndic
/usr/local/etc/rc.d/salt_minion
/usr/local/etc/rc.d/salt_master
/usr/local/etc/rc.d/rsyncd
/usr/local/etc/rc.d/poudriered
/usr/local/etc/rc.d/mdnsresponderposix
/usr/local/etc/rc.d/mdnsd
/usr/local/etc/rc.d/lighttpd
/etc/rc.d/ypxfrd
/etc/rc.d/ypupdated
/etc/rc.d/wpa_supplicant
/etc/rc.d/ubthidhci
/etc/rc.d/syscons
/etc/rc.d/swaplate
/etc/rc.d/sshd
/etc/rc.d/sendmail
/etc/rc.d/cron
/etc/rc.d/jail
/etc/rc.d/localpkg
/etc/rc.d/securelevel
/etc/rc.d/power_profile
/etc/rc.d/othermta
/etc/rc.d/nfscbd
/etc/rc.d/natd
/etc/rc.d/msgs
/etc/rc.d/moused
/etc/rc.d/mixer
/etc/rc.d/kpasswdd
/etc/rc.d/kfd
/etc/rc.d/kadmind
/etc/rc.d/ipropd_slave
/etc/rc.d/ipropd_master
/etc/rc.d/inetd
/etc/rc.d/hostapd
/etc/rc.d/gptboot
/etc/rc.d/geli2
/etc/rc.d/ftpd
/etc/rc.d/ftp-proxy
/etc/rc.d/dhclient
/etc/rc.d/bsnmpd
/etc/rc.d/bluetooth
/etc/rc.d/bgfsck
/etc/rc.d/autounmountd
/etc/rc.d/automount
/etc/rc.d/automountd
```


----------



## junovitch@ (Oct 16, 2015)

Ok.  Thanks for the information.  What I was looking for is if you were running DNS software on the local machine that the DNS software is started before /etc/rc.d/mountcritremote executes.

In this case, that shouldn't be an issue.  However I think I'm a bit lost on one piece since the thread started mentioning both NFS clients and an NFS server.  Is this one of the clients?  If so, does it use DHCP?  An /etc/rc.conf option on an NFS client with DHCP may be to use:

```
ifconfig_<interface>="SYNCDHCP"
```

The logic is the synchronous DHCP option holds up the boot process until it gets an address and this will make absolutely sure name resolution is available when mounting remote shares.


----------



## Grazzy (Oct 16, 2015)

Both the name server VM (nfs client) and the nfs server physical machine use static IP address and /etc/rc.conf does not contain:

```
ifconfig_<interface>="SYNCDHCP"
```


----------



## junovitch@ (Oct 26, 2015)

Grazzy said:


> Both the name server VM (nfs client) and the nfs server physical machine use static IP address and /etc/rc.conf does not contain:
> 
> ```
> ifconfig_<interface>="SYNCDHCP"
> ```



The static IPs should ensure the name lookup for the NFS server's hostname works.  At the moment I am out of solid ideas on what this could be.


----------



## Grazzy (Oct 26, 2015)

junovitch@ said:


> At the moment I am out of solid ideas on what this could be.


That's fine. Now that I am able to boot the machine with the workaround, there are other fires to put out and this issue has dropped to a lower priority. Thanks again for the help!


----------

