# Starting daemons requires CTRL-D



## raasmith (Nov 20, 2013)

A problem I don't understand has developed on my system (currently 9.2-RELEASE). When I invoke any rc script, whether from /etc/rc.d or from /usr/local/etc/rc.d, the return is invariably just a hash character. The reply from the script doesn't come until I enter CTRL-D (and evidently the script itself hangs until it gets that reply). Worse, this problem is now keeping me from rebooting my system (which is actually a VM on a machine I have only remote access to). When it's rebooted, none of the rc scripts executes; I have to get to the VM via VNC, where I find myself in single-user mode, and then remote the file systems and run the rc scripts manually (even /etc/rc.d/ldconfig, /etc/rc.d/netif and /etc/rc.d/routing), issuing a CTRL-D to the hash prompt that comes back from each one.  

I have no idea what causes this; it began happening after some upgrade of ports about three weeks ago (and still running 9.1-RELEASE), and when I rebooted after the initial install from `freebsd-update upgrade` I found the machine hung like this. Can someone tell me what's up?


----------



## SirDice (Nov 21, 2013)

What's in /etc/rc.conf? I have a feeling there's something there that shouldn't be.


----------



## raasmith (Nov 21, 2013)

Here's the contents of /etc/rc.conf (nothing problematic that I can see):


```
defaultrouter="65.19.130.1"
hostname="apacentral.org"
ifconfig_le0="inet 65.19.130.61 netmask 255.255.255.192"
firewall_enable="YES"
firewall_type="/etc/ipfw.conf"
firewall_logging="YES"
sshd_enable="YES"
apache22_enable="YES"
postgresql_enable="YES"
postfix_enable="YES"
sendmail_enable="NO"
sendmail_msp_queue_enable="NO"
sendmail_outbound_enable="NO"
sendmail_submit_enable="NO"
spamd_enable="YES"
linux_enable="YES"
rpcbind_enable="YES"
mailman_enable="YES"
dbus_enable="YES"
ntpd_enable="YES"
sshguard_enable="YES"
```
I'm now convinced the problem must instead be in rc somehow. The problem arises whenever any rc script is executed, even in single-user mode. Here's an example using the postfix startup script:


```
sudo /usr/local/etc/rc.d/postfix status
#
```
And that is it: no reply from the script. However, when I then enter CTRL-D I get the script's reply:


```
sudo /usr/local/etc/rc.d/postfix status
# ^Dpostfix is running as pid 1848.
```
It's not related to sudo or to the shell I'm using, since it also occurs even in single-user mode at startup with the scripts in /etc/rc.d/


----------



## wblock@ (Nov 21, 2013)

sudo was not mentioned before.  Try without it.

This could be due to a locale or TERM setting, or something specific to the root user.  Or possibly a change to termcap.


----------



## raasmith (Nov 21, 2013)

As I said, it occurs even in single-user mode without sudo. A sample from a root login:

```
root@apacentral:~ # /usr/local/etc/rc.d/postfix status
# ^Dpostfix is running as pid 1848.
```
(Same problem: return hangs until it gets `CTRL-D`.)

Use of sudo in any other context is unproblematic: the problem only occurs when an rc script is being run.  

Any suggestion about what to look for in termcap or what TERM setting could be involved?


----------



## jmccue (Nov 21, 2013)

Maybe try using service(8) like `# service postfix status`.

CTRL-D can mean 'End of Input". It is as if the postfix rc issues a grep (or some such command) without a file or pipe.

John


----------



## ondra_knezour (Nov 21, 2013)

Wild guess - newline missing in some file which is included in all/most of the rc scripts or is part of processing of those scripts?


----------



## kpa (Nov 21, 2013)

Do this and post the resulting log here or upload it to for example http://pastebin.com if it's rather large and post a link here.


```
script /tmp/postfix-status.log sh -x /usr/local/etc/rc.d/postfix status
```


----------



## raasmith (Nov 21, 2013)

jmccue said:
			
		

> Maybe try using service(8) like `# service postfix status`.
> 
> Actually, that just results in the same problem, though with a buck instead of a hash as the return character (you don't get the reply until you type `CTRL-D`)
> 
> ...



It's not just the postfix script: every rc script whatever does this. It has nothing to do with postfix. In fact, the rc scripts hang at bootup: after booting, the system sits in single-user mode, with none of the services running and without even having mounted the file system read/write.


----------



## raasmith (Nov 21, 2013)

kpa said:
			
		

> Do this and post the resulting log here or upload it to for example http://pastebin.com if it's rather large and post a link here.
> 
> 
> ```
> ...



I can do this, but I think it won't show anything helpful. The problem has nothing to do with postfix: I just used that as an example (and postfix loads and runs just fine; it's just that its rc script can't get through executing without the help of a `CTRL-D`.  The result is the same with any of the scripts in /etc/rc,d/ or in /usr/local/etc/rc.d/, and it happens using sudo or while logged in as root, and it happens in single-user mode from the console.  I don't believe it has anything to do with sudo, or anything to do with  sh, bash, or any other shell.  I'm repeating myself, but please note: this happens even at system startup, during the initial execution of the rc scripts in /etc/rc.d/ (so sudo has nothing to do with it).  The system hangs before it has even run such scripts as /etc/rc.d/netif, /etc/rc.d/routing, and /etc/rc.d/ldconfig, and when those are executed manually from the console each one generates the same hang until a `CTRL-D` is entered.  

So, any ideas what could be affecting every single rc script in this way?


----------



## kpa (Nov 21, 2013)

I'm asking for the log because it would show the whole sequence of what happens when an rc(8) script gets run on your system, I could have chosen a different script just as well.


----------



## raasmith (Nov 21, 2013)

*Solved*



			
				kpa said:
			
		

> I'm asking for the log because it would show the whole sequence of what happens when an rc(8) script gets run on your system, I could have chosen a different script just as well.



Sorry for my hasty reaction.  The whole thing is at http://pastebin.com/6Y7SehTy, but I think you can ignore it. The hang is at line 448 (see 445-448 below):


```
+ [ -r /etc/rc.conf ]
 + . /etc/rc.conf
 + sh
 # ^D+ defaultrouter=65.19.130.1
```
And that makes it obvious what the problem was: there's no shebang at the beginning of /etc/rc.conf, and the first line is just 
	
	



```
sh
```
not 
	
	



```
#!/bin/sh
```
 This could have happened during the merge of files during the upgrade from 9.1-RELEASE to 9.2-RELEASE, though I'd swear it started before that.  Sorry for causing everyone so much trouble, since I should have spotted this myself long ago. It's not even interesting.


----------



## kpa (Nov 22, 2013)

That's good that you solved it. However, rc.conf does not even need the shebang line. It's never executed as a shell script, only sourced with the . operator as you can see from the trace log.


----------



## wblock@ (Nov 22, 2013)

/etc/rc.conf should not have a #!sh line, although one with a shebang won't cause a problem because it's behind a comment.


----------



## raasmith (Nov 22, 2013)

kpa said:
			
		

> That's good that you solved it. However, rc.conf does not even need the shebang line. It's never executed as a shell script, only sourced with the . operator as you can see from the trace log.



Yes, of course you're right, and the bit of execution code I posted showed that.  What was happening is that the naked `sh` at the top was being executed, and of course it just waited forever for a file to source until it got a `CTRL-D`.


----------

