# Serial console blocks/hangs at speeds > 9600 baud



## pvoigt (Mar 18, 2015)

I am on 10.1-RELEASE (amd64) and I have configured the serial console for emergency access. At 9600 baud it is working reliable, e.g. I can navigate through the boot loader, can see kernel and services messages and can finally login.

When I increase the serial speed above 9600 baud, everything seems to be OK. However, after some time, e.g. after some login processes, suddenly the console blocks or hangs. There is no output anymore and I cannot login anymore.

When investigating the process list, nothing unusual can be found, e.g. getty seems to be still waiting for connections. There are no error messages in /var/log/messages or `dmesg`. If I restart init with `kill -HUP 1` a new getty is started, but login is still not possible.

Moreover, if I restart the server, even shutdown hangs and I have to hard reset on most of these cases. Lowering speed to 9600 baud brings me back a reliable serial console again.

I am currently out of ideas, because I am running a second server with the same mainboard and the same serial null modem cable under FreeNAS 9.3. Under FreeNAS the serial console works at 115200 baud without any issues.

Maybe I should add that I am using minicom under Linux to connect.

I am currently out of ideas and I do appreciate any suggestions.

Regards,
Peter


----------



## junovitch@ (Mar 19, 2015)

Can you show your /boot/loader.conf console related items and the console related items in /etc/ttys?


----------



## wblock@ (Mar 19, 2015)

Try the simple stuff first: press ctrl-q.  Would a shutdown be paused if it could not send to the root console?  Maybe.


----------



## pvoigt (Mar 19, 2015)

junovitch said:


> Can you show your /boot/loader.conf console related items and the console related items in /etc/ttys?



I have:

```
# cat /boot/loader.conf
geom_eli_load="YES"
aesni_load="YES"
coretemp_load="YES"
console="comconsole,vidconsole"
boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
# Avoid openjdk core dumps - see PR187238.
vm.pmap.pcid_enabled="0"
# Enable NewCons vt instead of default syscons. See UPDATING 20141001
# entry.
kern.vty=vt
```
and:

```
# grep -E "ttyu0|115200" /etc/ttys
ttyu0  "/usr/libexec/getty std.115200"  vt100  on  secure
```
Besides that I have experimented with:

```
hint.uart.0.baud="115200"
```
in /boot/device.hints and with

```
-P -S115200
```
in /boot.config without success.

Regards,
Peter


----------



## pvoigt (Mar 19, 2015)

wblock@ said:


> Try the simple stuff first: press `ctrl-q`.  Would a shutdown be paused if it could not send to the root console?  Maybe.



Sorry, but what doess this magic `ctrl-q` do? Will it bring back the serial console or does it continue shutdown process?

One general question in this context: Should I be able to change serial speed for the getty alone, if I change /etc/ttys and issue a subsequent `kill -HUP 1`? My observation is, that getty indeed will be started with the new speed according to the process list, but I am not able to login with the new speed. Instead a minicom login is possible only with the old speed.

Regards,
Peter

EDIT: After some search I have stumbled accross this about the `ctrl-q` mystery - at least to me:
http://unix.stackexchange.com/quest...er-accidentally-pressing-ctrl-s-in-a-terminal


----------



## wblock@ (Mar 20, 2015)

Ctrl-s and ctrl-q are the software flow control characters.  When a terminal is receiving characters too rapidly to display, it sends a ctrl-s.  When it can accept more characters, it sends a ctrl-q.  These are also called XON/XOFF.  These characters can also be typed manually to halt or continue output.

It's possible to use hardware flow control instead, provided the cable is wired correctly and the software is configured that way at both ends.  There can be problems if the computer and terminal are not configured consistently.


----------



## pvoigt (Mar 20, 2015)

wblock@ said:


> Ctrl-s and ctrl-q are the software flow control characters.  When a terminal is receiving characters too rapidly to display, it sends a ctrl-s.  When it can accept more characters, it sends a ctrl-q.  These are also called XON/XOFF.  These characters can also be typed manually to halt or continue output.
> 
> It's possible to use hardware flow control instead, provided the cable is wired correctly and the software is configured that way at both ends.  There can be problems if the computer and terminal are not configured consistently.



Thanks for clarifying. I was not aware that you can manually control XON/XOFF. I would like to check this with higher baud rates, or more precisely: I would like to check, if my "hanging" console is stopped by a `ctrl-s` and can be "reactivated" with `ctrl-q`. However, I would like to avoid rebooting my server. Therefore I have asked in my last post, if the getty baud rate in /etc/ttys can be changed without a reboot. Could you please give me some advice on this?

Regards,
Peter


----------



## wblock@ (Mar 21, 2015)

Restarting init(8) with `kill -HUP 1` is probably enough.  But why not just type ctrl-q and see if the output resumes?


----------



## pvoigt (Mar 21, 2015)

wblock@ said:


> Restarting init(8) with `kill -HUP 1` is probably enough.  But why not just type ctrl-q and see if the output resumes?



Because I have currently switched back to reliable 9600 baud. And according to my above described observations, changing /etc/ttys and `kill -HUP 1` is not enough to change getty speed. My hope is, I have simply forgotten a step but I am afraid I will have to reboot.

Regards,
Peter


----------



## pvoigt (Sep 10, 2015)

Well, time passed by and I would like to reactivate this thread.

I am on 10.2-RELEASE now and I have experimented a bit more with the serial console. For any speed above 9600 baud I am observing this unpredictable hang. Unfortunately, I cannot reactivate the serial console with `ctrl-s` or `ctrl-q`. Even `kill -HUP 1` or killing the tty process does not bring back the serial console once it hangs. I am forced to reboot to get my serial console back.

I would appreciate any instructions on how to configure hardware flow control instead. I am using minicom on the client machine. The very same serial cable is working with FreeNAS at 115200 baud on another machine with the same motherboard.

Peter


----------



## wblock@ (Sep 10, 2015)

It acts like you are already using hardware flow control, but without a correctly wired cable.  I don't use serial consoles, and do not know how it is set up by default, but am investigating.  How many wires are in the cable, and how are they connected?


----------



## pvoigt (Sep 13, 2015)

Thank you for your reply. I have just found some time to messure the pin connections of my DB9-DB9 null modem cable. Unfortunately, the battery of my digital meter is empty. As soon as I have replaced it I will report to the forum.

Meanwhile I have experimented with different speeds. With 115200 baud I am still having the described hang mostlly within one minute. At 38400 baud the serial connection is working stable. I have done my tests both with a serial port on client side and a USB port on client side with same results.

Regards,
Peter


----------



## pvoigt (Sep 16, 2015)

Well, finally I have replaced my battery and have determined the DB9/female-DB9/female pin connections of my cable:

1 - 8,7
2 - 3
3 - 2
4 - 6
5 - 5
6 - 4
7 - 1
8 - 1
9 - unused

Just for clarification: The pin numbers are determined when looking into the female DB9 connector:

5 4 3 2 1
 9 8 7 6

I am not an expert on the correct pin connections of a null modem cable. I have been searching the web for the correct connections of a null modem cable with contradicting information. I therefore cannot judge, if my cable has proper null modem layout. My FreeNAS machines, my Soekris and my Alix2D13 are all working properly with 115200 baud with this cable. So far I am having just problems with FreeBSD and highest baud rates. After several days of testing 38400 baud turn out to be as stable as 9600 baud. 115200 baud definitely produces hangs and 57600 is so far untested with FreeBSD.

I would really like to know, if my cable layout is suited for hardware flow control. And if so: How can I use it with e.g. Minicom on a Linux box.

Thanks in advance for feedback
Peter


----------



## wblock@ (Sep 16, 2015)

Based on that, I'd say no, it is not meant for hardware flow control.  CTS and RTS (pins 7 and 8) are tied to CD (pin 1).  It ought to work with software flow control, though, which would explain the situations where it does work.


----------



## pvoigt (Sep 16, 2015)

wblock@ said:


> Based on that, I'd say no, it is not meant for hardware flow control.  CTS and RTS (pins 7 and 8) are tied to CD (pin 1).  It ought to work with software flow control, though, which would explain the situations where it does work.



How should pins 7 and 8 be tied in order to provide a hardware flow compatible null modem? I have searched for it and found somewhat contradicting answers. Is this the correct layout: http://lavalink.com/wp-content/uploads/2012/05/full_handshaking.jpg

Are there other settings required besides a correctly wired cable, e.g. on the FreeBSD machine or the client side, in order to profit from hardware flow control?

Do you have an idea why pure FreeBSD is obviously more sensitive to not ideally wired null modem cables like FreeNAS or pfSense? My FreeNAS 9.2 machines are even running the same motherboard and do not make any problem at 115200 baud.

Peter


----------



## wblock@ (Sep 16, 2015)

That looks better.  In that diagram, they show CTS connected to RTS, and vice versa.

The other systems have to be configured for software flow control.  The question is where that is done.  Many they have an added hint to set software flow control.  You did now show your whole /etc/ttys earlier, particularly the console entry.


----------



## pvoigt (Sep 17, 2015)

I am not aware that there are other settings besides the above in /etc/ttys that control serial console behavior and in particular usage of software flow control. Nevertheless, here is my current complete file content:

```
console none  unknown off secure
#
ttyv0  "/usr/libexec/getty Pc"  xterm  on  secure
# Virtual terminals
ttyv1  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv2  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv3  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv4  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv5  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv6  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv7  "/usr/libexec/getty Pc"  xterm  on  secure
ttyv8  "/usr/local/bin/xdm -nodaemon"  xterm  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0  "/usr/libexec/getty std.38400"  vt100  on  secure
ttyu1  "/usr/libexec/getty std.9600"  dialup  off secure
ttyu2  "/usr/libexec/getty std.9600"  dialup  off secure
ttyu3  "/usr/libexec/getty std.9600"  dialup  off secure
# Dumb console
dcons  "/usr/libexec/getty std.9600"  vt100  off secure
```

Peter


----------



## pvoigt (Sep 19, 2015)

While still reading about this topic I stumbled over this nice article:
http://www.lammertbies.nl/comm/info/RS-232_null_modem.html

According to this reference my null modem cable corresponds to what is called "null modem with partial handshaking".

I would really like to know how systems can be configured to use software flow control.

wblock@, is there anything unusual with my /etc/ttys?

Peter


----------



## wblock@ (Sep 19, 2015)

A software flow control cable really only needs ground, TX, and RX.  Technically, signal ground also.  For null modem, TX and RX are crossed over.  That is the bare minimum necessary.  This does not work on all systems because some require that certain signals on the connector are valid, like carrier detect.  The extra connections on your cable might actually be preventing it from working.

Remember also that the systems on both ends of the cable have to be set for the same type of handshake.

I don't see anything weird in your /etc/ttys except the std.38400 entry, but that should not be a problem.

There are serial break-out boxes that give control over which pins are connected and where.  Before molded cables, it was possible to remove connector pins non-destructively.


----------



## pvoigt (Sep 19, 2015)

wblock@ said:


> A software flow control cable really only needs ground, TX, and RX.  Technically, signal ground also.  For null modem, TX and RX are crossed over.  That is the bare minimum necessary.  This does not work on all systems because some require that certain signals on the connector are valid, like carrier detect.  The extra connections on your cable might actually be preventing it from working.


Ah, that might be the reason.



wblock@ said:


> Remember also that the systems on both ends of the cable have to be set for the same type of handshake.


I usually just start minicom on a Linux client where any kind of flow control is disabled. I am working this way for years without any issue - except the "native" FreeBSD machine. My Soekris, Alix and FreeNAS machines (with identical motherboard) working with this setting at 115200 baud and the same null modem cable.

Additionally, I have checked the serial port settings with `stty` on FreeBSD

```
# stty -a -f /dev/ttyu0 |grep -E "xon|off|crtscts"
iflags: -istrip -icrnl -inlcr -igncr -ixon -ixoff -ixany -imaxbel -ignbrk
cflags: cread cs8 -parenb -parodd -hupcl clocal -cstopb -crtscts -dsrflow
```
and on Linux side:

```
# stty -a -F /dev/ttyS0 |grep -E "xon|off|crtscts"
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff
```



wblock@ said:


> I don't see anything weird in your /etc/ttys except the std.38400 entry, but that should not be a problem.


38400 baud is still working reliably on my FreeBSD server.

There is obviously no simple answer why 115200 baud are not working reliably on my "native" FreeBSD machine compared to the others. But tests during this discussion revealed that increasing baud rate up to 38400 is working reliably on the FreeBSD machine.

Thank you  very much for your feedback so far.

Peter


----------



## wblock@ (Sep 20, 2015)

It would still be worth comparing with FreeNAS.  Finding out what they have changed from the default FreeBSD settings should allow the use of that same cable.


----------



## pvoigt (Sep 21, 2015)

wblock@ said:


> It would still be worth comparing with FreeNAS.  Finding out what they have changed from the default FreeBSD settings should allow the use of that same cable.


You are right and that was what I tried before posting here. But to be honest I have not be successfull in analyzing the differences and in drawing conclusions for 115200 baud on my "native" FreeBSD machine.

I am running FreeNAS 9.3 stable (FreeNAS-9.3-STABLE-201503120630) based on FreeBSD-9.3-RELEASE-p10. It is some month old and I should upgrade soon to the 9.3.1 stable branch. FreeNAS has a ZFS root file system and is booted by Grub2. System is not directly comparable with my FreeBSD 10.2-RELEASE-p3. Nevertheless, serial console at 115200 baud is working correctly. I see no differences in
/etc/ttys, /boot.config, /boot/device.hints and stty output. /boot/loader.conf does not contain serial console configuration parameters because FreeNAS is booted by grub2.

My pfSense machine is based on FreeBSD 10.1-RELEASE-p15 and is working fine with 115200 baud as well. I can observe just differences in /etc/ttys:

```
ttyv0  "/usr/libexec/getty Pc" cons25  on  secure
ttyu0  "/usr/libexec/getty std.115200" cons25  onifconsole  secure
```
Furthermore, /boot/device.hints does not contain a baud rate. That is all and does not help me. May be anybody else has some ideas.

Peter


----------



## ColdfireMC (Sep 22, 2015)

From what are you hearing the serial console? This sounds like a DCE/DTE confusion


----------



## pvoigt (Sep 22, 2015)

ColdfireMC said:


> From what are you hearing the serial console? This sounds like a DCE/DTE confusion


I am not sure what you mean with "hearing". If you ask for the client: I am using minicom from a KDE konsole under Linux.
And could you please be a bit more specific with DCE (client machine with minicom)/DTE (getty on FreeBSD side) confusion?

Peter


----------



## Juha Nurmela (Sep 23, 2015)

If/when it hangs, check if the hardware still handles the characters.

`iostat; echo foo > /dev/ttyu0; iostat`. `vmstat -i`should show an increase in interrupts as well. `ps lax` might show what the hung processes are waiting for.

I see a *3wire.115200* entry in /etc/gettytab, commented "Entries for 3-wire serial terminals".

Juha


----------



## pvoigt (Sep 27, 2015)

Juha Nurmela said:


> If/when it hangs, check if the hardware still handles the characters.
> 
> `iostat; echo foo > /dev/ttyu0; iostat`. `vmstat -i`should show an increase in interrupts as well. `ps lax` might show what the hung processes are waiting for.
> 
> ...



Well, thanks for your new ideas.

When the serial console hangs with std.115200  it does not handle any characters anymore. And if I interpret the output of `iostat` correctly there is no increase of interrupts.

When using 3wire.115200 I do not get a login prompt at all. More precise even console output stopped at some point which is probably when init should start getty processes. This first looked very promising to me because the comment of the 3wire terminals seemed to match my serial cable. And a check with `stty` revealed the required serial parameters:

```
# stty -a -f /dev/ttyu0 |grep -E "xon|off|crtscts|clocal"
iflags: -istrip icrnl inlcr -igncr ixon -ixoff ixany imaxbel -ignbrk
cflags: cread cs8 -parenb -parodd -hupcl clocal -cstopb -crtscts -dsrflow
```
Peter


----------



## Juha Nurmela (Sep 27, 2015)

Some more "useful" commands


```
stty -f device 0
procstat -af | grep /dev
```

First one used to reset a terminal, very driver-dependent thing.

Reaching at straws, terminal *lflags* also contains some potential culprits, pendin and flush0. *cflags* wraps to another line, containing mdmbuf and dtrflow. Use `stty -a` without the grep, to rule those gremlins out.

Juha

oo la laa ~ man termios


----------



## pvoigt (Sep 28, 2015)

I rarely use `stty` and moreover I am not aware of terminal lflags possibly being culprits. I have never used `stty` to make any changes to a serial port so far.

To make things clearer I give here full `stty` output:

```
# stty -a -f /dev/ttyu0
speed 38400 baud; 0 rows; 0 columns;
lflags: -icanon -isig -iexten -echo -echoe -echok -echoke -echonl
  -echoctl -echoprt -altwerase -noflsh -tostop -flusho -pendin
  -nokerninfo -extproc
iflags: -istrip -icrnl -inlcr -igncr -ixon -ixoff -ixany -imaxbel -ignbrk
  -brkint -inpck -ignpar -parmrk
oflags: -opost onlcr -ocrnl tab3 -onocr -onlret
cflags: cread cs8 -parenb -parodd -hupcl clocal -cstopb -crtscts -dsrflow
  -dtrflow -mdmbuf
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>;
  eol2 = <undef>; erase = ^?; erase2 = ^H; intr = ^C; kill = ^U;
  lnext = ^V; min = 1; quit = ^\; reprint = ^R; start = ^Q;
  status = ^T; stop = ^S; susp = ^Z; time = 0; werase = ^W;
```
I have seen on my soekris that ACPI has been disabled in /boot/loader.conf.local. On my FreeNAS machine, however, where 115200 baud are working correctly as well I did not explicitly disable ACPI.

And thanks for the `# stty -f /dev/ttyu0 0` hint. During my next test with 115200 baud I will try to reset terminal.

Peter


----------

