# How to make top/systat constantly running on a spare virtual console?



## Bobi B. (Aug 9, 2018)

Greetings, All. Is it possible to have top(1) or systat(1) constantly running on a spare virtual console, like ttyv8? In the best case they will also accept keyboard input when virtual console is active, so `top`/`systat` toggles could be changed.

Here is my case. A testing server misbehaved half an hour ago, stopped doing whatever it is usually doing (there is a queue of tasks that left waiting), virtual console login stopped working -- one can press Enter on `login:` prompt and console will print prompt again, but entering `root` will make console hang. sshd(8) stopped accepting remote connections. Since machine is forwarding log events to a remote server I can see it generates couple of `kernel: pid XXX (getty), uid 0: exited on signal 11` messages every 10 seconds; basically an hour ago processes started persistently dumping core (including cron), but nothing else interesting in syslog. I would like to have a diag tool left running, so I'll be able to analyse it if it happens again.

Can you suggest any other tool that might come in handy, or another way to approach such a problem? Thank you!


----------



## ShelLuser (Aug 9, 2018)

That's when I'd be booting into single user mode to start diagnosing stuff.


----------



## nactusberrilli (Aug 9, 2018)

Bobi B.

This sounds to me like it could be a SIGSEGV signal(3). I would make sure that all running processes have (correct) read/write memory access. Also, if your server starts failing after a while (to me) that also indicates a memory problem. What are you running on that server? You've probably seen this already.


----------



## Bobi B. (Aug 10, 2018)

This is a server-grade Supermicro machine running in a conditioned room, couple of years old, being fine so far. It is running fine right now, as well (restarted it yesterday). Machine is working as a testing IPTV server, doing some live & offline transcoding, some recording & streaming, basically a mix between testing and staging machine. Mostly constant I/O load, low & occasional higher CPU load.

Having said this, I suspect it is more of a software problem, hence I'm looking into ways to locate where the problem is, if it happens to manifest itself again. I would hate this to happen on a production machine. And if it is a my mistake I prefer to find this now, so I can fix it.


----------



## Nicola Mingotti (Aug 10, 2018)

This is maybe a silly suggestion, but it happened to me more than a few times... have you checked the disk is not full ?  ... i will look into you original question shortly


----------



## Bobi B. (Aug 10, 2018)

Nicola Mingotti said:


> This is maybe a silly suggestion, but it happened to me more than a few times... have you checked the disk is not full ?  ... i will look into you original question shortly


Nope; rudimentary monitoring were still working, showing neither of partitions/filesystems were full. However this would be visible in the syslog.

There were also no excess CPU or memory utilisation.


----------



## Nicola Mingotti (Aug 10, 2018)

Bobi B. to your original question here are my considerations

-] if you don't require interaction with "top" (or systat) you could
simply forward their output on the local network. like in the code below.

-] if you require interaction with "top" etc. i guess the best is option
is to keep a "ssh" connection to the controlled machines always open.
It must be possible to tap into the "/dev/pts/X" connected
to ssh. I remember i saw something similar in the Expect book, but i can't find the book !
... I tried to "tap into" with "cu" directly, but ok, too much rudimentary ... if i find a clean solution to this I will tell you, it is an interesting thing.


```
------------ client ---------------
#!/usr/bin/env ruby

require 'socket'
host = 'localhost'
port = '12345'

TCPSocket.open(host, port) do |s|
  while line = s.gets
    puts line.chop
  end
end
-----------------------------------------
```


```
----------- server ----------------
#!/usr/bin/env ruby

require 'socket'

server = TCPServer.open(12345)
loop {
  client = server.accept
  out = `top -n`
  client.puts(out)
  client.close
}
---------------------------------------
```


----------



## VladiBG (Aug 10, 2018)

did you check the memory of the server? It's smell like a bad memory or CPU problem (overheating/unstable voltage PSU problems)

If the getty is terminated then also the program that you run at the tty will be terminated but you can try this:

gettytab(5)

Edit your/etc/gettytab and add on the bottom the following line


```
MyApp|console|My application:\
    :ht:np:sp#9600:lo=/usr/bin/systat
```

edit your /etc/ttys and change some of the ttyv to the following for example:


```
ttyv1 "/usr/libexec/getty console" cons25 on secure
```

When you switch to console 1 (ctrl+alt+f2) and type any name without password it will start the systat. If the program (systat) exit it will show you login prompt again.

The credits goes to Matthew
https://lists.freebsd.org/pipermail/freebsd-questions/2004-March/042040.html

ps.
maybe it will be easy just to loging on several consoles and leave the programs running on them.


----------



## Bobi B. (Aug 11, 2018)

VladiBG said:


> did you check the memory of the server? It's smell like a bad memory or CPU problem (overheating/unstable voltage PSU problems)


Not checked yet, however this is not a new machine and everything is running fine after restart, following that weird freeze.



VladiBG said:


> If the getty is terminated then also the program that you run at the tty will be terminated but you can try this:
> 
> gettytab(5)
> 
> ...


I'll definitely dig into this, thanks!



VladiBG said:


> ps.
> maybe it will be easy just to loging on several consoles and leave the programs running on them.


I thought of it, as well. Will probably just log-in and leave few utilities running all the time. But were looking for a way to automate this, if possible.


----------



## Bobi B. (Aug 11, 2018)

Nicola Mingotti said:


> Bobi B. to your original question here are my considerations
> 
> -] if you don't require interaction with "top" (or systat) you could
> simply forward their output on the local network. like in the code below.
> ...


We've a small, "proprietary" monitoring agent running on those servers providing some metrics via HTTP. I could see the machine "looks" fine, but not much under the hood.

`top >/dev/ttyv8 </dev/ttyv8` works; not sure this is the right way, tho; `top` is still owned by the TTY it have been started under (i.e. ssh session). Is there a utility to start a program on another TTY?


----------



## Nicola Mingotti (Aug 11, 2018)

Bobi B. said:


> `top >/dev/ttyv8 </dev/ttyv8` works; not sure this is the right way, tho; `top` is still owned by the TTY it have been started under (i.e. ssh session). Is there a utility to start a program on another TTY?



Reading man pages i found only that a new `pts` should be created with `mknod` . Then probabily we can use the strategy you say: `top >/dev/pts/X </dev/pts/X` ... But i still i did not check well; i have found another way.

The solution i propose you is using the tool `screen`, wich was introduced to me by aragats a few months ago. Actually i never used it because its shortcuts interfere with my "emacs" / "mg" habits, but in this case it seems a good solution.

So, I write you a procedure wich can of course be automatized a lot, but i guess you can do it as well as i can So here is the bare-bone

---------------------------------------------------------------------------------------------
1] let's call C your desktop and S the server you want to monitor

2] You will be typing in C; S is remote and considered not nearby, but it has
sshd active and you can log into it. Indeed, it is supposed you start this
monitoring procedure before the server S goes critical and forbids you to login.

3] Open an xterm and type `screen`
Now you will see a shell, but this sheel is under the control of "screen".
`foo@client> ssh bar@server`
`bar@server> top`

4] Now press <C-a C-c> this will open a new window into "screen", with a new shell
`foot@client> ssh bar@server`
`bar@server> systat`

5] Now press <C-a d> this will detach "screen" from your current xterm.
You can also kill the "xterm" after it. This is important because if we
would have liked to keep many "xterm" always open for monitoring there
would have been no need to implement this solution.

6] Open a new xterm

7] "screen" should be still alive, look for its PID (i write precedure for completeness
i know this is knid of too basics to say )
`foo@client> ps aux | grep screen` 
In my case the pid is 23529

8] Connect "screen" program to the old instance of "screen" you run before
`foo@client> screen -r 23529`

9] You will see now "top" or "stat" running in S

10] You can change the view from "top" to "stat" pressing <C-a ">  and selecting
the appropriate window in the list it appears.

11] To quit the "screen" press <C-a d>

12] You can kill "screen" in the usual way `kill 23529`
--------------------------------------------------------------------------------------

In this way you are always monitoring "S" and you are always logged in there.

... of course ssh may fall; but this is another story


----------



## ShelLuser (Aug 11, 2018)

Nicola Mingotti said:


> Reading man pages i found only that a new `pts` should be created with `mknod`.


That's not entirely correct. While creating a new /dev entry with mknod will work, your changes won't retain after a reboot:

```
peter@zefiris:/home/peter $ mount | grep dev
devfs on /dev (devfs, local, multilabel)
```
/dev is usually mounted as a virtual file system which means that any changes you make won't retain and will get reset right after a reboot. That's because specific devfs rules are followed when populating the filesystem, see /etc/defaults/devfs.rules.

Another issue is that the system is managed through devd(8) which can dynamically create (or remove) devices.

So although mknod will work for that one session, you really want to check either devfs(5) or devd.conf(5) if you want to add device entries.


----------



## Nicola Mingotti (Feb 28, 2019)

Today I stumbled on `man 8 watch` and I remembered this thread.
I think this tool solves nicely Bobi B. original question.

Procedure:

1] I am typing in a xterm (konsole)
`$> tty`
/dev/pts/2

2] I want to run `top` in a virtual terminal, let's say ttyv3 so
`$> sudo watch -W /dev/ttyv3`
[do the login]
`$> tty`
/dev/ttyv3
`$> top`

3] I quit the watch command pressing *Ctrl+G*

4] I check where i am 
`$> tty`
/dev/pts/2

5] Now i go to see the virtual terminal pressing *Ctrl+Alt+F4*
=> I see top running 

6] I go back to XWindows *Ctrl+Alt+F9*
[attention, here i must wait about 15 seconds till the XWindows becomes fully operative]

bye
n.


----------



## Eric A. Borisch (Mar 1, 2019)

Automically run dialog based shell script on ttyv0 like a kiosk mode
					

I am currently automatically execute a script on boot that will run some tests on an embedded device.  My actual goal is to have this menu running 100% of the time without any sort of keyboard input needed, with other ttys disabled.  Not so much for security as to deter the casually curious...




					forums.freebsd.org


----------



## Phishfry (Mar 27, 2019)

Thank You Bobi B. for posting this question.
Similarly in my mind I was thinking about my 8 unused virtual terminals
Opening a separate program in each.
I really liked the idea of ytree on one virtual terminal and pftop in another virtual terminal.
With each console requiring a login this does become way more complicated.
To a point that I think Tmux instances with each command would be better.
Tmux instances for bhyve has worked out superbly for me. Clt-B and W is nice.


----------

