# Strange lines in /var/log/messages



## Buck (May 15, 2019)

Eight character long random strings that go like


```
May 12 21:42:56 mybox kernel: aYRnH85U
May 12 21:42:56 mybox kernel: 947.91
May 12 21:42:56 mybox kernel: aYRnH85U
May 12 21:42:56 mybox kernel: 947.91
May 12 21:42:56 mybox kernel: aYRnH85U
May 12 21:42:56 mybox kernel: 947.91
...
May 14 09:35:33 mybox kernel: UqqsLzYf
May 14 09:35:33 mybox kernel: icdkk8kC
May 14 09:50:42 mybox kernel: tkSAfwSO
May 14 09:50:42 mybox last message repeated 42 times
May 14 09:50:42 mybox kernel: AGbtkSAf
May 14 09:50:42 mybox kernel: mEI8hmeU
May 14 09:50:42 mybox kernel: AGbtkSAf
May 14 09:50:42 mybox kernel: mEI8hmeU
May 14 09:57:54 mybox kernel: Z1STeY4i
May 14 09:57:54 mybox last message repeated 42 times
May 14 09:57:54 mybox kernel: WbaZ1STe
May 14 09:57:54 mybox kernel: PYJaxfrv
May 14 09:57:54 mybox kernel: WbaZ1STe
May 14 09:57:54 mybox kernel: PYJaxfrv
May 14 10:42:55 mybox kernel: oYTNhyZJ
May 14 10:42:56 mybox kernel: JwsaNul8
May 14 10:42:56 mybox last message repeated 90 times
May 14 10:42:56 mybox kernel: hyZJwsaN
May 14 10:42:56 mybox kernel: hyZJwsaN
May 14 10:42:56 mybox kernel: AEnFi9G6
```

What does that mean?


----------



## leebrown66 (May 15, 2019)

What version of FreeBSD?


----------



## SirDice (May 15, 2019)

Buck said:


> Eight character long random strings that go like


They're not entirely random if you look closely. A few of them are repeated a number of times too. Is there a kernel module you loaded that might have a version 947.91 in it? 


Buck said:


> What does that mean?


They're definitely weird, and I have no clue where they may have come from. I've never seen anything like this before.

Lets rule out a few things, I assume syslogd(8) is running? Does it perhaps have its network port open? It's fairly easy to spoof syslog messages over a network.


----------



## Buck (May 15, 2019)

Huh. I thought it was me missing something obvious. Well, that makes things interesting.
All right, so the version is 11.2-RELEASE-p9 (it actually shows -p10 now because I've just run freebsd-update but haven't rebooted yet).
syslogd is indeed running, but its ports aren't open, I've checked it using nmap (-sU along with the regular scan) on a different machine. Just the regular service ports like imaps and https.

kldstat output:

```
Id Refs Address            Size     Name
 1   15 0xffffffff80200000 11779d0  kernel
 2    1 0xffffffff81379000 381080   zfs.ko
 3    2 0xffffffff816fb000 24fd8    krpc.ko
 4    2 0xffffffff81720000 a380     opensolaris.ko
 5    1 0xffffffff8172b000 3a20     amdtemp.ko
 6    2 0xffffffff8172f000 1eb8     amdsmn.ko
```
`kldstat -v` gives a longer output but I haven't installed anything non-standard anyway.

Looking further at this log it started happening pretty early, about two days after the OS got installed. There was a regular maintenance reboot after which those lines started to appear.

Where should I look now? syslogd configuration maybe?


----------



## k.jacker (May 16, 2019)

The patterns look like sysutils/pwgen. `pwgen -1|logger`
Can't be from a userland program, since the kernel throws them, but maybe something similar that's out of control somehow.

Are you running a generic kernel?
krpc is build into the kernel on FreeBSD 11.2 and 12.0-RELEASE and you have it loaded as a module.


----------



## mfaridi (May 16, 2019)

This happens cause of upgrade and reboot.


----------



## SirDice (May 16, 2019)

Buck said:


> Where should I look now? syslogd configuration maybe?


Look at the output of `ps -ax | grep syslog`. You should see something like this:

```
dice@maelcum:~ % ps -ax | grep syslog
  314  -  Is       0:00.95 dhclient: system.syslog (dhclient)
 1148  -  Ss       4:12.53 /usr/sbin/syslogd -l /var/run/log -l /var/named/var/run/log -cc -s
 1383  -  I        0:21.87 /usr/local/libexec/ipsec/charon --use-syslog
98599  0  R+       0:00.00 grep syslog
```
The others are not important, it's just some stuff I have running. Look at the syslogd(8) process, in particular if it has the `-s` option:

```
-s      Operate in secure mode.  Do not log messages from remote
             machines.  If specified twice, no network socket will be opened
             at all, which also disables logging to remote machines.
```

If syslogd(8) doesn't have at least one `-s` it will listen on a network port. The `-s` is a default option, but this could have been changed at some point. Someone or something on your network would then be able to send syslog messages to the machine. Those are easily spoofed and could result in weird or bogus messages.


----------



## Buck (May 16, 2019)

`/usr/sbin/syslogd -ss -C` is the output of ps.
I don't have pwgen installed.

The kernel is custom and the config is:

```
cpu        HAMMER
ident        MYBOX

makeoptions    DEBUG=-g        # Build kernel with gdb(1) debug symbols
makeoptions    WITH_CTF=1        # Run ctfconvert(1) for DTrace support

options     SCHED_ULE        # ULE scheduler
options     PREEMPTION        # Enable kernel thread preemption
options     INET            # InterNETworking
options     INET6            # IPv6 communications protocols
options     IPSEC            # IP (v4/v6) security
options     IPSEC_SUPPORT        # Allow kldload of ipsec and tcpmd5
options     TCP_OFFLOAD        # TCP offload
options     SCTP            # Stream Control Transmission Protocol
options     FFS            # Berkeley Fast Filesystem
options     SOFTUPDATES        # Enable FFS soft updates support
options     UFS_ACL            # Support for access control lists
options     UFS_DIRHASH        # Improve performance on big directories
options     UFS_GJOURNAL        # Enable gjournal-based UFS journaling
options     QUOTA            # Enable disk quotas for UFS
options     MD_ROOT            # MD is a potential root device
#options     NFSCL            # Network Filesystem Client
#options     NFSD            # Network Filesystem Server
#options     NFSLOCKD        # Network Lock Manager
#options     NFS_ROOT        # NFS usable as /, requires NFSCL
options     MSDOSFS            # MSDOS Filesystem
options     CD9660            # ISO 9660 Filesystem
options     PROCFS            # Process filesystem (requires PSEUDOFS)
options     PSEUDOFS        # Pseudo-filesystem framework
options     GEOM_PART_GPT        # GUID Partition Tables.
options     GEOM_RAID        # Soft RAID functionality.
options     GEOM_LABEL        # Provides labelization
#options     COMPAT_FREEBSD32    # Compatible with i386 binaries
#options     COMPAT_FREEBSD4        # Compatible with FreeBSD4
#options     COMPAT_FREEBSD5        # Compatible with FreeBSD5
#options     COMPAT_FREEBSD6        # Compatible with FreeBSD6
#options     COMPAT_FREEBSD7        # Compatible with FreeBSD7
#options     COMPAT_FREEBSD9        # Compatible with FreeBSD9
#options     COMPAT_FREEBSD10    # Compatible with FreeBSD10
options     SCSI_DELAY=5000        # Delay (in ms) before probing SCSI
options     KTRACE            # ktrace(1) support
options     STACK            # stack(9) support
options     SYSVSHM            # SYSV-style shared memory
options     SYSVMSG            # SYSV-style message queues
options     SYSVSEM            # SYSV-style semaphores
options     _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options     PRINTF_BUFR_SIZE=128    # Prevent printf output being interspersed.
options     KBD_INSTALL_CDEV    # install a CDEV entry in /dev
options     HWPMC_HOOKS        # Necessary kernel hooks for hwpmc(4)
options     AUDIT            # Security event auditing
options     CAPABILITY_MODE        # Capsicum capability mode
options     CAPABILITIES        # Capsicum capabilities
options     MAC            # TrustedBSD MAC Framework
options     KDTRACE_FRAME        # Ensure frames are compiled in
options     KDTRACE_HOOKS        # Kernel DTrace hooks
options     DDB_CTF            # Kernel ELF linker loads CTF data
options     INCLUDE_CONFIG_FILE    # Include this file in kernel
options     RACCT            # Resource accounting framework
options     RACCT_DEFAULT_TO_DISABLED # Set kern.racct.enable=0 by default
options     RCTL            # Resource limits

# Debugging support.  Always need this:
options     KDB            # Enable kernel debugger support.
options     KDB_TRACE        # Print a stack trace for a panic.
options        KDB_UNATTENDED

# Make an SMP-capable kernel by default
options     SMP            # Symmetric MultiProcessor Kernel
options     DEVICE_NUMA        # I/O Device Affinity
options     EARLY_AP_STARTUP

# CPU frequency control
device        cpufreq

# Bus support.
device        acpi
options     ACPI_DMAR
device        pci
options     PCI_HP            # PCI-Express native HotPlug
options        PCI_IOV            # PCI SR-IOV support

# ATA controllers
device        ahci            # AHCI-compatible SATA controllers
device        ata            # Legacy ATA/SATA controllers

# ATA/SCSI peripherals
device        scbus            # SCSI bus (required for ATA/SCSI)
device        ch            # SCSI media changers
device        da            # Direct Access (disks)
#device        sa            # Sequential Access (tape etc)
#device        cd            # CD
device        pass            # Passthrough device (direct ATA/SCSI access)
#device        ses            # Enclosure Services (SES and SAF-TE)
#device        ctl            # CAM Target Layer

# NVM Express (NVMe) support
device        nvme            # base NVMe driver
device        nvd            # expose NVMe namespaces as disks, depends on nvme

# atkbdc0 controls both the keyboard and the PS/2 mouse
device        atkbdc            # AT keyboard controller
device        atkbd            # AT keyboard
device        psm            # PS/2 mouse

device        kbdmux            # keyboard multiplexer

device        vga            # VGA video card driver
options     VESA            # Add support for VESA BIOS Extensions (VBE)

device        splash            # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device        sc
options     SC_PIXEL_MODE        # add support for the raster text mode

# vt is the new video console driver
device        vt
device        vt_vga
device        vt_efifb

device        agp            # support several AGP chipsets

# Serial (COM) ports
device        uart            # Generic UART driver

# PCI Ethernet NICs.
device        igb            # Intel PRO/1000 PCIE Server Gigabit Family

# Wireless NIC cards
device        wlan            # 802.11 support
options     IEEE80211_DEBUG        # enable debug msgs
options     IEEE80211_AMPDU_AGE    # age frames in AMPDU reorder q's
options     IEEE80211_SUPPORT_MESH    # enable 802.11s draft support
device        wlan_wep        # 802.11 WEP support
device        wlan_ccmp        # 802.11 CCMP support
device        wlan_tkip        # 802.11 TKIP support
device        wlan_amrr        # AMRR transmit rate control algorithm
device        an            # Aironet 4500/4800 802.11 wireless NICs.
device        ath            # Atheros NICs
device        ath_pci            # Atheros pci/cardbus glue
device        ath_hal            # pci/cardbus chip support
options     AH_SUPPORT_AR5416    # enable AR5416 tx/rx descriptors
options     AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation
options     ATH_ENABLE_11N        # Enable 802.11n support for AR5416 and later
device        ath_rate_sample        # SampleRate tx rate control for ath
#device        bwi            # Broadcom BCM430x/BCM431x wireless NICs.
#device        bwn            # Broadcom BCM43xx wireless NICs.
device        ipw            # Intel 2100 wireless NICs.
device        iwi            # Intel 2200BG/2225BG/2915ABG wireless NICs.
device        iwn            # Intel 4965/1000/5000/6000 wireless NICs.
device        malo            # Marvell Libertas wireless NICs.
device        mwl            # Marvell 88W8363 802.11n wireless NICs.
device        ral            # Ralink Technology RT2500 wireless NICs.
device        wi            # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
device        wpi            # Intel 3945ABG wireless NICs.

# Pseudo devices.
device        loop            # Network loopback
device        random            # Entropy device
device        padlock_rng        # VIA Padlock RNG
device        rdrand_rng        # Intel Bull Mountain RNG
device        ether            # Ethernet support
device        vlan            # 802.1Q VLAN support
device        tun            # Packet tunnel.
device        md            # Memory "disks"
device        gif            # IPv6 and IPv4 tunneling
device        firmware        # firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device        bpf            # Berkeley packet filter

# USB support
#options     USB_DEBUG        # enable debug msgs
#device        uhci            # UHCI PCI->USB interface
#device        ohci            # OHCI PCI->USB interface
#device        ehci            # EHCI PCI->USB interface (USB 2.0)
#device        xhci            # XHCI PCI->USB interface (USB 3.0)
#device        usb            # USB Bus (required)
#device        ukbd            # Keyboard
#device        umass            # Disks/Mass storage - Requires scbus and da

# Sound support
device        sound            # Generic sound driver (required)
device        snd_cmi            # CMedia CMI8338/CMI8738
device        snd_csa            # Crystal Semiconductor CS461x/428x
device        snd_emu10kx        # Creative SoundBlaster Live! and Audigy
device        snd_es137x        # Ensoniq AudioPCI ES137x
device        snd_hda            # Intel High Definition Audio
device        snd_ich            # Intel, NVidia and other ICH AC'97 Audio
device        snd_via8233        # VIA VT8233x Audio

# VirtIO support
device        virtio            # Generic VirtIO bus (required)
device        virtio_pci        # VirtIO PCI device
device        vtnet            # VirtIO Ethernet device
device        virtio_blk        # VirtIO Block device
device        virtio_scsi        # VirtIO SCSI device
device        virtio_balloon        # VirtIO Memory Balloon device

# HyperV drivers and enhancement support
device        hyperv            # HyperV drivers 

# Xen HVM Guest Optimizations
# NOTE: XENHVM depends on xenpci.  They must be added or removed together.
options     XENHVM            # Xen HVM kernel infrastructure
device        xenpci            # Xen HVM Hypervisor services driver

# Netmap provides direct access to TX/RX rings on supported NICs
device        netmap            # netmap(4) support

# The crypto framework is required by IPSEC
device        crypto            # Required by IPSEC
```


----------



## k.jacker (May 16, 2019)

Before you seek further, you should simply boot the GENERIC kernel and see if the messages disappear.
That you'll at least know if your custom kernel is involved or not.


----------



## SirDice (May 16, 2019)

Syslog running with `-ss` is good, that excludes the issue being caused by remote syslog messages. And I agree with k.jacker switch to GENERIC to test, again to rule things out. The more things we can rule out the narrower our search will become. 

"Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth."


----------



## ralphbsz (May 16, 2019)

I'll ask my standard obnoxious question: Why are you using a custom kernel?  If the pre-compiled generic kernel would work, you should probably use it.  There have to be pretty good reasons to configure a custom kernel, to balance the unavoidable cost of using a custom one.  The biggest cost is the plausibility of making small mistakes, which then may have "interesting" effects.


----------



## Buck (May 16, 2019)

No need for obnoxious questions, as I've already recompiled and reinstalled GENERIC and those log lines keep appearing as before.

My custom kernel disables USB for instance which seriously misbehaves on my ASRock board, pollutes the log and slows down reboots considerably.


----------



## k.jacker (May 16, 2019)

I'd rather disable that USB controller in the BIOS, much easier.

I had an Asrock with an USB3.0 Asmedia controller than allways made the shutdown hang for a while.
If I recall correctly, the following in /etc/sysctl.conf fixed that.
`hw.usb.no_shutdown_wait=1`


----------



## Buck (May 16, 2019)

It's not only shutdown that is affected, startup too. For me configuring and recompiling a custom kernel is not much work at all. I've done it many times on many machines, and I know the pitfalls of upgrading to the next release so no trouble there. Still the issue with those log lines remains... I haven't seen those before anywhere ever, either. The config is pretty standard as well, I've only changed some lines in syslogd.conf to redirect apache and dovecot to their logs.


----------



## Datapanic (May 17, 2019)

Do you have anything in /etc/syslog.d/ or /usr/local/etc/syslog.d/ ?


----------



## k.jacker (May 17, 2019)

Buck said:


> I've only changed some lines in syslogd.conf to redirect apache and dovecot to their logs.


As syslogd was already suspected as the cause by SirDice earlier, why don't you just do the same thing as with your kernel, rule out your config causes it, by simply removing your lines and see if the messages disappear.


----------



## Buck (May 19, 2019)

Nothing in either /etc/syslog.d/ or /usr/local/etc/syslog.d/.
Went ahead and replaced syslog.conf with stock, rebooted to be sure - same thing.

The messages don't appear immediately after reboot. It takes approximately an hour and a half and then it's more or less a steady stream.


----------



## Terry_Kennedy (Jun 3, 2019)

Buck said:


> Eight character long random strings that go like
> [snip]
> What does that mean?


I've noticed something in 12-STABLE (amd64) that may be related:

```
May 31 04:51:49 www4 shutdown[87150]: reboot by terry: 
May 31 04:52:10 www4 kernel: .
May 31 04:52:11 www4 kernel: , 1279.
May 31 04:52:13 www4 kernel: .
May 31 04:52:13 www4 ntpd[1101]: ntpd exiting on signal 15 (Terminated)
May 31 04:52:13 www4 kernel: .
May 31 04:52:14 www4 kernel: , 1101.
May 31 04:52:14 www4 syslogd: exiting on signal 15
```
In this case, it looks like the kernel is trying to print information about some processes at shutdown time (see the correct message about ntpd PID 1101 exiting, and then some garbage and 1101 repeated) but has missing format specifiers in the kernel printf() leading to missing text.


----------

