# cron jobs running several seconds after scheduled



## mathison (Aug 31, 2010)

Greetings,

I have a basically clean install of FreeBSD 8.1 stable (i386) running under as a virtual machine on an ESX host.

uname:

```
FreeBSD NetRouter 8.1-STABLE FreeBSD 8.1-STABLE #0: Thu Aug 19 09:37:22 EDT 2010 /usr/obj/usr/src/sys/NETROUTER  i386
```

The kernel was recompiled using the kernel config shown below. 
Also, several sysctls parameters were adjusted as shown below.


The cron daemon is running only with the -s flag, (though I did specify -J0 to see if it had any effect - it did not.)

The issue I am having are that cron jobs are not executing until 10 seconds or later after 'scheduled'. The pattern is consistent regardless of the time of day, load, etc. Though I realize that cron is only accurate to the minute - meaning the task is only expected to run sometime in the scheduled minute (0-59 seconds), my other FreeBSD servers all seem to run these tasks at the 0 second, with no multi-second delays.

What should I be looking into to determine what is so different with this installation that it would insert 10 seconds or more of delay in the cron processing chain? [I did enable several debug flags on the cron daemon, hoping to see what was happening in those first ten seconds, but that did not enlighten me.]

Thanks for any pointers.
Neil 

/var/log/cron:

```
Aug 31 06:30:10 NetRouter /usr/sbin/cron[44544]: (root) CMD (/usr/libexec/atrun)
Aug 31 06:33:10 NetRouter /usr/sbin/cron[44546]: (operator) CMD (/usr/libexec/save-entropy)
Aug 31 06:35:10 NetRouter /usr/sbin/cron[44558]: (root) CMD (/usr/libexec/atrun)
Aug 31 06:40:11 NetRouter /usr/sbin/cron[44562]: (root) CMD (/usr/libexec/atrun)
```

sysctl.conf:

```
kern.ipc.somaxconn=1024

# polling tuneables
kern.polling.burst_max=1000
kern.polling.idle_poll=0
kern.polling.each_burst=50

# network tuneables
# -----------------
# maximum segment size (1500 bytes - 40 header bytes)
net.inet.tcp.mssdflt=1460
# use an even multiple of MSS (78840/1460 = 54)
net.inet.tcp.recvspace=78840
net.inet.tcp.sendspace=78840
# 54 packets fit into 78840 bytes
net.inet.tcp.slowstart_flightsize=54

# turn on tcp delayed ACK - increases likelyhood that the first response will
# have first ACK piggybacked
net.inet.tcp.delayed_ack=1
net.inet.tcp.delacktime=100

# enable tcp inflight (using manpage recommendations)
net.inet.tcp.inflight.enable=1
net.inet.tcp.inflight.min=6144

# loopback tuning
net.local.stream.sendspace=82320
net.local.stream.recvspace=82320
net.inet.tcp.local_slowstart_flightsize=10
net.inet.tcp.nolocaltimewait=1

# host cache
net.inet.tcp.hostcache.expire=1
```

Kernel config file:

```
cpu I686_CPU                                                                             
ident NETROUTER                                                                       

makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols

options         SCHED_ULE               # ULE scheduler
options         PREEMPTION              # Enable kernel thread preemption
options         INET                    # InterNETworking                
options         SCTP                    # Stream Control Transmission Protocol
options         FFS                     # Berkeley Fast Filesystem            
options         SOFTUPDATES             # Enable FFS soft updates support     
options         UFS_ACL                 # Support for access control lists    
options         UFS_DIRHASH             # Improve performance on big directories
options         UFS_GJOURNAL            # Enable gjournal-based UFS journaling  
options         MD_ROOT                 # MD is a potential root device         
options         NFSCLIENT               # Network Filesystem Client             
options         NFSSERVER               # Network Filesystem Server             
options         NFSLOCKD                # Network Lock Manager                  
options         NFS_ROOT                # NFS usable as /, requires NFSCLIENT   
options         MSDOSFS                 # MSDOS Filesystem                      
options         CD9660                  # ISO 9660 Filesystem                   
options         PROCFS                  # Process filesystem (requires PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework           
options         GEOM_PART_GPT           # GUID Partition Tables.                
options         GEOM_LABEL              # Provides labelization                 
options         COMPAT_43TTY            # BSD 4.3 TTY compat (sgtty)            
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4              
options         COMPAT_FREEBSD5         # Compatible with FreeBSD5              
options         COMPAT_FREEBSD6         # Compatible with FreeBSD6              
options         COMPAT_FREEBSD7         # Compatible with FreeBSD7              
options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI     
options         KTRACE                  # ktrace(1) support                     
options         STACK                   # stack(9) support                      
options         SYSVSHM                 # SYSV-style shared memory              
options         SYSVMSG                 # SYSV-style message queues             
options         SYSVSEM                 # SYSV-style semaphores                 
options         P1003_1B_SEMAPHORES     # POSIX-style semaphores                
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options         PRINTF_BUFR_SIZE=128    # Prevent printf output being interspersed.
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev             
options         HWPMC_HOOKS             # Necessary kernel hooks for hwpmc(4)      
options         AUDIT                   # Security event auditing                  
options         MAC                     # TrustedBSD MAC Framework                 
options         FLOWTABLE               # per-cpu routing cache                    

options         HZ=1000
options         DEVICE_POLLING
options         ACCEPT_FILTER_HTTP
options         ZERO_COPY_SOCKETS 

#options        KDTRACE_HOOKS           # Kernel DTrace hooks
options         INCLUDE_CONFIG_FILE     # Include this file in kernel

# To make an SMP kernel, the next two lines are needed
options         SMP                     # Symmetric MultiProcessor Kernel
device          apic                    # I/O APIC                       

# CPU frequency control
device          cpufreq

# packet filter support
device          pf     
device          pflog  
device          pfsync 

# ALTQ support
options         ALTQ
options         ALTQ_CBQ
options         ALTQ_RED
options         ALTQ_RIO
options         ALTQ_HFSC
options         ALTQ_CDNR
options         ALTQ_PRIQ

# Bus support.
device          acpi
device          pci 

# ATA and ATAPI devices
device          ata    
device          atadisk         # ATA disk drives
device          atapicd         # ATAPI CDROM drives
options         ATA_STATIC_ID   # Static device numbering

# SCSI Controllers
device          mpt             # LSI-Logic MPT-Fusion


# SCSI peripherals
device          scbus           # SCSI bus (required for SCSI)
device          da              # Direct Access (disks)       
device          cd              # CD                          
device          pass            # Passthrough device (direct SCSI access)
device          ses             # SCSI Environmental Services (and SAF-TE)

# atkbdc0 controls both the keyboard and the PS/2 mouse
device          atkbdc          # AT keyboard controller
device          atkbd           # AT keyboard
device          psm             # PS/2 mouse

device          kbdmux          # keyboard multiplexer

device          vga             # VGA video card driver

device          splash          # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device          sc

device          agp             # support several AGP chipsets

# PCI Ethernet NICs.
device          em              # Intel PRO/1000 Gigabit Ethernet Family

# Pseudo devices.
device          loop            # Network loopback
device          random          # Entropy device
device          ether           # Ethernet support
device          tun             # Packet tunnel.
device          pty             # BSD-style compatibility pseudo ttys
device          md              # Memory "disks"
device          gif             # IPv6 and IPv4 tunneling
device          faith           # IPv6-to-IPv4 relaying (translation)
device          firmware        # firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device          bpf             # Berkeley packet filter
```


----------



## graudeejs (Aug 31, 2010)

First of all read this:
http://forums.freebsd.org/showthread.php?t=8816

No clue, but I suspect something related to *options HZ=1000*


----------



## SirDice (Aug 31, 2010)

mathison said:
			
		

> The issue I am having are that cron jobs are not executing until 10 seconds or later after 'scheduled'.


Nowhere is it defined a cron job should start on the second precisely.  



> The cron utility then wakes up every minute, examining all stored crontabs, checking each command to see if it should be run in the current minute.


From cron(8)

As long as it's within that minute it's fine. But as killasmurf86 noted, it's most likely due to kern.hz=1000.


----------



## DutchDaemon (Aug 31, 2010)

Thanks for carefully reading your signup email, mathison. Much appreciated.


----------



## phoenix (Sep 2, 2010)

Running in a VM, you give up all requirements for accurate time.  In fact, it's very likely that time will go backwards, depending on the clock source.

Read through the dmesg() output to see which clock source is being selected.  Then play around with the different sources.

Search the 'Net as well.  There are some clock sources that are more accurate in a VM situation.

Also, disable powerd via /etc/rc.conf as that will affect the stability of the clock.

In the grand scheme of things, though, you aren't actually experiencing a problem until cron jobs begin to fire 70 seconds after they are scheduled.  

@DD:  We really need a sarcasm emoticon.


----------



## anomie (Sep 2, 2010)

mathison said:
			
		

> Thanks for any pointers.



Please read: 

 FreeBSD under VMWare
 FreeBSD forums: vmware-tools...

If you're not able to get to a happy place with the info contained at the first URL, then consider vmware-tools and read through the posts on the second URL.


----------

