# httpd: _umtx_op -1 errno 60 Operation timed out



## frijsdijk (Feb 27, 2014)

Hi,

On a couple of pretty highly loaded Apache servers running FreeBSD 9.1-p7, when ktracing such a process, I see that the kernel is wasting a lot of time in _umtx_op, also getting timeouts there (ERRNO 60). Some output:

```
68713 httpd    0.099217 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.142066 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.156235 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.182471 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.208281 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.223532 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.249026 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.256341 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.456772 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.469333 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.573386 RET   _umtx_op -1 errno 60 Operation timed out
 68713 httpd    0.986361 RET   _umtx_op -1 errno 60 Operation timed out
```
.. there are more in this ktrace (total duration of ktrace=~60 seconds), the total time summing up these time-outs is roughly 5 seconds.

CPU Specs:


```
FreeBSD 9.1-RELEASE-p7 #0: Mon Sep  9 21:34:37 UTC 2013
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (2400.13-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206c2  Family = 6  Model = 2c  Stepping = 2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ee3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 17184063488 (16388 MB)
avail memory = 16508317696 (15743 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <123011 APIC2000>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
```

At the time that the ktrace was made, the server's load was:


```
User: 40%
System: 16%
Nice: 4%
Idle: 40%
```

Memory usage (16GB total):


```
Active: 444MB
Inactive: 1.1GB
Wired: 5GB
Buffers: 1.6GB
Cache: 14.5MB
Free: 9GB
Swap: 114MB
```

The server was doing roughly 60Mbit of traffic.

Apache runs mod_pagespeed.

Apache stats:
130 requests/s

Apache config snippits:


```
Timeout 30
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 3
UseCanonicalName Off
AccessFileName .htaccess
ServerTokens prod
ServerSignature off
HostnameLookups Off
```


```
ServerLimit          1400
MaxClients          1400
MaxRequestsPerChild   0
```

This server has SSD's, load on these was never an issue (<1%). This seemed to be pure CPU load, with a lot of overhead (high system load).

While searching for this kind of issues on the web, I did come across some posts stating that this kernel call umtx_op is related to threading (libthr to which Apache is linked), and there were some 'issues' in FreeBSD 9, or 9.x, and that these should be working better in 10.0.

I'd be happy to upgrade these machines to 10.0, but I'd like to ask around if somebody can confirm this?

Some more info:


```
# httpd -V
Server version: Apache/2.2.26 (FreeBSD)
Server built:   Dec 15 2013 20:56:41
Server's Module Magic Number: 20051115:33
Server loaded:  APR 1.4.8, APR-Util 1.5.3
Compiled using: APR 1.4.8, APR-Util 1.5.3
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_FLOCK_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local"
 -D SUEXEC_BIN="/usr/local/sbin/suexec"
 -D DEFAULT_PIDLOG="/var/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
 -D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"
```


```
# file /usr/local/sbin/httpd
/usr/local/sbin/httpd: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), for FreeBSD 9.1, not stripped
```


```
# ldd /usr/local/sbin/httpd
/usr/local/sbin/httpd:
        libm.so.5 => /lib/libm.so.5 (0x800876000)
        libpcre.so.3 => /usr/local/lib/libpcre.so.3 (0x800a97000)
        libaprutil-1.so.5 => /usr/local/lib/libaprutil-1.so.5 (0x800cfa000)
        libdb-4.2.so.2 => /usr/local/lib/libdb-4.2.so.2 (0x800f1e000)
        libgdbm.so.4 => /usr/local/lib/libgdbm.so.4 (0x801204000)
        libintl.so.9 => /usr/local/lib/libintl.so.9 (0x80140b000)
        libexpat.so.6 => /usr/local/lib/libexpat.so.6 (0x801615000)
        libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x801839000)
        libapr-1.so.4 => /usr/local/lib/libapr-1.so.4 (0x801b35000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x801d61000)
        libthr.so.3 => /lib/libthr.so.3 (0x801f80000)
        libc.so.7 => /lib/libc.so.7 (0x8021a2000)
```


```
# pkg info apache22
apache22-2.2.26
Name           : apache22
Version        : 2.2.26
Installed on   : Wed Dec 18 11:59:05 CET 2013
Origin         : www/apache22
Architecture   : freebsd:9:x86:64
Prefix         : /usr/local
Categories     : www ipv6
Maintainer     : apache@FreeBSD.org
WWW            : http://httpd.apache.org/
Comment        : Version 2.2.x of Apache web server with prefork MPM.
Options        :
        ACTIONS        : on
        ALIAS          : on
        ASIS           : on
        AUTHNZ_LDAP    : off
        AUTHN_ALIAS    : on
        AUTHN_ANON     : on
        AUTHN_DBD      : off
        AUTHN_DBM      : on
        AUTHN_DEFAULT  : on
        AUTHN_FILE     : on
        AUTHZ_DBM      : on
        AUTHZ_DEFAULT  : on
        AUTHZ_GROUPFILE: on
        AUTHZ_HOST     : on
        AUTHZ_OWNER    : on
        AUTHZ_USER     : on
        AUTH_BASIC     : on
        AUTH_DIGEST    : on
        AUTOINDEX      : on
        BUCKETEER      : off
        CACHE          : on
        CASE_FILTER    : off
        CASE_FILTER_IN : off
        CERN_META      : on
        CGI            : on
        CGID           : off
        CHARSET_LITE   : on
        DAV            : on
        DAV_FS         : on
        DAV_LOCK       : off
        DBD            : off
        DEFLATE        : on
        DIR            : on
        DISK_CACHE     : on
        DUMPIO         : on
        ENV            : on
        EXPIRES        : on
        EXT_FILTER     : off
        FILE_CACHE     : on
        FILTER         : on
        HEADERS        : on
        IMAGEMAP       : on
        INCLUDE        : on
        INFO           : on
        IPV4_MAPPED    : on
        LDAP           : off
        LOGIO          : on
        LOG_CONFIG     : on
        LOG_FORENSIC   : off
        MEM_CACHE      : off
        MIME           : on
        MIME_MAGIC     : on
        NEGOTIATION    : on
        OPTIONAL_FN_EXPORT: off
        OPTIONAL_FN_IMPORT: off
        OPTIONAL_HOOK_EXPORT: off
        OPTIONAL_HOOK_IMPORT: off
        PROXY          : on
        PROXY_AJP      : off
        PROXY_BALANCER : off
        PROXY_CONNECT  : on
        PROXY_FTP      : on
        PROXY_HTTP     : on
        PROXY_SCGI     : off
        REQTIMEOUT     : on
        REWRITE        : on
        SETENVIF       : on
        SPELING        : on
        SSL            : on
        STATUS         : on
        SUBSTITUTE     : off
        SUEXEC         : on
        SUEXEC_RSRCLIMIT: off
        SUEXEC_USERDIR : off
        UNIQUE_ID      : on
        USERDIR        : on
        USERTRACK      : on
        VERSION        : on
        VHOST_ALIAS    : on
Shared Libs required:
        libpcre.so.3
        libintl.so.9
        libiconv.so.3
        libgdbm.so.4
        libexpat.so.6
        libdb-4.2.so.2
        libaprutil-1.so.5
        libapr-1.so.4
Shared Libs provided:
        mod_vhost_alias.so
        mod_version.so
        mod_usertrack.so
        mod_userdir.so
        mod_unique_id.so
        mod_suexec.so
        mod_status.so
        mod_ssl.so
        mod_speling.so
        mod_setenvif.so
        mod_rewrite.so
        mod_reqtimeout.so
        mod_proxy_http.so
        mod_proxy_ftp.so
        mod_proxy_connect.so
        mod_proxy.so
        mod_negotiation.so
        mod_mime_magic.so
        mod_mime.so
        mod_logio.so
        mod_log_config.so
        mod_info.so
        mod_include.so
        mod_imagemap.so
        mod_headers.so
        mod_filter.so
        mod_file_cache.so
        mod_expires.so
        mod_env.so
        mod_dumpio.so
        mod_disk_cache.so
        mod_dir.so
        mod_deflate.so
        mod_dav_fs.so
        mod_dav.so
        mod_charset_lite.so
        mod_cgi.so
        mod_cern_meta.so
        mod_cache.so
        mod_autoindex.so
        mod_authz_user.so
        mod_authz_owner.so
        mod_authz_host.so
        mod_authz_groupfile.so
        mod_authz_default.so
        mod_authz_dbm.so
        mod_authn_file.so
        mod_authn_default.so
        mod_authn_dbm.so
        mod_authn_anon.so
        mod_authn_alias.so
        mod_auth_digest.so
        mod_auth_basic.so
        mod_asis.so
        mod_alias.so
        mod_actions.so
Flat size      : 15.9MiB
Description    :
The Apache HTTP Server Project is an effort to develop and maintain an
open-source HTTP server for various modern desktop and server operating
systems, such as UNIX and Windows NT. The goal of this project is to
provide a secure, efficient and extensible server which provides HTTP
services in sync with the current HTTP standards.
The 2.x branch of Apache Web Server includes several improvements like
threading, use of APR, native IPv6 and SSL support, and many more.

WWW: http://httpd.apache.org/
```


----------



## SirDice (Feb 27, 2014)

Are the clients actually experiencing issues? Or is it just something you noticed? 

If you really have a high load I can highly recommend using net/haproxy and load-balancing the traffic over multiple Apache servers. I've recently set this up for a small internet company and they are extremely happy with the way it performs. Traffic comes in on a carp(4) interface on two HAProxy machines. These balance traffic to 3 or 4 web servers. The benefit (besides the load-balancing) is that you can take one web server offline for maintenance for example and HAProxy will automatically spread the load over the remaining servers. And with HAProxy you can, for example, send traffic for Site 1 to Server 1 and 2 and for Site 2 to Server 3, 4 and 5. If you use net/haproxy-devel you can even balance SSL sites.


----------



## frijsdijk (Feb 27, 2014)

SirDice said:
			
		

> Are the clients actually experiencing issues? Or is it just something you noticed?
> 
> If you really have a high load I can highly recommend using net/haproxy and load-balancing the traffic over multiple Apache servers. I've recently set this up for a small internet company and they are extremely happy with the way it performs. Traffic comes in on a carp(4) interface on two HAProxy machines. These balance traffic to 3 or 4 web servers. The benefit (besides the load-balancing) is that you can take one web server offline for maintenance for example and HAProxy will automatically spread the load over the remaining servers. And with HAProxy you can, for example, send traffic for Site 1 to Server 1 and 2 and for Site 2 to Server 3, 4 and 5. If you use net/haproxy-devel you can even balance SSL sites.



Actually, I was showing data of one of 5 servers behind a keepalived lvs setup using Linux.

No info/knowlegde on the 'problems' that would/could exist in 9.x version of FreeBSD?


----------



## SirDice (Feb 27, 2014)

frijsdijk said:
			
		

> No info/knowlegde on the 'problems' that would/could exist in 9.x version of FreeBSD?


Not that I know of, all our servers are running 9.2-RELEASE at the moment. Although, because of the load-balancers, none of the Apache servers are actually showing a high load (we are pushing around 60Mb/s upstream in total). We did tweak kern.ipc.somaxconn a bit but that's pretty much all the tuning we did.

I actually replaced a Linux Heartbeat/ldirector with two FreeBSD HAProxy machines. It's a slightly different setup as the Linux HA/Ldirector balanced based on IP (layer 3) and HAProxy is a full reverse proxy (balances on Layer 7). The HAProxy has keep-alive on towards the clients but it's off towards the web servers (to keep the number of concurrent connections down).


----------



## frijsdijk (Feb 27, 2014)

SirDice said:
			
		

> frijsdijk said:
> 
> 
> 
> ...



Any specific options in your /etc/make.conf if I may ask?

Btw, I do have less problems with 9.2-RELEASE, compared to 9.1-RELEASE, specificaly under high loads.


----------



## SirDice (Feb 27, 2014)

frijsdijk said:
			
		

> Any specific options in your /etc/make.conf if I may ask?


None actually. I do all building on one of our control servers with ports-mgmt/poudriere but besides WITHOUT_X11 I haven't set anything special.

Oh, I just noticed you're using mod_pagespeed. That's something we don't use, maybe that's the culprit?


----------



## frijsdijk (Feb 27, 2014)

SirDice said:
			
		

> frijsdijk said:
> 
> 
> 
> ...



ports-mgmt/poudriere Rules! I do the same.

I'll be upgrading one of these 5 machines to 10.0-RELEASE first. See what happens..

Thanks!


-- Frederique


----------



## frijsdijk (Mar 4, 2014)

Well, after the upgrade to 10.0-RELEASE, I no longer see the "errno 60" in ktraces of Apache. That's good!


----------

