# My server keeps running out of RAM



## ghostcorps (Apr 6, 2012)

Hi guys,

 As of a day or two ago my server is shutting down due to low RAM, according to my host. But, I can not pin down the cause, I have replicated it by running a find command on /usr/local/etc/apache22/extras so no huge task but it has occurred both times I ran the command today. It is a media streaming server so it should not have any trouble with such a small request.

 I have started looking through /var but so far I can not find out what is freaking it out. 

 Here is *top* at idle:



```
last pid:  2750;  load averages:  0.00,  0.00,  0.00
62 processes:  2 running, 60 sleeping
CPU:  0.3% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.5% idle
Mem: 76M Active, 83M Inact, 51M Wired, 444K Cache, 85M Buf, 521M Free
Swap: 988M Total, 988M Free
Order to sort: [B]res[/B]
  PID USERNAME     THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 2609     88        18  44    0 86896K 45168K ucond    0:01  0.00% mysqld
 2694 www            1  45    0   124M 27496K lockf    0:01  0.00% httpd
 2698 www            1  50    0   122M 25460K lockf    0:01  0.00% httpd
 2695 www            1  50    0   122M 25420K lockf    0:01  0.00% httpd
 2697 www            1  50    0   122M 25420K lockf    0:01  0.00% httpd
 2696 www            1  44    0   112M 14808K lockf    0:00  0.00% httpd
 2706 www            1  44    0   112M 14752K lockf    0:00  0.00% httpd
 2705 www            1  44    0   112M 14752K kqread   0:00  0.00% httpd
 2693 root           1  44    0   112M 14744K select   0:00  0.00% httpd
 1450 www            1  44    0 71420K  7248K accept   0:00  0.00% httpd
 1451 www            1  63    0 71420K  7248K accept   0:00  0.00% httpd
 1452 www            1  63    0 71420K  7248K accept   0:00  0.00% httpd
 1453 www            1  63    0 71420K  7248K accept   0:00  0.00% httpd
 1454 www            1  63    0 71420K  7248K accept   0:00  0.00% httpd
 1376 root           1  44    0 71420K  7244K select   0:00  0.00% httpd
 2667 admin          1  44    0 38104K  5176K RUN      0:00  0.00% sshd
 2664 root           1  45    0 38104K  5168K sbwait   0:00  0.00% sshd
 1405 root           1  44    0 26172K  4500K select   0:00  0.00% sshd
 1242 root           1  44    0 11092K  4184K select   0:00  0.00% openvpn
 1411 root           1  44    0 12096K  4080K select   0:00  0.00% sendmail
 1417 smmsp          1  76    0 12096K  4012K pause    0:00  0.00% sendmail
 1780 smmsp          1  76    0 12004K  3864K pause    0:00  0.00% sendmail
 1774 root           1  44    0 12004K  3616K select   0:00  0.00% sendmail
 2670 root           1  44    0 10216K  2800K wait     0:00  0.00% bash
 2668 admin          1  44    0 10216K  2796K wait     0:00  0.00% bash
 2750 root           1  44    0  9336K  2288K RUN      0:00  0.00% top
 2669 admin          1  44    0 21668K  2008K wait     0:00  0.00% su
 2040     88         1  76    0  8264K  1860K wait     0:00  0.00% sh
 2077 root           1  44    0  8080K  1636K nanslp   0:00  0.00% cron
 1787 root           1  44    0  7952K  1612K nanslp   0:00  0.00% cron
 1424 root           1  44    0  7952K  1612K nanslp   0:00  0.00% cron
 1913 root           1  44    0  7024K  1584K select   0:00  0.00% syslogd
 1089 root           1  44    0  7024K  1564K select   0:00  0.00% syslogd
 1611 root           1  44    0  6896K  1560K select   0:00  0.00% syslogd
 2287 root           1  76    0  9008K  1396K select   0:00  0.00% inetd
 2440 root           1  76    0  6892K  1288K ttyin    0:00  0.00% getty
 2445 root           1  76    0  6892K  1288K ttyin    0:00  0.00% getty
 2441 root           1  76    0  6892K  1288K ttyin    0:00  0.00% getty
 2446 root           1  76    0  6892K  1288K ttyin    0:00  0.00% getty
 2447 root           1  76    0  6892K  1288K ttyin    0:00  0.00% getty
 2442 root           1  76    0  6892K  1288K ttyin    0:00  0.00% getty
 2443 root           1  76    0  6892K  1288K ttyin    0:00  0.00% getty
 2444 root           1  76    0  6892K  1288K ttyin    0:00  0.00% getty
  115 root           1  76    0  2744K  1024K pause    0:00  0.00% adjkerntz
  852 root           1  44    0  3204K   724K select   0:00  0.00% devd
```

 Can anyone please suggest a way to find the culprit? This is a production server and I am getting my arse kicked every time it goes down


----------



## gkontos (Apr 6, 2012)

1) How exactly does it shut down?

2) What do your logs say when this happens? (/var/log/messages)

3) Is this a dedicated or a VPS?


----------



## blakjak (Apr 6, 2012)

*Your sever is running out of ram space*

you need to have a SWAP partition during your installation of the *F*reeBSD OS. This 
SWAP partition is used when your computer is running out of RAM space. I hope you have a swap partition?


----------



## ghostcorps (Apr 6, 2012)

gkontos said:
			
		

> 1) How exactly does it shut down?
> 
> 2) What do your logs say when this happens? (/var/log/messages)
> 
> 3) Is this a dedicated or a VPS?



Thanks for the questions.

It is a VPS

It is not entirely clear how it stalls, but we are forced to reboot it through the VM to get it back. It was down this morning, we restarted it and it ran for a few hours. I logged in via ssh to make some changes to the apache config (unrelated). I ran a find search looking for a string in the /extras folder and after opening the file the session stalled. At that point the website which runs off a jailed webserver went offline, but the fail over page on the host was still live, albeit very slow to load.

After a short time there is nothing on either page and I am forced to reboot. 


A few rules from /etc/ipfw.rules that are mentioned in messages:


```
$IPF 801 deny log all from any to HOST.SERVER 22-25
$IPF 900 deny log all from any to WEBSERVER.JAIL 1-79          
$IPF 910 allow log all from any to WEBSERVER.JAIL 80
$IPF 920 allow log all from any to WEBSERVER.JAIL 443
$IPF 930 deny log all from any to WEBSERVER.JAIL 81-442
$IPF 940 deny log all from any to WEBSERVER.JAIL 444-1934
```


/var/log/messages Starting from a flood of SYSERRs before the first crash, to now'ish. I have cut out a bunch of stuff that I didn't think was necessary.


```
Mar 27 13:32:12 DOMAIN sm-mta[48309]: q2RHW6bp048307: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 27 14:02:12 DOMAIN sm-mta[48509]: q2RI27mM048507: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 27 14:32:11 DOMAIN sm-mta[48630]: q2RIW6Fm048628: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 27 23:01:11 DOMAIN sm-mta[51704]: q2S316ug051625: SYSERR(root): webserver.URL.com. config error: mail loops back to me (MX problem?)
Mar 27 23:02:52 DOMAIN sm-mta[51860]: q2S32kYN051810: SYSERR(root): webserver.URL.com. config error: mail loops back to me (MX problem?)
Mar 27 23:02:52 DOMAIN sm-mta[51863]: q2S32lId051856: SYSERR(root): webserver.URL.com. config error: mail loops back to me (MX problem?)
Mar 27 23:47:10 DOMAIN sm-mta[56507]: q2S3l5fM056505: SYSERR(root): webserver.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 09:02:12 DOMAIN sm-mta[64836]: q2SD27Tx064834: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 09:32:11 DOMAIN sm-mta[64957]: q2SDW6uO064955: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 10:02:12 DOMAIN sm-mta[65156]: q2SE26gm065154: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 10:32:12 DOMAIN sm-mta[65280]: q2SEW7Xw065278: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 11:02:11 DOMAIN sm-mta[65482]: q2SF26lJ065480: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 11:32:12 DOMAIN sm-mta[65608]: q2SFW6H2065606: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 12:02:12 DOMAIN sm-mta[66145]: q2SG27eb066091: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 12:02:12 DOMAIN sm-mta[66147]: q2SG27ss066092: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 12:05:29 DOMAIN sm-mta[66335]: q2SG5OXj066330: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 12:05:29 DOMAIN sm-mta[66338]: q2SG5Oqt066331: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 12:10:01 DOMAIN sm-mta[70765]: q2SG9uMm070763: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 12:32:12 DOMAIN sm-mta[70880]: q2SGW7n6070878: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 13:02:12 DOMAIN sm-mta[71081]: q2SH27qc071079: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 13:32:12 DOMAIN sm-mta[71203]: q2SHW6EY071201: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 14:02:12 DOMAIN sm-mta[71407]: q2SI27SK071405: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 14:32:11 DOMAIN sm-mta[71531]: q2SIW6iW071529: SYSERR(root): database.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 23:01:11 DOMAIN sm-mta[74646]: q2T316R2074503: SYSERR(root): webserver.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 23:02:50 DOMAIN sm-mta[74804]: q2T32jvM074753: SYSERR(root): webserver.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 23:02:50 DOMAIN sm-mta[74807]: q2T32j9p074802: SYSERR(root): webserver.URL.com. config error: mail loops back to me (MX problem?)
Mar 28 23:50:41 DOMAIN sm-mta[79450]: q2T3oeYv079448: SYSERR(root): webserver.URL.com. config error: mail loops back to me (MX problem?)

***dmesg***

Apr  2 15:48:45 DOMAIN su: URLadmin to toor on /dev/pts/0
Apr  3 23:05:06 DOMAIN su: admin to root on /dev/pts/0
Apr  5 20:05:19 DOMAIN kernel: arp: XXX.XXX.XXX.3 moved from 00:ff:2d:81:3b:3c to 00:ff:03:09:cd:79 on tap0
Apr  5 20:05:35 DOMAIN sshd[xxx86]: error: PAM: authentication error for root from YYY.YYY.YYY
Apr  5 20:05:58 DOMAIN sshd[25888]: error: PAM: authentication error for toor from YYY.YYY.YYY
Apr  5 20:06:01 DOMAIN sshd[25888]: error: PAM: authentication error for toor from YYY.YYY.YYY
Apr  5 20:07:47 DOMAIN su: URLadmin to toor on /dev/pts/0
Apr  5 20:29:58 DOMAIN sshd[1401]: error: accept: Software caused connection abort
Apr  5 20:35:41 DOMAIN su: admin to root on /dev/pts/1
Apr  5 20:37:53 DOMAIN su: URLadmin to toor on /dev/pts/0
Apr  5 20:39:32 DOMAIN su: admin to root on /dev/pts/1
Apr  6 01:17:05 DOMAIN su: admin to root on /dev/pts/0
Apr  6 01:52:54 DOMAIN su: admin to root on /dev/pts/1


Apr  6 02:09:14 DOMAIN kernel: ipfw: limit 5 reached on entry 900
Apr  6 02:09:14 DOMAIN kernel: ipfw: limit 5 reached on entry 930
Apr  6 02:09:14 DOMAIN kernel: ipfw: limit 5 reached on entry 940
Apr  6 02:09:51 DOMAIN kernel: ipfw: limit 5 reached on entry 920
Apr  6 02:09:51 DOMAIN kernel: ipfw: limit 5 reached on entry 910
Apr  6 02:10:43 DOMAIN kernel: ipfw: limit 5 reached on entry 910
Apr  6 02:10:47 DOMAIN kernel: ipfw: limit 5 reached on entry 920
Apr  6 02:24:46 DOMAIN kernel: ipfw: limit 5 reached on entry 801

***dmesg***

Apr  6 03:58:42 DOMAIN kernel: ipfw: limit 5 reached on entry 801
Apr  6 03:59:38 DOMAIN fsck: /dev/da0s1e: 38 files, 145 used, 253670 free (30 frags, 31705 blocks, 0.0% fragmentation)
Apr  6 04:00:24 DOMAIN fsck: /dev/da0s1f: PARTIALLY TRUNCATED INODE I=711230
Apr  6 04:00:24 DOMAIN fsck: /dev/da0s1f: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY.

Apr  6 04:03:01 DOMAIN kernel: ipfw: limit 5 reached on entry 910
Apr  6 04:05:47 DOMAIN kernel: ipfw: limit 5 reached on entry 920
Apr  6 04:19:33 DOMAIN kernel: ipfw: limit 5 reached on entry 900
Apr  6 04:19:33 DOMAIN kernel: ipfw: limit 5 reached on entry 930
Apr  6 04:19:33 DOMAIN kernel: ipfw: limit 5 reached on entry 940

***dmesg***

Apr  6 04:52:01 DOMAIN kernel: ipfw: limit 5 reached on entry 910

Apr  6 04:52:54 DOMAIN fsck: /dev/da0s1e: 39 files, 145 used, 253670 free (30 frags, 31705 blocks, 0.0% fragmentation)
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: LINK COUNT FILE I=70669  OWNER=operator MODE=100400
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: SIZE=2048 MTIME=Apr  6 02:22 2012  COUNT 2 SHOULD BE 1 (ADJUSTED)
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: LINK COUNT FILE I=70680  OWNER=operator MODE=100400
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: SIZE=2048 MTIME=Apr  6 04:00 2012  COUNT 2 SHOULD BE 1 (ADJUSTED)
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: LINK COUNT FILE I=70688  OWNER=operator MODE=100400
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: SIZE=2048 MTIME=Apr  6 04:22 2012  COUNT 2 SHOULD BE 1 (ADJUSTED)
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: LINK COUNT FILE I=70689  OWNER=operator MODE=100400
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: SIZE=2048 MTIME=Apr  6 03:55 2012  COUNT 2 SHOULD BE 1 (ADJUSTED)
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: LINK COUNT FILE I=70692  OWNER=operator MODE=100400
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: SIZE=2048 MTIME=Apr  6 04:11 2012  COUNT 2 SHOULD BE 1 (ADJUSTED)
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: LINK COUNT FILE I=70694  OWNER=operator MODE=100400
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: SIZE=2048 MTIME=Apr  6 04:33 2012  COUNT 2 SHOULD BE 1 (ADJUSTED)
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: LINK COUNT FILE I=70705  OWNER=operator MODE=100400
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: SIZE=2048 MTIME=Apr  6 03:44 2012  COUNT 2 SHOULD BE 1 (ADJUSTED)
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: Reclaimed: 0 directories, 1 files, 1 fragments
Apr  6 04:53:05 DOMAIN fsck: /dev/da0s1d: 25334 files, 126338 used, 623684 free (7228 frags, 77057 blocks, 1.0% fragmentation)

***dmesg***
```


----------



## gkontos (Apr 6, 2012)

I don't think it is a memory related issue. It looks more like a filesystem corruption to me.
I would suggest a full backup of your data, sites and dbs, and then a *fsck* from single user mode. 

Also, try fixing sendmail in your database jail by either disabling it or making the proper aliases.
I don't have much experience with IPFW syntax but I would find a way to keep only 1 rule there also.


----------



## ghostcorps (Apr 6, 2012)

blakjak said:
			
		

> you need to have a SWAP partition during your installation of the freeBSD OS.This
> SWAP partition is used when your computer is running out of RAM space. I hope you have a swap partition?



Thanks Blakjak, yes, I do  

`#df -h` 

```
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    496M    331M    125M    72%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/da0s1e    496M    290K    456M     0%    /tmp
/dev/da0s1f     16G     12G    2.6G    82%    /usr
/dev/da0s1d    1.4G    247M    1.1G    18%    /var
devfs          1.0K    1.0K      0B   100%    /usr/gaols/webserver/dev
procfs         4.0K    4.0K      0B   100%    /usr/gaols/webserver/proc
devfs          1.0K    1.0K      0B   100%    /usr/gaols/database/dev
procfs         4.0K    4.0K      0B   100%    /usr/gaols/database/proc
```

`#pstat -T`

```
324/12072 files
0M/987M swap space
```


gkontos:

 It looks like one of the guys at the host has run an *fsck* on it already, but I will check with them and give it a go otherwise. We have gone a night without a crash but who knows what the new day will bring.

 I don't need sendmail on the db so I'll turn that off too. Thanks for pointing that out, I didn't even realise it was on.

 Not sure what you mean about keeping 1 rule in ipfw. The rules above are only a small handful of the hundred or so rules I use to keep the ports blocked. If I remove any of them it will expose me.



Thanks again for you help


----------



## gkontos (Apr 6, 2012)

ghostcorps said:
			
		

> Not sure what you mean about keeping 1 rule in ipfw. The rules above are only a small handful of the hundred or so rules I use to keep the ports blocked. If I remove any of them it will expose me.



When dealing with firewall rules, you try to write them in such way that they don't bring extra burden in to the filtering engine. 
Like I said before, I have absolutely no idea how script IPFW rules. But you can use this as a general rule of thumb:

1) Have your most frequent rules processed first.
2) Explicitly deny all other ports using a more general statement.

Pseudocode example:


```
permit any to <webeserver> <webservice_tcp_ports>
deny any  any
```

This pretty much works with any type of firewall.


----------



## ghostcorps (Apr 7, 2012)

gkontos said:
			
		

> When dealing with firewall rules, you try to write them in such way that they don't bring extra burden in to the filtering engine.
> 
> 1) Have your most frequent rules processed first.
> 2) Explicitly deny all other ports using a more general statement.



Thanks,

 I have made the rule list loosely adhering to that idea. I will see what I can do to optimise it.


----------



## olav (Apr 7, 2012)

I had the same problem with FreeBSD 8.2-RELEASE, an upgrade to FreeBSD 8.2-STABLE solved it. The STABLE branch is a good branch, and can be used on production systems.


----------



## ghostcorps (Apr 7, 2012)

Thanks olay,

 I should have mentioned that this box is running FreeBSD 8.1-RELEASE-p2. There are a few patches outstanding because I have modified the kernel and rolling it back will take the site offline for a day and we have just gotten some articles out in the news so we don't want to take it down just yet.

So far it hasn't crashed again though. * fingers crossed*


----------



## debguy (Apr 12, 2012)

sshd is 3mb? *T*alk about using ash not sh to reserve memory and ssh blows through it 

112M is wrong. *A*pache would use 4MB for a process having a small web page open.

(1) I would check httpd config files to see what setting would *allow* apache to cache that much data: apache is designed not to break memory limits by any kind of web hits.

(2) *L*look at your web content. *I*s apache loading a corrupt webpage that is in fact 100M to load?

Please say if the *top* you show is a httpd process waiting for a web hit or already having loaded the home page.  (i.e., in the setting it may load 5 waiting - which get recycled)

Use *netstat -a* to see what's LISTENING v. CONNECTED.

BTW is that multi-processor / threaded apache processes or regular ones?  *T*he mp version *I* think may be a litt*le* wild on memory it may say so in the docs. *U*se the right apache2 install pkg.

*B*y what is allowed *I* mean "apache mods" that your config say apache should / can load - there are so many *I* don't know if you_'re_ loading all of perl, python, php, and all else and the kitchen sink per process for no reason.

*I*f you are migrating, you might not run the new apache with a*n* old website - maybe use the apache the website had been working fine with.


----------



## ghostcorps (Apr 13, 2012)

Thanks Debguy.

Did you mean bash instead of sh?

 I will look into everything you have mentioned, there looks to be some fine-tuning to be done. I should say that top was run on the host which holds two jailed servers. Both the host and one of the virtual servers hosts an Apache installation. The webserver is a video streaming server. I would expect that Apache would run pretty heavy in this situation but I will still see what I can so about lightening the load.

 I am not sure which modules are safe to disable and which are not, whenever I try to thin them out I always end up breaking something that is not obvious.

 It looks to be the multiprocess version, which is the version portmaster chose to install.

/usr/ports/www/apache22/Makefile

```
PORTNAME=       apache
PORTVERSION=    2.2.22
PORTREVISION=   5
CATEGORIES=     www
MASTER_SITES=   ${MASTER_SITE_APACHE_HTTPD}
DISTNAME=       httpd-${PORTVERSION}
DIST_SUBDIR=    apache22

MAINTAINER?=    apache@FreeBSD.org
COMMENT?=       Version 2.2.x of Apache web server with ${WITH_MPM:L} MPM.
```

`netstat -a`

```
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address       (state)
tcp4       0    104 XXX.ssh              ME.35008       ESTABLISHED
tcp4       0      0 *.*                    *.*                    CLOSED
tcp46      0      0 *.http                 *.*                    LISTEN
tcp4       0      0 *.https                *.*                    LISTEN
tcp4       0      0 *.http                 *.*                    LISTEN
tcp4       0      0 *.8080                 *.*                    LISTEN
tcp4       0      0 SITENAME.com..smtp  *.*                    LISTEN
tcp4       0      0 *.ftp                  *.*                    LISTEN
tcp4       0      0 *.submission           *.*                    LISTEN
tcp6       0      0 *.smtp                 *.*                    LISTEN
tcp4       0      0 *.smtp                 *.*                    LISTEN
tcp4       0      0 XXX.ssh              *.*                    LISTEN
tcp4       0      0 XXX.ssh              *.*                    LISTEN
```


 Thankfully we have not had any trouble since posting this thread, but that doesn't mean it can not happen again.


----------



## User23 (Apr 13, 2012)

If php is used as apache module, 112MB per process is nothing special. Running low on RAM could happen if too many processes are running at the same time. Monitor your services and count of processes and you may find the problem easily.

I had similiar problems with a wordpress + statistics plugin. The plugin stored the statistics in a mysql db so slow, that the whole server could process only 2 request per second ... so the number of running processes raised sometimes to 200 or more and the server began to swap.

Use apachebench (ab) for a stress test.


----------



## ghostcorps (Apr 13, 2012)

I am told we are using a statistics plugin on wordpress. But I ran ab and it held up fine I think.

`ab -n 1000 -c 5 [url]https://URL.com/[/url]`

```
Server Software:        Apache
Server Hostname:        URL.com.au
Server Port:            443
SSL/TLS Protocol:       TLSv1/SSLv3,DHE-RSA-AES256-SHA,2048,256

Document Path:          /
Document Length:        7756 bytes

Concurrency Level:      5
Time taken for tests:   848.079 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      8188432 bytes
HTML transferred:       7756000 bytes
Requests per second:    1.18 [#/sec] (mean)
Time per request:       4240.396 [ms] (mean)
Time per request:       848.079 [ms] (mean, across all concurrent requests)
Transfer rate:          9.43 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:     1194 1278 235.6   1203    3841
Processing:   803 2950 1326.2   2775   14652
Waiting:      802 2593 1299.9   2402   14282
Total:       2004 4227 1337.0   4042   16455

Percentage of the requests served within a certain time (ms)
  50%   4042
  66%   4337
  75%   4558
  80%   4722
  90%   5376
  95%   6395
  98%   7668
  99%  10180
 100%  16455 (longest request)
```

nb. That I ran ab from Australia and the server is in the US.

Looking at the info.php it says I am using mod_php5, which if I have googled correctly means that I am not using PHP as a module, is this right?


----------



## User23 (Apr 13, 2012)

The main problem is not PHP but the number of processes, running at the same time, due to the slow statistic plugin. 
Make a stress test with and without the wordpress statistics plugin enabled to verify the problem.

PHP as apache module should be the fastest option, so stay with it.


----------



## User23 (Apr 13, 2012)

ghostcorps said:
			
		

> I am told we are using a statistics plugin on wordpress.
> 
> Looking at the info.php it says I am using mod_php5, which if I have googled correctly means that I am not using PHP as a module, is this right?



"mod_php5" is the PHP5 apache module. So, everything is ok.


----------



## ghostcorps (Apr 13, 2012)

User23 said:
			
		

> The main problem is not PHP but the number of processes, running at the same time, due to the slow statistic plugin.
> Make a stress test with and without the wordpress statistics plugin enabled to verify the problem.
> 
> PHP as apache module should be the fastest option, so stay with it.



 I have been running a 50000 pass test over night, it is almost done. I'll try without the plugin when it finishes however during the  test I see no more than 14 threads at any time. Is this good or bad?


----------



## ghostcorps (Apr 14, 2012)

Could it have something to do with the time being out of sync by a day between the webserver and the database? I found my servers were not using the same timezone for some reason. But I have corrected it now.

Will see if this has any affect on the number of threads.


----------



## ghostcorps (Apr 15, 2012)

Hello again,

 By taking some time to research the mods and testing each one one at a time, I have cut the httpd threads down to about 95-120mb. Disabling the stats plugin WassUp did not reduce the thread size noticeably.

 This is an improvement of about 20mb and all services I can think of are working but I get the feeling I could go further. Would you mind having a look at my mod list below and letting me know if I have blocked anything subtly crucial? Or If there is anything more I could block 

/usr/local/etc/apache22/httpd.conf

```
LoadModule authn_file_module libexec/apache22/mod_authn_file.so
#LoadModule authn_dbm_module libexec/apache22/mod_authn_dbm.so
#LoadModule authn_anon_module libexec/apache22/mod_authn_anon.so
#LoadModule authn_default_module libexec/apache22/mod_authn_default.so
#LoadModule authn_alias_module libexec/apache22/mod_authn_alias.so
LoadModule authz_host_module libexec/apache22/mod_authz_host.so
LoadModule authz_groupfile_module libexec/apache22/mod_authz_groupfile.so
LoadModule authz_user_module libexec/apache22/mod_authz_user.so
#LoadModule authz_dbm_module libexec/apache22/mod_authz_dbm.so
#LoadModule authz_owner_module libexec/apache22/mod_authz_owner.so
#LoadModule authz_default_module libexec/apache22/mod_authz_default.so
LoadModule auth_basic_module libexec/apache22/mod_auth_basic.so
#LoadModule auth_digest_module libexec/apache22/mod_auth_digest.so
#LoadModule file_cache_module libexec/apache22/mod_file_cache.so
#LoadModule cache_module libexec/apache22/mod_cache.so
#LoadModule disk_cache_module libexec/apache22/mod_disk_cache.so
#LoadModule dumpio_module libexec/apache22/mod_dumpio.so
LoadModule reqtimeout_module libexec/apache22/mod_reqtimeout.so
LoadModule include_module libexec/apache22/mod_include.so
#LoadModule filter_module libexec/apache22/mod_filter.so
#LoadModule charset_lite_module libexec/apache22/mod_charset_lite.so
LoadModule deflate_module libexec/apache22/mod_deflate.so
LoadModule log_config_module libexec/apache22/mod_log_config.so
#LoadModule log_forensic_module libexec/apache22/mod_log_forensic.so
#LoadModule logio_module libexec/apache22/mod_logio.so
LoadModule env_module libexec/apache22/mod_env.so
#LoadModule mime_magic_module libexec/apache22/mod_mime_magic.so
#LoadModule cern_meta_module libexec/apache22/mod_cern_meta.so
LoadModule expires_module libexec/apache22/mod_expires.so
LoadModule headers_module libexec/apache22/mod_headers.so
LoadModule usertrack_module libexec/apache22/mod_usertrack.so
LoadModule unique_id_module libexec/apache22/mod_unique_id.so
LoadModule setenvif_module libexec/apache22/mod_setenvif.so
#LoadModule version_module libexec/apache22/mod_version.so
LoadModule ssl_module libexec/apache22/mod_ssl.so
LoadModule mime_module libexec/apache22/mod_mime.so
#LoadModule dav_module libexec/apache22/mod_dav.so
#LoadModule status_module libexec/apache22/mod_status.so
#LoadModule autoindex_module libexec/apache22/mod_autoindex.so
#LoadModule asis_module libexec/apache22/mod_asis.so
#LoadModule info_module libexec/apache22/mod_info.so
LoadModule cgi_module libexec/apache22/mod_cgi.so
#LoadModule dav_fs_module libexec/apache22/mod_dav_fs.so
LoadModule vhost_alias_module libexec/apache22/mod_vhost_alias.so
#LoadModule negotiation_module libexec/apache22/mod_negotiation.so
LoadModule dir_module libexec/apache22/mod_dir.so
#LoadModule imagemap_module libexec/apache22/mod_imagemap.so
LoadModule actions_module libexec/apache22/mod_actions.so
LoadModule speling_module libexec/apache22/mod_speling.so
#LoadModule userdir_module libexec/apache22/mod_userdir.so
LoadModule alias_module libexec/apache22/mod_alias.so
LoadModule rewrite_module libexec/apache22/mod_rewrite.so
LoadModule unique_id_module libexec/apache22/mod_unique_id.so
LoadModule security2_module libexec/apache22/mod_security2.so
LoadModule php5_module        libexec/apache22/libphp5.so
```


----------



## ghostcorps (Apr 15, 2012)

Just when I thought I was ready to mark this as solved... It crashed again!!



I have worked through all the suggestions and still crashing!

I am lost now ...


----------



## User23 (Apr 16, 2012)

ghostcorps said:
			
		

> I have been running a 50000 pass test over night, it is almost done. I'll try without the plugin when it finishes however during the  test I see no more than 14 threads at any time. Is this good or bad?



Depends on how many simultaneous queries you used to test and on the server hardware. 

Try

```
ab -c 20 -n 1000 http://yourdomain.tld
```
for example.


----------



## ghostcorps (Apr 16, 2012)

Thanks, 



			
				User23 said:
			
		

> Try
> 
> ```
> ab -c 20 -n 1000 http://yourdomain.tld
> ...



 I ran the test but it timed out after 4 completed requests.


```
Benchmarking URL.com (be patient)
apr_poll: The timeout specified has expired (70007)
Total of 4 requests completed
```

I couldn't browse to the site and my ssh session crashed out too, but I was able to log in with some patience. I found alot of threads were still open, I guess they were the threads created by ab had not closed..

After restarting apache the website came back up and access is normal again. Is it the RAM or is more likely that the interface between the site and SQL is too slow? Apache runs on one jailed server and the database is on another.


----------



## ghostcorps (Apr 17, 2012)

It is strange, if I run a 20 thread test with a concurrency of 20, it pulls through and the threads clear. But when I run a 40 thread test with the same concurrency the test times out and the threads lock up.

I have added this to /usr/local/etc/apache22/httpd.conf

```
RequestReadTimeout header=1-3,MinRate=500
```

But it has not had any noticeable effect.


----------



## ghostcorps (Apr 17, 2012)

By turning off the KeepAlive entry in the config I no longer lock up the server when ab times out. Which is excellent news.

But I still need to work out how to stop the server grinding to a halt when I run 20 consecutive threads for example:

`ab -c 20 -n 20 [url]https://some.site.cd/[/url]`


----------



## User23 (Apr 17, 2012)

ghostcorps said:
			
		

> Is it the RAM or is more likely that the interface between the site and SQL is too slow? Apache runs on one jailed server and the database is on another.



You could run the ab test on a static html page. This will show how the apache perform without mysql.

As *I* said, *I* guess it is the Wordpress statistic plugin. Keep an eye on the mysql slow queries log and use 
	
	



```
show full processlist;
```
 on the mysql console, while stress testing. If the statistics inserts are the bottleneck it should be easy to identify them in the processlist.


----------



## ghostcorps (Apr 17, 2012)

User23 said:
			
		

> You could run the ab test on a static html page. This will show how the apache perform without mysql.
> 
> As i said, i guess it is the wordpress statistic plugin. Keep an eye on the mysql slow queries log and use
> 
> ...



Thanks 

  I had tried without the statistics plugin (it is called Wassup) and the result was the same. 

 Yet, when I run the same test it on a static webpage with a fairly large flash element, it rips though it so fast that I don't even see new threads opening in top.

 Though it seems it is related to the database I can't immediatly tell much from the results of your query. I am looking into mgm_scheduling which makes a large part of the log:

`show full processlist;` 
^click for results in pastebin.^

Is it worth looking into moving the database to MyISAM?


----------



## ghostcorps (Apr 18, 2012)

Sir Dice suggested I disable caching on the database filesystem on an previous thread but he was unaware of how to do it on FreeBSD.

 I have just found the vfs.vmiodirenable variable for sysctl which I think is what I am looking for and if I am reading the manual correctly I should add this:

/etc/sysctl.conf

```
vfs.vmiodirenable=0
```

 Does this sound reasonable? I doubt it will break anything if I just give it a go but on the off chance that it corrupts everything, I wanted to ask first.


----------



## User23 (Apr 18, 2012)

ghostcorps said:
			
		

> I had tried without the statistics plugin (it is called Wassup) and the result was the same.
> 
> Yet, when I run the same test it on a static webpage with a fairly large flash element, it rips though it so fast that I don't even see new threads opening in top.



Ok, good.



			
				ghostcorps said:
			
		

> Though it seems it is related to the database I can't immediatly tell much from the results of your query. I am looking into mgm_scheduling which makes a large part of the log:




```
"...  WHERE `option_name` = 'cron' "
"...  WHERE `option_name` = '_transient_doing_cron' "
```

The queries are all from the (stupid?) wordpress cron solution. This script runs every time a page is loaded. So it can burn CPU time if there are like 20 or more request per second, pretty useless. But it shouldnt result in such a bad performance. How many request per second do you got now, using 
	
	



```
ab -c20 -n1000
```
 ?



			
				ghostcorps said:
			
		

> Is it worth looking into moving the database to MyISAM?



Well, I think this should not be the first step to do now.

Please post your 
	
	



```
show variables;
```
 and 
	
	



```
show status;
```
 output from mysql console. Remove hostnames and paths if you like.


----------



## ghostcorps (Apr 18, 2012)

Thanks for your patience! 

Disabling cache with the below config did not help:

/etc/sysctl.conf

```
vfs.vmiodirenable=0
```


20 concurrent threads is enough to lock up apache so I ran this:
`ab -c10 -n100`

And got the results below:

```
HTML transferred:       703494 bytes
Requests per second:    1.30 [#/sec] (mean)
Time per request:       7686.952 [ms] (mean)
Time per request:       768.695 [ms] (mean, across all concurrent requests)
Transfer rate:          9.49 [Kbytes/sec] received
```


If I run the same command on the static page I get his:

```
HTML transferred:       219700 bytes
Requests per second:    6.67 [#/sec] (mean)
Time per request:       1499.014 [ms] (mean)
Time per request:       149.901 [ms] (mean, across all concurrent requests)
Transfer rate:          16.13 [Kbytes/sec] received
```


`[url=http://pastebin.com/X26e0dpi]show variables;[/url]` 
^Pastebin^

`[url=http://pastebin.com/df4rps7U]show status;[/url]` 
^Pastebin^


----------



## aa (Apr 18, 2012)

try nginx + php-fpm


----------



## ghostcorps (Apr 18, 2012)

aa said:
			
		

> try nginx + php-fpm



Thanks 

 I have never heard of these, what is the advantage of ngnix over apache?

 As far as I can tell, apache is working fine as long as it doesn't have to query the database. I will look at php-fpm + ngnix later, but for now I don't want to introduce any potential new issues.


----------



## aa (Apr 18, 2012)

> I will look at php-fpm + ngnix later, but for now I don't want to introduce any potential new issues.


You should try, soon, it's really worth it.


----------



## einthusan (Apr 18, 2012)

Yes, you*'re* better off fixing new issues related to nginx th*a*n fixing old issues related to apache. The fact you said you don't even know nginx is scary! You*'re* really behind man.


----------



## ghostcorps (Apr 18, 2012)

Lol  Ok I give in I'll do it. 

But I really want to fix the current problem first. All evidence points away from the webserver actually causing the lockup and fixing that is more important than getting with the times 

I'll have a play with it on my test server and drop it in once the database has been fixed.


----------



## aa (Apr 18, 2012)

einthusan said:
			
		

> Yes, you*'re* better off fixing new issues related to nginx th*a*n fixing old issues related to apache. The fact you said you don't even know nginx is scary! You*'re* really behind man.


Be patient, please. I haven't even read your problem when posting here.


----------



## ghostcorps (Apr 19, 2012)

First thing I notice about nginx, is that it is optimised for serving static pages. We do not use static pages and if we did we currently have no issue with doing so using apache.

nginx looks to be sold on the idea of doing a few basic things as fast as possible, which is great for static pages and ultra light websites. But our site is dynamic and content heavy. On a day to day basis we do not expect to have a huge amount of traffic and certainly do not expect to be serving 1000s of pages at a time. It is nice to have the overhead availability but I use a few of the advanced apache functions that nginx does not support, one specificly is 'digest access authentication' which we use for our secured streaming sessions. Without that I will not be able to get the webmaster to sign off on the Change.

 I have also read that there is no point looking at php-fpm without nginx because mod_php serves the same function within apache.

 Our webmaster is doing his own investigation, but I doubt we will be doing nginx or php-fpm until it supports the more complex tasks we need from a webserver.



 Back to the real task at hand...


----------



## User23 (Apr 19, 2012)

ghostcorps said:
			
		

> 20 concurrent threads is enough to lock up apache so I ran this:
> `ab -c10 -n100`
> 
> And got the results below:
> ...



6.6 Requests per second, on a static page ... horrible. What kind of hardware is that server (CPU type, RAM size ...)? What is the usual load on this machine? It looks like your problem is the apache config and/or your load/hardware.

For example, *I* got 10 request per second testing a Wordpress website, using -c20 -n1000 on a slow 4 core Opteron 4130 (2.6 GHz). The server usually has a load around 1.0 without ab testing.


----------



## ghostcorps (Apr 19, 2012)

Hmmm, ok, I am not surprised if it ends up being a config issue on my end, as you can tell I an groping in the dark with a lot of this stuff.

The following are taken from the Host server not the jailed webserver. 

`# dmesg | grep CPU`

```
CPU: Intel(R) Xeon(R) CPU           X5680  @ 3.33GHz (3324.51-MHz K8-class CPU)
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
```

`# dmesg | grep memory`

```
real memory  = 805306368 (768 MB)
avail memory = 759275520 (724 MB)
```


`# swapinfo -k`

```
Device          1K-blocks     Used    Avail Capacity
/dev/da0s1b       1011592    64864   946728     6%
```


`#top` (at idle)

```
last pid: 38797;  load averages:  0.03,  0.04,  0.01                                                                                                                         up 3+19:10:58  23:21:25
98 processes:  1 running, 97 sleeping
CPU:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
Mem: 64M Active, 114M Inact, 143M Wired, 1136K Cache, 85M Buf, 408M Free
Swap: 988M Total, 63M Used, 925M Free, 6% Inuse
```

`# sysctl -a | less`
^Pastebin^

---------

 Below are my apache config files on the jailed webserver:

/usr/local/etc/apache22/httpd.conf

```
ServerRoot "/usr/local"

Listen 80

LoadModule authn_file_module libexec/apache22/mod_authn_file.so
LoadModule authz_host_module libexec/apache22/mod_authz_host.so
LoadModule authz_groupfile_module libexec/apache22/mod_authz_groupfile.so
LoadModule authz_user_module libexec/apache22/mod_authz_user.so
LoadModule auth_basic_module libexec/apache22/mod_auth_basic.so
LoadModule reqtimeout_module libexec/apache22/mod_reqtimeout.so
LoadModule include_module libexec/apache22/mod_include.so
LoadModule deflate_module libexec/apache22/mod_deflate.so
LoadModule log_config_module libexec/apache22/mod_log_config.so
LoadModule env_module libexec/apache22/mod_env.so
LoadModule expires_module libexec/apache22/mod_expires.so
LoadModule headers_module libexec/apache22/mod_headers.so
LoadModule usertrack_module libexec/apache22/mod_usertrack.so
LoadModule unique_id_module libexec/apache22/mod_unique_id.so
LoadModule setenvif_module libexec/apache22/mod_setenvif.so
LoadModule ssl_module libexec/apache22/mod_ssl.so
LoadModule mime_module libexec/apache22/mod_mime.so
LoadModule cgi_module libexec/apache22/mod_cgi.so
LoadModule vhost_alias_module libexec/apache22/mod_vhost_alias.so
LoadModule dir_module libexec/apache22/mod_dir.so
LoadModule actions_module libexec/apache22/mod_actions.so
LoadModule speling_module libexec/apache22/mod_speling.so
LoadModule alias_module libexec/apache22/mod_alias.so
LoadModule rewrite_module libexec/apache22/mod_rewrite.so
LoadModule unique_id_module libexec/apache22/mod_unique_id.so
LoadModule security2_module libexec/apache22/mod_security2.so
LoadModule php5_module        libexec/apache22/libphp5.so

AddType application/x-httpd-php .php .phtml
AddType application/x-httpd-php-source .phps

ServerAdmin ME@gmail.com

ServerName www.URL.com:80

DocumentRoot "/usr/local/www/apache22/data"

ServerSignature Off
ServerTokens Prod

<Directory />
    AllowOverride All
    Order deny,allow
    Deny from all
</Directory>

Alias /errors/ "/usr/local/www/apache22/errors/"

<Directory "/usr/local/www/apache22/errors/">
	Order Deny,Allow
	Allow from 192.168.0.0/24
	Deny from all
</Directory>

<Directory "/usr/local/www/apache22/offline/">
	Order deny,allow
	Deny from all
    Allow from 192.168.0.0/24
	AllowOverride All
</Directory>

<Directory "/usr/local/www/apache22/data">
    Options Indexes +FollowSymLinks MultiViews
    AllowOverride All
    Order allow,deny
    Allow from all
</Directory>

<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>

ErrorLog "/var/log/httpd-error.log"

LogLevel error

<IfModule log_config_module>

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    CustomLog "/var/log/httpd-access.log" combined
</IfModule>

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/usr/local/www/apache22/cgi-bin/"
</IfModule>


<Directory "/usr/local/www/apache22/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

DefaultType text/plain

<IfModule mime_module>
    TypesConfig etc/apache22/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
</IfModule>

ErrorDocument 401 /errors/autherror.html

Include etc/apache22/extra/httpd-mpm.conf

Include etc/apache22/extra/httpd-vhosts.conf

Include etc/apache22/extra/httpd-default.conf

Include etc/apache22/extra/httpd-ssl.conf

<IfModule ssl_module>
	SSLRandomSeed startup builtin
	SSLRandomSeed connect builtin
</IfModule>

Include etc/apache22/Includes/*.conf
TraceEnable off

RequestReadTimeout header=1-3,MinRate=500
```


/usr/local/etc/apache22/extras/httpd-ssl.conf

```
Listen 443

AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

SSLPassPhraseDialog  builtin

SSLSessionCache        "shmcb:/var/run/ssl_scache(512000)"
SSLSessionCacheTimeout  300

SSLMutex  "file:/var/run/ssl_mutex"

<VirtualHost _default_:443>

DocumentRoot "/usr/local/www/apache22/data"
ServerName www.URL.com
ServerAdmin ME@gmail.com
ErrorLog "/var/log/httpd-error.log"
TransferLog "/var/log/httpd-access.log"

SSLEngine on
SSLProtocol -all +SSLv3 +TLSv1

SSLCipherSuite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM

SSLCertificateFile "/usr/local/etc/apache22/ssl/certificate.crt"
SSLCertificateKeyFile "/usr/local/etc/apache22/ssl/server.key"
SSLCACertificateFile "/usr/local/etc/apache22/ssl/RapidSSL_CA_bundle.pem"

<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>
<Directory "/usr/local/www/apache22/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

CustomLog "/var/log/httpd-ssl_request.log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>
```


/usr/local/etc/apache22/extras/httpd-vhosts.conf

```
NameVirtualHost HOST.EXTERNAL.IP:80

<VirtualHost WEBSERVER.JAIL.EXTERNAL.IP:80>
    DocumentRoot "/usr/local/www/apache22/data"
    ServerName URL.com
    ServerAlias URL.com
</VirtualHost>

<VirtualHost WEBSERVER.JAIL.EXTERNAL.IP:80>
    ServerAdmin webmaster@URL.com
    DocumentRoot "/usr/local/www/apache22/offline"
    ServerName offline.URL.com
    ServerAlias offline.URL.com
</VirtualHost>

<VirtualHost WEBSERVER.JAIL.VPN.IP:80>
    DocumentRoot "/usr/local/www/apache22/test"
    ServerName test.URL.com
    ServerAlias test.URL.com
</VirtualHost>
```


/usr/local/etc/apache22/extras/httpd-default.conf

```
Timeout 10

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 1
UseCanonicalName Off
AccessFileName .htaccess
ServerTokens Prod
ServerSignature Off
HostnameLookups Off
```

/usr/local/etc/apache22/extra/httpd-mpm.conf

```
<IfModule mpm_prefork_module>
StartServers 3
MinSpareServers 2
MaxSpareServers 5
ServerLimit 10
MaxClients 10
MaxRequestsPerChild 100
</IfModule>

<IfModule mpm_worker_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>
```

And finally the jailed webservers /usr/local/etc/my.cnf

```
[client]
host		= JAILED.DATABASE.VPN.IP
port            = 3306

[mysqld]
port		= 3306
host		= JAILED.DATABASE.VPN.IP
skip-external-locking
key_buffer = 16K
max_allowed_packet = 1M
table_cache = 4
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 64K

server-id	= 1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 8M
sort_buffer_size = 8M

[myisamchk]
key_buffer = 8M
sort_buffer_size = 8M

[mysqlhotcopy]
interactive-timeout
```


----------



## ghostcorps (Apr 19, 2012)

I have been using the performance tuning tool HERE to tune the MySQL install. So far I have increased the table cache to 512 but I need to wait a couple days to get a good report on some of the other settings.


----------



## einthusan (Apr 20, 2012)

ghostcorps said:
			
		

> First thing I notice about nginx, is that it is optimised for serving static pages. We do not use static pages and if we did we currently have no issue with doing so using apache.
> 
> nginx looks to be sold on the idea of doing a few basic things as fast as possible, which is great for static pages and ultra light websites. But our site is dynamic and content heavy. On a day to day basis we do not expect to have a huge amount of traffic and certainly do not expect to be serving 1000s of pages at a time. It is nice to have the overhead availability but I use a few of the advanced apache functions that nginx does not support, one specificly is 'digest access authentication' which we use for our secured streaming sessions. Without that I will not be able to get the webmaster to sign off on the Change.
> 
> ...



Don't be so misinformed.
http://wiki.nginx.org/HttpAuthDigestModule

These isn't anything that you*'re* doing which can't be supported by nginx. A few years back, nginx WAS as you described, but that's a long time ago.

This link should be VERY helpful for you.
A faster Web server: ripping out Apache for Nginx

It was built for static content but that doesn't mean it is slow or doesn't work with dynamic pages. There are many content-rich sites using nginx. Ultimately, APACHE eats your memory like hell.

In fact I run a video site using nginx and serves dynamic php pages. We have tons of content.
http://www.einthusan.com

Anyways, sorry for going off topic.


----------



## ghostcorps (Apr 20, 2012)

Thanks for clarifying, I guess I was reading old docs.

 I spoke to our webmaster and it turns out we don't use the digest access auth any more anyway. I'm going to do some testing with nginx over the next few days.

Thanks again


----------



## ghostcorps (Apr 20, 2012)

First benchmark on nginx, just the same static page I used for apache:



```
Requests per second:    2130.88 [#/sec] (mean)
```

This is starting to look very worthwhile


----------



## einthusan (Apr 21, 2012)

ghostcorps said:
			
		

> First benchmark on nginx, just the same static page I used for apache:
> 
> 
> 
> ...



I'm glad to see that you're testing nginx now and seeing good results. As you tune it further, along with FreeBSD tuning using sysctl.conf and loader.conf, you will see further improvements.

There are many modules (add-ons) for nginx that will help you accomplish your goals.

Have fun!


----------



## ghostcorps (Apr 21, 2012)

einthusan said:
			
		

> I'm glad to see that you're testing nginx now and seeing good results. As you tune it further, along with FreeBSD tuning using sysctl.conf and loader.conf, you will see further improvements.
> 
> There are many modules (add-ons) for nginx that will help you accomplish your goals.
> 
> Have fun!




Yeh, so far so good. 

I would still like to know why apache was being such a dog, but oh well it will be behind me soon enough


----------



## ghostcorps (Apr 23, 2012)

Hi guys,

nginx is killing me  I can get html pages to load, but php files return 'no input files'.

 I have been through a tonne of tutes and confirmed the path to the webroot and to fastcgi_params are correct.

 I confirmed php-fpm is listening on port 9000 with `sockstat | grep 9000`

 Would anyone mind posting their /usr/local/etc/nginx/nginx.conf and /usr/local/etc/php-fpm.conf?

This is what I have in my gninx.conf so far:

/usr/local/etc/nginx/nginx.conf

```
worker_processes  1;

error_log                       /var/log/nginx-error.log;

events  {
        worker_connections      1024;
        }

http    {
        include                 mime.types;
        default_type            application/octet-stream;
        log_format      main    '$remote_addr - $remote_user [$time_local] "$request" '
                                '$status $body_bytes_sent "$http_referer" '
                                '"$http_user_agent" "$http_x_forwarded_for"';

        access_log              /var/log/nginx-access.log  main;

        sendfile                on;
        keepalive_timeout       65;
        gzip                    on;

        server {
                listen                  192.168.0.2:90;
                server_name             URL.com;

                location /
                        {
                        root            /usr/local/www/nginx;
                        index           index.php index.html index.htm;
                        }
                error_page              500 502 503 504  /50x.html;

                location = /50x.html
                        {
                        root            /usr/local/www/nginx-dist;
                        }

                location ~ \.php
                        {
                        fastcgi_pass    127.0.0.1:9000;
                        fastcgi_index   index.php;
                        fastcgi_param   SCRIPT_FILENAME         /scripts$fastcgi_script_name;
                        include         /usr/local/etc/nginx/fastcgi_params;
                        }

                location ~ /\.ht
                        {
                        deny            all;
                        }
                }
      }
```

/usr/local/etc/php-fpm.conf

```
[global]
pid = run/php-fpm.pid
error_log = /var/log/php-fpm.log
log_level = notice
daemonize = yes

[www]
user = www
group = www
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
```


Thanks


----------



## einthusan (Apr 24, 2012)

Why are you listening on port 90? If its a website, shouldn't you be listening on port 80?


----------



## einthusan (Apr 24, 2012)

Here is parts of my config, this is on a ubuntu box so the paths are different than yours. I have my config files separated but you could just include them all in one file.

nginx.conf

```
user www-data;
worker_processes 1;
pid /var/run/nginx.pid;

events {
  worker_connections 1024;
  # multi_accept on;
}

http {

  ##
  # Basic Settings
  ##
  sendfile off;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 2;
  types_hash_max_size 2048;
  charset utf-8;
 
  # server_tokens off;
  # server_names_hash_bucket_size 64;
  server_name_in_redirect off;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;
  index index.php index.htm index.html;

  ##
  # Gzip Settings
  ##
  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 9;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

  ##
  # If HTTPS, then set a variable so it can be passed along.
  ##
  #map $scheme $server_https {
  #  default off;
  #  https on;
  #}

  ##
  # Virtual Host Configs
  ##
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;

}
```

Notice the lines,  include /etc/nginx/conf.d/*.conf; and include /etc/nginx/sites-enabled/*;

/etc/nginx/sites-enabled/mysite.conf

```
server {
  listen   80;
  server_name  mysite.com;
  rewrite ^ http://www.mysite.com$request_uri? permanent;
}

server {
  listen   80;
  server_name www.mysite.com;

  access_log off;
  error_log /var/www/public_html/www.mysite.com/log/error.log error;

  root   /var/www/public_html/www.mysite.com/public;
  try_files $uri $uri/ /index.php?q=$uri&$args;

  error_page  401  /errorpages/index.php?error=401;
  error_page  403  /errorpages/index.php?error=403;
  error_page  404  /errorpages/index.php?error=404;
  error_page  500 502 503 504  /errorpages/index.php?error=500;

  # serve static files directly
  location ~* ^.+\.(js|css|jpg|jpeg|gif|png|ico|swf|eot|woff|ttf|svg)$ {
    expires 30d;
  }

  # Deny all attempts to access hidden files such as .htaccess, .htpasswd
  location ~ /\. {
    deny all;
    access_log off;
    log_not_found off;
  }

  location ~ \.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    include /etc/nginx/fastcgi_params;
    fastcgi_pass unix:/var/run/php5-fpm.sock;
  }
}
```


----------



## einthusan (Apr 24, 2012)

Your config seems to be incorrect I think. You have the following, 

```
listen                  192.168.0.2:90;
server_name       bsd.removed.tld;
```

From my understating, listen should only be the port number. server_name is actual domain name, not the name of your server. So if your website was http://www.mysite.com, then server_name should be http://www.mysite.com

Read this for more info,
http://wiki.nginx.org/Configuration


----------



## ghostcorps (Apr 24, 2012)

Argh  I forgot to sanitise the configs. einthusan, would you mind removing my domain from your post? [ done -- Mod.] Thanks 

I am using port 90 as it is running side by side with apache on 80. Once it is working on my test server I will move the configs over to the production box, but will keep port 90 until it is running exactly as I need. Then I will stop apache for good and move nginx to 80.

I will follow up on your suggestions when I get home from work later tonight.


----------



## ghostcorps (Apr 25, 2012)

It was a silly mistake.

I had the root dir set in the wrong section. I moved it from:

/usr/local/etc/nginx.conf

```
...
        server {
                listen                  192.168.0.2:90;
                server_name             URL.com;

                location /
                        {
                        root            /usr/local/www/nginx;
...
```

To

/usr/local/etc/nginx.conf

```
...
http    {
        root            /usr/local/www/nginx;
...
```


----------



## ghostcorps (Apr 25, 2012)

After installing nginx on the jailed webserver it does not look to run any faster than apache. 

I'll try to tweak it but, at least it does not crash when I run 20 threads. 


Apache test
`an -n 100 -c 10 MY.VPN.IP:80`

```
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.254.2 (be patient).....done


Server Software:        Apache
Server Hostname:        192.168.254.2
Server Port:            80

Document Path:          /
Document Length:        6332 bytes

Concurrency Level:      10
Time taken for tests:   62.338 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      676400 bytes
HTML transferred:       633200 bytes
Requests per second:    1.60 [#/sec] (mean)
Time per request:       6233.757 [ms] (mean)
Time per request:       623.376 [ms] (mean, across all concurrent requests)
Transfer rate:          10.60 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      277  284  27.7    280     549
Processing:  1621 5660 3657.7   4952   19707
Waiting:      867 4835 3643.7   4234   18866
Total:       1899 5944 3656.3   5230   19987

Percentage of the requests served within a certain time (ms)
  50%   5230
  66%   6324
  75%   7561
  80%   7907
  90%  10376
  95%  14547
  98%  19144
  99%  19987
 100%  19987 (longest request)
```

Nginx test
`an -n 100 -c 10 MY.VPN.IP:90`

```
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.254.2 (be patient).....done


Server Software:        nginx/1.0.14
Server Hostname:        192.168.254.2
Server Port:            90

Document Path:          /
Document Length:        6867 bytes

Concurrency Level:      10
Time taken for tests:   63.760 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      724300 bytes
HTML transferred:       686700 bytes
Requests per second:    1.57 [#/sec] (mean)
Time per request:       6375.965 [ms] (mean)
Time per request:       637.596 [ms] (mean, across all concurrent requests)
Transfer rate:          11.09 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      274  281   4.9    280     314
Processing:  2383 5659 1188.1   5664    9638
Waiting:     2012 5362 1146.0   5303    9172
Total:       2662 5939 1187.3   5943    9918

Percentage of the requests served within a certain time (ms)
  50%   5943
  66%   6411
  75%   6653
  80%   6753
  90%   7585
  95%   8034
  98%   8296
  99%   9918
 100%   9918 (longest request)
```


----------



## aa (Apr 25, 2012)

ghostcorps said:
			
		

> ... at least it does not crash when I run 20 threads.



Isn't that enough a reason to convert?


----------



## ghostcorps (Apr 25, 2012)

aa: It is enough reason and I am currently working through the migration. If all else fails nginx will still be a step towards more stability. But this result also suggests that the bottleneck brought to light by the crashes is not caused the webserver. If I could find what it is then I expect the site will run much faster and be more stable regardless of which webserver I use.

 Then I can do a real comparison of the two without them being shackled by whatever is causing the low thread per-second count.


----------



## einthusan (Apr 26, 2012)

ghostcorps said:
			
		

> aa: It is enough reason and I am currently working through the migration. If all else fails nginx will still be a step towards more stability. But this result also suggests that the bottleneck brought to light by the crashes is not caused the webserver. If I could find what it is then I expect the site will run much faster and be more stable regardless of which webserver I use.
> 
> Then I can do a real comparison of the two without them being shackled by whatever is causing the low thread per-second count.



Look at the time taken to complete 90% of the requests. The problem may be that the computer running ab can't handle the test load. Maybe you should try testing from a different client. Also, I doubt any setting on your computer is causing the super low req/s. have you tried testing with no concurrency? If the numbers were much higher than it's your client PC which is not able to do concurrent tests.


----------



## ghostcorps (Apr 26, 2012)

The tests above were performed from a very high end games PC on an ADSL+ connection. I am pretty sure it would have been able to handle the task.

If I run the tests from the server hosting the jailed webserver the results are 998 for apache vs 2231 tasks p/s. This made me suspect it may simply be because I am on the otherside of the world from the server I used another VPS I manage that is also in the US but the results were the same as above.

Even with the concurrency turned off the results are the same.

Could it be a networking issue between the webserver > jail host > public?


----------



## anon12b (Apr 26, 2012)

I know the issue has already been approached, but I had a client have a lot of problems with memory when using Wordpress. The easiest solution for that situation was to put a caching nginx in front of apache/Wordpress.

If you can be bothered with fiddling, Varnish is pretty amazing (and by phk <3). It does use quite a bit of memory by design, though. For that reason alone it is not my primary recommendation here.


----------



## einthusan (Apr 27, 2012)

Run a iperf test. Very simple to do. Install it on both client and server. This will test your network cards capability and maybe your network card is not working correctly? Just a guess. I have never used jails so I don't know but I don't think that should be a problem. I am in the progress of moving my web server from Ubuntu back to FreeBSD so I'll run some test to see how it looks on my machine.


----------



## ghostcorps (Apr 27, 2012)

anon12b said:
			
		

> I know the issue has already been approached, but I had a client have a lot of problems with memory when using Wordpress. The easiest solution for that situation was to put a caching nginx in front of apache/Wordpress
> 
> If you can be bothered with fiddling, Varnish is pretty amazing (and by phk <3). It does use quite a bit of memory by design, though. For that reason alone it is not my primary recommendation here.



Thanks anon, I have been running some of the ab tests against flat index.htmlfiles to avoid any influence from Wordpress, PHP or MySQL.

I'll try to keep the solution simple if I can 


einthusan: I'll play with iperf tonight and post some results later on.


----------



## ghostcorps (Apr 27, 2012)

Just finished the first set of iperf tests and thankfully the bottleneck is not the Jail 

Is this as slow/fast as you would expect for a connection from Aust to the US? I can't use the other US based server I manage to do a comparative test.



Listener on Jail Host in the US			
`# iperf -s -P 2 -i 5 -p 88 -f k` 
Server listening on TCP port 88			
TCP window size: 64.0 KByte (default)			


```
Interval/sec	Transfer KB	Bandwidth KB/s
0.0-5	        358		586
5.0-10.0	491		804
10.0-15.0	457		748
15.0-20.0	464		761
20.0-25.0	457		749
25.0-30.0	414		678
30.0-35.0	421		690
35.0-40.0	475		778
40.0-45.0	444		727
45.0-50.0	345		566
50.0-55.0	389		637
```

Client in Aust			
`# iperf -c xxx.xxx.xxx.xx2 -P 1 -i 5 -p 88 -f B -t 60 -T 1` 
Client connecting to xxx.xxx.xxx.xx2, TCP port 88			
TCP window size: 33396 Byte (default)			


```
Interval/sec	Transfer KB	Bandwidth KB/s
0.0-5	        393		78
5.0-10.0	393		78
10.0-15.0	524		104
15.0-20.0	524		104
20.0-25.0	393		78
25.0-30.0	524		104
30.0-35.0	393		78
35.0-40.0	524		104
40.0-45.0	393		78
45.0-50.0	393		78
50.0-55.0	393		78
```

Listener on Jailed Webserver in US			
`# iperf -s -P 2 -i 5 -p 88 -f k` 
Server listening on TCP port 88			
TCP window size: 64.0 KByte (default)			


```
Interval/sec	Transfer KB	Bandwidth KB/s
0.0-5	        317		519
5.0-10.0	417		683
10.0-15.0	487		797
15.0-20.0	478		784
20.0-25.0	453		742
25.0-30.0	462		757
30.0-35.0	461		755
35.0-40.0	461		755
40.0-45.0	461		755
45.0-50.0	462		757
0.0-55.0	461		755
```


Client in aust			
`# iperf -c xxx.xxx.xxx.xx3 -P 1 -i 5 -p 88 -f B -t 60 -T 1` 
Client connecting to xxx.xxx.xxx.xx3, TCP port 88			
TCP window size: 33396 Byte (default)			


```
Interval/sec	Transfer KB	Bandwidth KB/s
0.0-5	        262		52
5.0-10.0	524		104
10.0-15.0	393		78
15.0-20.0	524		104
20.0-25.0	524		104
25.0-30.0	524		104
30.0-35.0	393		78
35.0-40.0	524		104
40.0-45.0	393		78
45.0-50.0	524		104
50.0-55.0	524		104
```


----------



## einthusan (Apr 28, 2012)

To me that looks bad! My results are below, however I have a 1 GbE connection.

[CMD="iperf"]-P 10 -c IP.ADD.SAN.ITIZED -i 5 -f k[/CMD]

```
------------------------------------------------------------
Client connecting to IP.ADD.SAN.ITIZED, TCP port 5001
TCP window size: 47.8 KByte (default)
------------------------------------------------------------
[  3] local IP.ADD.SAN.ITIZED port 26689 connected with IP.ADD.SAN.ITIZED port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec  231040 KBytes  378536 Kbits/sec
[  3]  5.0-10.0 sec  232960 KBytes  381682 Kbits/sec
[  3]  0.0-10.0 sec  464128 KBytes  380186 Kbits/sec
```


----------



## ghostcorps (Apr 28, 2012)

einthusan said:
			
		

> To me that looks bad! My results are below, however I have a 1 GbE connection.



To give an idea of the degradation: from my other server in the US that has a max 5Mbps download I get about 1.9Mbps from the VPS. From Australia on a 100Mbps line I get about 200Kbps!

The above was tested with a 100MB zip file and wget.

Unfortunately I can not install iperf on the other US server. Would you mind if I PM'd you with my url to run iperf against?


----------



## einthusan (Apr 28, 2012)

ghostcorps said:
			
		

> To give an idea of the degradation: From my other server in the US that has a max 5Mbps download I get about 1.9Mbps from the VPS. From Aust on an 100Mbps line I get about 200Kbps!
> 
> The above was tested with a 100mb zip file and wget.
> 
> Unfortunately I can not install iperf on the other US server. Would you mind if I PM'd you with my url to run iperf against?



After running the test, your throughput was about 6 MB/s. Keep in mind that Mbps and MB per second are not the same. In your comment above, you were probably talking about MB per second but you stated it as Mbps.

1 MB/s (Megabytes per second) = 8 Mbps (Megabits per second).

Anyhow, 6 MB/s is low, extremely low. An old computer can do better  However, that should be enough to do more than 1 req/sec from both web servers. I suggest looking into your network card configurations first. iperf only tests the network card, so it can't be any other issue such as hard drive I/O or server setting.


----------



## einthusan (Apr 28, 2012)

einthusan said:
			
		

> So yea after running the test, your throughput about 6 MB/s. Keep in mind that Mbps and MB per second are not the same. In your comment above, you were probably talking about MB per second but you stated it as Mbps.
> 
> 1 MB/s (Megabytes per second) = 8 Mbps (Megabits per second).
> 
> Anyhow, 6 MB/s is low, extremely low. A old computer can do better  However, that should be enough to do more than 1 req/sec from both web servers. I suggest looking into your network card configurations first. iperf only tests the network card, so it can't be any other issue such as hard drive I/O or server setting.



Well, actually, maybe it's not worth checking your network card, I know it*'*s very low throughput but it's not like you*'re* doing file serving or video streaming. So keep debugging


----------



## ghostcorps (Apr 28, 2012)

Sorry, you are right, it should have been KB/s and MB/s.

I am speaking with the host now to get an idea of what the max throughput should be. It is a pretty cheap package I think.


----------



## ghostcorps (Apr 28, 2012)

Actually it is a streaming server


----------



## ghostcorps (Apr 30, 2012)

Something I should have done ages ago:

 I turned off the jails to negate the virtual NICs and ran the benchmark against a minimal apache install on the host server. The results were no different. Still waiting on the host to give an approximation on the minimal bandwidth I should expect.


----------



## ghostcorps (May 6, 2012)

I am still stumped lol

I have asked to have a vanilla FreeBSD VPS set up so I can get an idea of the expected throughput. But I have definitely done something screwy, traceroute starts timing out as soon as it touches anything on my system and I can not trace out at all. I assumed it was ipfw but I stopped it and still have the same issue.

It has to be a basic networking issue, which makes sence cos I am pretty hit-and-miss when it comes to that.

I installed webmin and munin and had a far more experienced admin have a look and we still couldn't find anything wrong. Below are the basic system configs is there anything that stands out?

/etc/inetd.conf is completely commented out.

This is what is left of rc.conf after I remove OpenVPN, apache the jails and a bunch of other services that are stopped during the failed traceroute

/etc/rc.conf

```
hostname="MY.URL.COM"
ifconfig_em0="inet XXX.XXX.XXX.XX2 netmask 255.255.255.248"
defaultrouter="XXX.XXX.XX2.XXX"
gateway_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a XXX.XXX.XXX.XX2"
rpcbind_enable="NO"
```


/boot/defaults/loader.conf

```
##############################################################
###  Networking modules  #####################################
##############################################################
if_disc_load="NO"               # Discard device
if_ef_load="NO"                 # pseudo-device providing support for multiple
                                # ethernet frame types
if_epair_load="NO"              # Virtual b-t-b Ethernet-like interface pair
if_faith_load="NO"              # IPv6-to-IPv4 TCP relay capturing interface
if_gif_load="NO"                # generic tunnel interface
if_gre_load="NO"                # encapsulating network device
if_stf_load="NO"                # 6to4 tunnel interface
if_tap_load="NO"                # Ethernet tunnel software network interface
if_tun_load="NO"                # Tunnel driver (user process ppp)
if_vlan_load="NO"               # IEEE 802.1Q VLAN network interface
ipfw_load="NO"                  # Firewall
pf_load="NO"                    # packet filter
```


I setup the security ages ago with a guide that I can't find right now. Looking back at it I can see some things that sound very much like what I am experiencing:... a blackhole.

/etc/sysctl.conf

```
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.ip.random_id=1
```

I commented out the blackholes and reloaded it, but this had no effect either.


----------



## ghostcorps (May 7, 2012)

Just found another stupid thing I did 

 I had somehow managed to have two rules in my firewall for ICMP type 8. One allowing and then one denying! After removing the second one it I can traceroute out from any location on the system, but whether tracerouting in or out it still times out when it gets to the VPSs' hop. I'm not concerned with this though because I now observe the following behaviour:

Running ab from remote location whilst on VPN:

*Against host server*

`ab -n 20 -c 10 192.168.254.1  <= VPN network`

```
3.05 [#/sec]
2726ms Longest Request
```

`ab -n 20 -c 10 xxx.xxx.xxx.1  <= External IP`

```
3.21 [#/sec]
2726ms Longest Request
```

`ab -n 20 -c 10 MY.HOST.URL.COM`

```
3.23 [#/sec]
2742ms Longest Request
```


*Against jailed webserver*

`ab -n 20 -c 10 192.168.254.2  <= VPN network`

```
1.28 [#/sec]
13754ms Longest Request
```

`ab -n 20 -c 10 xxx.xxx.xxx.2  <= External IP`

```
1.43 [#/sec]
12599ms Longest Request
```

`ab -n 20 -c 10 MY.SITE.URL.COM`

```
1.42 [#/sec]
12761ms Longest Request
```

`ab -n 20 -c 10 VHOST.SITE.URL.COM <= Test site`

```
[B]3.23 [#/sec]
2766ms Longest Request[/B]
```

 Any reasonable person would look at this and say it is clearly something to do wih the default site config in the jailed webservers' apache and would expect the same test against nginx to work correctly and yet:

`ab -n 20 -c 10 xxx.xxx.xxx.xx2:90 <= nginx site is on port 90`

```
1.01 [#/sec]
10677ms Longest Request
```

 It is kind of odd that the test site should have the same throughput as the host, while the other two sites on the jailed webserver are so slow. Is there anything that an nginx site and the default apache site have in common that is not shared by an apache vhost?


----------



## ghostcorps (May 10, 2012)

I am not surprised if you are all sick of my dumb moves, but I have one more for you...

 My Vhost has an auth file, so when I was testing ab against it it was failing the auth be registering as a completed request. Conversely, the main site was able to load all the content in each request and so was about 10x larger, hence why it took 10x longer.

 *dumb*

 Anyway, back on track now I am about ready to move to nginx, just doing some final checks.


Thanks again for your persistence everyone, I am sorry to have wasted your time.


----------



## einthusan (May 10, 2012)

ghostcorps said:
			
		

> I am not surprised if you are all sick of my dumb moves, but I have one more for you...
> 
> My Vhost has an auth file, so when I was testing ab against it it was failing the auth be registering as a completed request. Conversely, the main site was able to load all the content in each request and so was about 10x larger, hence why it took 10x longer.
> 
> ...



It's just that I have been very busy myself. I am testing out Varnish in front of Nginx but can't seem to get it working. Varnish is an http accelerator that will cache http requests.


----------



## ghostcorps (May 10, 2012)

Thanks for getting back to me 

Today I did an ab test with 100 concurrency on nginx and it didn't flinch. Providing things don't lock up again I guess this is resolved.


As for the latency, I won't post any more logs but I have been using nytimes.com as a benchmark and am more or less comparable. So that will have to do 

I'll mark this solved once nginx is live and properly stress tested.


----------



## einthusan (May 10, 2012)

ghostcorps said:
			
		

> Thanks for getting back to me
> 
> Today I did an ab test with 100 concurrency on nginx and it didn't flinch. Providing things don't lock up again I guess this is resolved.
> 
> ...



Awesome man! I know it was a huge pain and a lot of learning but it's good to know you have progressed so much. By the way, Varnish isn't that good and not worth the time and effort to set it up.


----------

