# CPU limit for jails under ULE scheduler



## kostjn (Apr 17, 2009)

Hello all!
I`m rewrire original cdjones patch  for cpu limit for jail under ULE scheduler.
So,  this work simple.
We count cpu usage for all jails, and if jail use cpu more than have shared cpu, we move his threads to IDLE queue.
Jailed thread can use all avaliable cpu time, if  system has avaliable cpu.
If system under heavy load, jailed thread can`t use cpu long as ratio (shared cpu for jail/ all shared cpu) < (estimate usage cpu for jail / all usage cpu) .
Unjailed thread are not subject to this regime.
Interactive thread also are not subject to this regime.
Add 2 sysctl
*kern.sched.total_sched_shares* - total count shares cpu in system, increase if we have more cpu
*kern.sched.flush_estcpu_interval* - flush estcpu interval in ticks, default is 2560 = 2 * 128 * 10, NCPU*stathz*sec, increase if we have more cpu
For use cpu limit, you need use flag -S NSharedCPU in /usr/sbin/jail program.
My example 
	
	



```
jail -S100 /usr/jails/root/ root.kostjn.pht  192.168.0.245 /bin/csh
```

I`m tested this under 10 simultaneous process in jail and in main system. test program is infinity cycle an 8 core xeon, use RELENG_7.
First run process in jail, and after in main system.
This one process tracking cpu usage
Jail

```
root    1052  0.0  0.0  3692   784  p1  RJ    7:38PM   0:00.39 /test.o
root    1052 21.2  0.0  3692   784  p1  RJ    7:38PM   0:02.40 /test.o
root    1052 35.6  0.0  3692   784  p1  RJ    7:38PM   0:04.40 /test.o
root    1052 47.5  0.0  3692   784  p1  RJ    7:38PM   0:06.41 /test.o
root    1052 39.9  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 33.2  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 27.6  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 22.9  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 19.0  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 15.8  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 13.0  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052 10.8  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /test.o
root    1052  8.9  0.0  3692   784  p1  RJ    7:38PM   0:06.62 /tes
```
Main system

```
root    1088 14.9  0.0  3692   780  p0  R     7:38PM   0:01.57 /root/test.o
root    1088 30.8  0.0  3692   780  p0  R     7:38PM   0:03.60 /root/test.o
root    1088 43.8  0.0  3692   780  p0  R     7:38PM   0:05.60 /root/test.o
root    1088 51.0  0.0  3692   780  p0  R     7:38PM   0:07.25 /root/test.o
root    1088 50.8  0.0  3692   780  p0  R     7:38PM   0:08.28 /root/test.o
root    1088 49.1  0.0  3692   780  p0  R     7:38PM   0:09.21 /root/test.o
root    1088 48.1  0.0  3692   780  p0  R     7:38PM   0:10.24 /root/test.o
root    1088 46.2  0.0  3692   780  p0  R     7:38PM   0:11.17 /root/test.o
root    1088 42.9  0.0  3692   780  p0  R     7:38PM   0:11.95 /root/test.o
```
So we see, that after run in main system, jailed process can`t usage cpu.

I`m don`t have big expirience in kernel programming, consequently best if you see source code.
Please test this patch  and post this you restult or all problem.
This is initial version, without tune jail parameter in runtime.
Thank.
Original cdjones  cpu and memory limit patch http://wiki.freebsd.org/JailResourceLimits


----------



## kostjn (Apr 18, 2009)

Patch for RELENG7 http://kostjn.spb.ru/patch-jail-cpu-limit-7RELENG
Patch for RELEASE71 http://kostjn.spb.ru/patch-jail-cpu-limit-71RELEASE


----------



## DutchDaemon (Apr 18, 2009)

*modded up*


----------



## kostjn (May 20, 2009)

*Jail limit for CURRENT*

Hi.
I`m rewrite jail limit patch under CURRENT.
New patch limited CPU, memory, filedesc, process.
And allow change limit on the fly.

===========================================================================
How to use.
===========================================================================
Build

```
cvsup CURRENT
cd /usr/src
patch -p0 < patch-jail-limit-8CURRENT
make buildkernel
make buildworld
make installkernel
reboot
make installworld
```

Create new entry in login.conf, for example class jail128

```
jail128:\
	:cputime=10:\
	:memoryuse=128M:\
	:maxproc=256:\
	:openfiles=1024:\
	:tc=default:
```
Cputime is percent on 1 core.
Openfiles is sum filedesc for all proc in jail.

Create new jail.
...
Add in /etc/rc.conf

```
jail_test_flags="-Ljail128"
```

Run new jail

```
/etc/rc.d/jail start test
```

===========================================================================
Sysctl
===========================================================================

```
[root@book ~]# sysctl security.jail.limit
security.jail.limit.enable: 1
security.jail.limit.memory_exceed_kill: 0
[root@book ~]# sysctl -d security.jail.limit
security.jail.limit: Jail limit
security.jail.limit.enable: Enable jail limit
security.jail.limit.memory_exceed_kill: Kill biggest proc in jail, if jail excee
d memory limit
```


===========================================================================
Jset and Jget
===========================================================================
jset and jget is program for set new jail limit and get current limit
Example

```
[root@book ~]# cat /etc/rc.conf | grep jail2
jail_list="jail1 jail2 jail3 jail4 jail5 jail6 jail7 jail8 jail9 jail10"
jail_jail2_rootdir="/usr/jails/jail2/"
jail_jail2_hostname="jail2.book.pht"
jail_jail2_interface="re0"
jail_jail2_ip="192.168.200.22"
jail_jail2_flags="-Ljail64"
[root@book ~]# /etc/rc.d/jail start jail2
Configuring jails:.
Starting jails: jail2.book.pht.
[root@book ~]# cd ~kostjn/

[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65

[root@book /home/kostjn]# ./jset.o 1 jail2048
Set new jail limits, jid = 1
Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
[root@book /home/kostjn]# ./jget.o 1

Jail limits and rusage, jid = 1
Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65
```

You see that new limit is set.

===========================================================================
Test
===========================================================================
Cpu limit
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Script 

```
[root@book /home/kostjn]# cat test.sh
#!/bin/sh
for i in `jot 8 1`; do cpuset -l0  jexec 1 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 2 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 3 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 4 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 5 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 6 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 7 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 8 /a.out & done
for i in `jot 8 1`; do cpuset -l0  jexec 9 /a.out & done

cpuset -l0  jexec 10 /a.out &

Set class for all jail.

[root@book /home/kostjn]# for i in `jot 10 1`; do ./jset.o $i jail128 ;done
Set new jail limits, jid = 1
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 2
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 3
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 4
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 5
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 6
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 7
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 8
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 9
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 10
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024

[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# cat cpu.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <math.h>

int main(int argc,char *argv[]){
        int64_t i,j=0;
        char *s;
        for (;;){
        }

}

Run test.sh

Result 
top
last pid:  3513;  load averages: 70.87, 37.58, 16.40    up 0+00:44:02  14:19:46
185 processes: 74 running, 111 sleeping
CPU: 49.9% user,  0.0% nice,  0.0% system,  0.2% interrupt, 49.9% idle
Mem: 139M Active, 24M Inact, 47M Wired, 192K Cache, 29M Buf, 1785M Free
Swap: 4044M Total, 4044M Free

  PID JID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAN
 3502  10 root          1  97    0  1480K  1244K CPU0    0   0:13  8.79% a.out
 3474   6 root          1  97    0  1480K  1244K RUN     0   0:04  4.69% a.out
 3431   2 root          1  96    0  1480K  1244K RUN     0   0:03  4.30% a.out
 3454   4 root          1  97    0  1480K  1244K RUN     0   0:03  4.05% a.out
 3422   1 root          1  96    0  1480K  1244K RUN     0   0:04  3.86% a.out
 3482   7 root          1  97    0  1480K  1244K RUN     0   0:03  3.86% a.out
 3447   3 root          1  97    0  1480K  1244K RUN     0   0:03  3.86% a.out
 3429   1 root          1  96    0  1480K  1244K RUN     0   0:03  3.66% a.out
 3485   8 root          1  97    0  1480K  1244K RUN     0   0:05  3.56% a.out
 3424   1 root          1  96    0  1480K  1244K RUN     0   0:04  3.56% a.out
 3464   5 root          1  97    0  1480K  1244K RUN     0   0:02  3.56% a.out
 3438   2 root          1  96    0  1480K  1244K RUN     0   0:03  3.47% a.out
 3494   9 root          1  96    0  1480K  1244K RUN     0   0:03  3.27% a.out
 3497   9 root          1  97    0  1480K  1244K RUN     0   0:05  3.17% a.out
 3433   2 root          1  96    0  1480K  1244K RUN     0   0:03  2.88% a.out
 3428   1 root          1  96    0  1480K  1244K RUN     0   0:02  2.88% a.out
 3487   8 root          1  97    0  1480K  1244K RUN     0   0:04  2.78% a.out
```

We see that jail 10 (1 thread), used  ~10 % cpu under heavy load.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Resourse compute

```
[root@book /home/kostjn]# ./jset.o 1 jail64
Set new jail limits, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65
[root@book /home/kostjn]#
[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# apachectl stop
/usr/local/sbin/apachectl stop: httpd stopped
[root@jail1 /]# exit
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
```

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Resource limit

```
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# cat mem.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <math.h>

int main(int argc,char *argv[]){
        int64_t i,j=0;
        char *s;
        for (i=0; i  < 1000 ;i++){
                s = malloc(100000 * sizeof(char));
        }
        sleep(1000);
}

[root@jail1 /]# cc mem.c && ./a.out &
[1] 1320
[root@jail1 /]# ls
bash: fork: Cannot allocate memory
[root@jail1 /]# exit
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 1, MEM 103M, NPROC 5, NOFILE 31
```

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
We see that jail exceed memory limit. And new fork, mmap syscall not 
permitted.
If you set  sysctl 


```
[root@book /home/kostjn]# sysctl security.jail.limit.memory_exceed_kill=1
security.jail.limit.memory_exceed_kill: 1 -> 1
[root@book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
[root@book /home/kostjn]# jexec 1 bash
[root@jail1 /]# ./a.out
Killed: 9
[root@jail1 /]# exit
[root@book /home/kostjn]# tail -n 1 /var/log/messages
May 20 14:10:17 book kernel: pid 1337 (a.out), uid 0, jid 1 was killed: Prison e
xceed memory limit
```

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
If you attempt set nonexisten class, limit set to infinity.


```
[root@book /home/kostjn]# ./jset.o 1 jail123
Set new jail limits, jid = 1
Limits: CPU 9223372036854775807, MEM 20M, NPROC 9223372036854775807, NOFILE 9223
372036854775807
```


===========================================================================
Problem
===========================================================================
If you have problem in this patch.
Add to kernel config 

```
options     KTR
options     KTR_ENTRIES=1024
options KTR_COMPILE=(KTR_PROC|KTR_JAIL|KTR_SCHED|KTR_RUNQ|KTR_LOCK|KTR_CONTENTIO N)
options     KTR_MASK=KTR_JAIL
options     KTR_CPUMASK=0x3
options     KTR_VERBOSE
options     PRINTF_BUFR_SIZE=128
```
Rebuild kernel.
Reboot.
Set sysctl

```
sysctl debug.ktr.mask=65536
```
and check /var/log/messages


----------



## vivek (May 27, 2009)

Thanks for posting this. Have you tested it with 72RELEASE?


----------



## kostjn (May 28, 2009)

vivek said:
			
		

> Thanks for posting this. Have you tested it with 72RELEASE?



This patch will not work in 7-STABLE.
The reason in that in CURRENT developers have altered a code jail, new system calls and so on have been added.


----------



## Koduc (May 29, 2009)

To kostjn: thx for your staff.

Are there ways to limit cpu & ram for jails in 7.2-RELEASE? Is it real?


----------



## kostjn (May 30, 2009)

Koduc said:
			
		

> To kostjn: thx for your staff.
> 
> Are there ways to limit cpu & ram for jails in 7.2-RELEASE? Is it real?


Yes, certainly.
But now, I am engaged in improvement of a patch for CURRENT.
FreeBSD 8 leaves already soon, therefore I do not see sense to transfer a patch to 7 branch.


----------

