# Kernel memory issues with PAE u/g



## radix (Feb 20, 2011)

I know the drill about it not being the best idea to use PAE, but AMD64 is not an option for my circumstances.

First, Here is an output of uname -a:


```
FreeBSD xxxx.net 8.0-RELEASE FreeBSD 8.0-RELEASE #1: Thu Aug 12 19:04:50 UTC 2010
[email]root@xxxx.net[/email]:/usr/obj/usr/src/sys/PAE_KERNEL_9_2_0  i386
```

I have 12GB of memory installed and this server runs a large number of jails and has operated satisfactorily for half a year just using the 4GB of memory addressable with a GENERIC kernel.

Recently, I have begun to experience some memory resource issues that caused me to start looking into utilizing the full 12GB that I have available on this machine. With that in mind, I diligently followed the advice and instructions to build in PAE to my otherwise fairly GENERIC kernel (see attached kernel config file). I included 

```
options KVA_PAGES=512
device acpi
device ispfw
```
 as indicated. I also tweaked my kern.maxfiles to 110000, up from 100000. The build and installation went faultless.

On reboot with the new PAE kernel, it booted just fine. However, after the 16th jail, it stopped loading the remaining jails.

In the /var/logs/messages, I started seeing the following entries repeating:


```
kernel: vm_thread_new: kstack allocation failed
```

Checking the values of sysctl vm.kvm_free it said "0".

Here are my sysctl vm.kvm* settings:


```
vm.kmem_size_max: 335544320
vm.kmem_size: 335544320
```

I thought okay, I need to up the vm.kmem_size_max and vm.kmem_size values in my /boot/loader.conf file, so I set them, as such:


```
vm.kmem_size_max="1073741824"
vm.kmem_size="1073741824"
```

On reboot, it froze with the following console error:


```
panic: kmem_suballoc: bad status return of 3
cpuid = 0
```

I had to boot into single user to remove the kmem stuff from loader.conf.

I am back using my old stalwart GENERIC kernel, but would sure like to understand what I could do to increase my kmem_size and kmem_size_max. I understand that there is supposed to be a kernel mempory hard limit of 2048M in 32bit FBSD, but that is twice what I had set and I still received the error.

Grateful for any advice.


Jake


----------



## radix (Feb 26, 2011)

In case anyone reads this issue, I have not solved this yet and welcome any insight.

Jake


----------



## Terry_Kennedy (Feb 28, 2011)

radix said:
			
		

> In case anyone reads this issue, I have not solved this yet and welcome any insight.


I don't have any useful information for you. I just wanted to say that since the amd64 support stabilized (particularly in the ports area), there aren't that many people running PAE kernels any more, so it may be some time before you get a useful answer.

If there's one particular thing preventing you from using amd64, you might try seeking an answer for that rather than hoping someone with PAE experience shows up.


----------



## rstone@ (Mar 1, 2011)

It seems that i386 defaults to only using 1GB for the kernel address space, not 2GB.  You can increase the kernel by setting the kernel option KVA_PAGES larger.  To get 2GB of kernel address space on a PAE kernel, set the option as follows:


```
option KVA_PAGES=1024
```

see sys/i386/conf/NOTES for more information about tuning this value.


----------

