# IBM Server with 12GB - FreeBSD only sees 3GB



## DrinkingGuinness (Mar 27, 2009)

Hi Guys,

I recently upgraded the memory in my proxy servers from 4GB to 12GB.  Now when I run a sysctl hw.physmem I receive this.  

# sysctl hw.physmem
hw.physmem: 3207835648

Can someone tell me if this is a limitation of the OS or is there something else I must do?

Thanks,
Sean :beergrin


----------



## DutchDaemon (Mar 27, 2009)

You can't use more than 3GB of RAM on a 32-bit installation.


----------



## rbelk (Mar 27, 2009)

According to the FreeBSD manual 4.2 Memory.

Yes. FreeBSD as an operating system generally supports as much physical memory (RAM) as the platform it is running on does. Keep in mind that different platforms have different limits for memory; for example i386â„¢ without PAE supports at most 4 GB of memory (and usually less than that because of PCI address space) and i386 with PAE supports at most 64 GB memory. AMD64 platforms currently deployed support up to 1 TB of physical memory.


----------



## DutchDaemon (Mar 27, 2009)

And PAE is not exactly advisable. It does tend to break some apps.


----------



## rbelk (Mar 27, 2009)

It has worked with no issues for me. I am running FreeBSD 7.1-RELEASE #0 with all patches on a Dell Workstation with 8 gig's of memory. I have noticed that it seems memory management is slower, if that makes any sense.


----------



## DutchDaemon (Mar 27, 2009)

It is slower, yes (Chapter 4.2 does say so).


----------



## DrinkingGuinness (Mar 28, 2009)

Thanks for the help guys.  Well I think I am stuck.  I have IBM Servers with Xeon processors.  I read that only the itanium processor is compatible with the 64bit version.  Does anyone have any insight using PAE with squid?

Thanks again guys!!
:beergrin
Sean


----------



## fronclynne (Mar 28, 2009)

DrinkingGuinness said:
			
		

> I have IBM Servers with Xeon processors.  I read that only the itanium processor is compatible with the 64bit version.


I'm not sure where you read that, but all modern (err, wikipedia saith: 'The last processors Intel manufactured which did not use AMD's x86-64 design were early versions of the Pentium 4 "Prescott", introduced in February 2004.' http://en.wikipedia.org/wiki/Em64t ) xeons are going to be amd64 compatable.  And if your motherboard supports 12G of ram, I'd be pretty safe in betting yours is amd64.

Good luck.


----------



## vermaden (Mar 28, 2009)

DrinkingGuinness said:
			
		

> Thanks for the help guys.  Well I think I am stuck.  I have IBM Servers with Xeon processors.  I read that only the itanium processor is compatible with the 64bit version.



Depends on the model, check you model here:
http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors


----------



## fronclynne (Mar 29, 2009)

```
> dmesg -a | grep Feat
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x2001<SSE3,CX16>
  AMD Features=0xea500800<SYSCALL,[color=red]NX[/color],MMX+,FFXSR,RDTSCP,[color=red]LM[/color],3DNow!+,3DNow!>
  AMD Features2=0x11f<LAHF,CMP,SVM,ExtAPIC,CR8,Prefetch>
```
other features may be dead givaways of amd64, too.

I was pleasantly surprised* at how informative /usr/src/sys/amd64/amd64/identcpu.c (or /usr/src/sys/i386/i386/identcpu.c) is.


*not that I honestly should be: personal weakness


----------



## Djn (Mar 29, 2009)

DrinkingGuinness said:
			
		

> Thanks for the help guys.  Well I think I am stuck.  I have IBM Servers with Xeon processors.  I read that only the itanium processor is compatible with the 64bit version.  Does anyone have any insight using PAE with squid?
> 
> Thanks again guys!!
> :beergrin
> Sean



Just for clarity: Intel makes two 64-bit CPU series.
- Itanium (rare and expensive, for highend servers). You'll often see this labeled as IA64.
- x86 with 64-bit extensions. Intel calls this EMT64, the FreeBSD port is called amd64 (for historical reasons), and you'll also see x86-64. Almost all modern x86 CPUs support this.


----------



## ApoC (Mar 29, 2009)

DutchDaemon said:
			
		

> You can't use more than 3GB of RAM on a 32-bit installation.



This is BIG FUD.

On modern "i386" hardware you can use more than 3GB of RAM exactly because of PAE (http://en.wikipedia.org/wiki/Physical_Address_Extension#FreeBSD). Only limitation is that application is not able to address more than 32b pointer allows in one segment. How this can be avoided is for another discussion (it is possible).

About speed of PAE. It is little bit slower "by desing" but by meassuress it is about 1-5% in real applications. So no big deal.


----------



## ApoC (Mar 29, 2009)

For more information I suggest to read "man 4 pae"

"The PAE option provides support for the physical address extensions capa-
     bility of the Intel Pentium Pro and above CPUs, and allows for up to 64
     gigabytes of memory to be used in systems capable of supporting it."


----------



## Djn (Mar 29, 2009)

ApoC said:
			
		

> This is BIG FUD.
> 
> On modern "i386" hardware you can use more than 3GB of RAM exactly because of PAE (http://en.wikipedia.org/wiki/Physical_Address_Extension#FreeBSD). Only limitation is that application is not able to address more than 32b pointer allows in one segment. How this can be avoided is for another discussion (it is possible).
> 
> About speed of PAE. It is little bit slower "by design" but by meassuress it is about 1-5% in real applications. So no big deal.



Still no reason not to use the amd64 version, though - it's a mature and useful port, with some specific benefits (e.g. that the applications get 64 bits of address space to play with - now and then that could be useful).


----------



## DutchDaemon (Mar 29, 2009)

ApoC said:
			
		

> This is BIG FUD.



And that's bullshit. 

PAE is an *extension* to a 32-bit installation, requiring a new kernel build and a new build of KLDs, so my response about a 32-bit installation was factually correct, not some opinion (and I know it's ~3GB, usually about 3.2).


----------



## ApoC (Mar 29, 2009)

DutchDaemon said:
			
		

> And that's bullshit.
> 
> PAE is an *extension* to a 32-bit installation, requiring a new kernel build and a new build of KLDs, so my response about a 32-bit installation was factually correct, not some opinion (and I know it's ~3GB, usually about 3.2).



So You should wrote that default FreeBSD 32b installation doesn't support PAE, BUT it can be easily enabled by kernel "options PAE" and rebuilding.

May be your answer was correct but not helpful to the guy who asked. So I think conclusion is:

1. Default FreeBSD 32b installation not support PAE, so cca 3.2GB of RAM is maximum what OS can see.
2. PAE can be enabled by "options PAE" in kernel config. Recompiling kernel and KLDs is required.
3. Some drivers may have problems with PAE enabled.
4. There is a little slowdown in memory performance 1-5%
5. Processes will not be able to allocate more memory than what can be addressed by single 32b pointer (this can be problem for some HUGE DB servers)
6. For further information about PAE and kernel configure/compile/install anyone can follow man pages and handbook 

DutchDaemon if You take the "FUD" personally and You feel harmed, I do my apology.


----------



## DutchDaemon (Mar 29, 2009)

That's fine. I just thought 'fear, uncertainty, doubt' was a little bit strong. I'm not Ballmer ..


----------



## vivek (Mar 29, 2009)

Why you wanna use 32 bit OS for 12GB RAM? 64bit works nicely, we have couple of boxes with 8GB RAM running FreeBSD 7.1. Though I never tried 4.x series.


----------



## ApoC (Mar 29, 2009)

I think author has already deployed server where upgraded to 12GB from 4GB. May be he won't reinstall all server, applications and so on because of memory upgrade.


----------



## DrinkingGuinness (Mar 30, 2009)

WOW!  Thanks for all the responses.  I already have multiple proxy servers built with the 32bit version.  Because they are load balanced I plan on rebuilding one of them with the 64 bit version, test and hopefully rebuild the others.  I will post any results I find with squid and the 64 bit version of FreeBSD after I rebuild it as well as any issues with the hardware and memory.

Once again thanks guys!!!  :beergrin

Sean


----------



## DutchDaemon (Mar 31, 2009)

I tested Squid on a FreeBSD 7.1 64-bit HP blade a few weeks ago. No problem whatsoever. Note that PAE does not circumvent what ApoC mentioned earlier -> a process is not able to access more than 4 gigabytes of VM space. So the single Squid process will still be limited to 2^32 bytes on a PAE installation; you may give additional stuff like diskd and statistics (Calamaris, Sarg) some more breathing room, but that's about it.


----------



## DrinkingGuinness (Apr 1, 2009)

OK Here are my results.

FreeBSD-AMD64 installed on my server without any problems and squid is running great!  Memory on the server shows 12GB, I installed a cool app called FREE and it shows all memory statistics.  Here are my new results with FreeBSD-AMD64.

```
SYSTEM MEMORY INFORMATION:
mem_wire:         208478208 (    198MB) [  1%] Wired: disabled for paging out
mem_active:  +     38502400 (     36MB) [  0%] Active: recently referenced
mem_inactive:+     50601984 (     48MB) [  0%] Inactive: recently not referenced
mem_cache:   +       139264 (      0MB) [  0%] Cached: almost avail. for allocation
mem_free:    +  12163497984 (  11600MB) [ 97%] Free: fully available for allocation
mem_gap_vm:  +       450560 (      0MB) [  0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all:     =  12461670400 (  11884MB) [100%] Total real memory managed
mem_gap_sys: +    409632768 (    390MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    =  12871303168 (  12275MB)        Total real memory available
mem_gap_hw:  +     13598720 (     12MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      =  12884901888 (  12288MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:         670662656 (    639MB) [  5%] Logically used memory
mem_avail:   +  12214239232 (  11648MB) [ 94%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   =  12884901888 (  12288MB) [100%] Logically total memory
```
I am still having one problem, and I hope someone here can help.  I am trying to use socks5 v1.0r6 but when I try to start the application I receive this error ELF interpreter /usr/libexec/ld-elf.so.1 not found.  Now I have noticed there is a size difference in the working server and the new 64bit server.  

```
working (32)     - root  wheel  173000 Jan  1 06:49 ld-elf.so.1
not working (64) - root  wheel  198048 Jan  1 02:48 ld-elf.so.1
```
I have done a little searching on google and cannot find much regarding socks5 and FreeBSD-AMD64.  Does anybody have any insight on this?  

As always, thanks guys!!!!  
:beergrin
Sean

I updated this post to make sure everyone understands I am talking about FreeBSD and not anything else.  However, I think most of you understood that already


----------



## tingo (Apr 3, 2009)

@DrinkingGuinness:
in the interest of correctness, the operating system you are using is called FreeBSD, not "BSD". Or are you perhaps using NetBSD or OpenBSD?


----------

