# How much memory FreeBSD/amd64 can allocate to an i386 binary ?



## Adrien2002 (Aug 25, 2017)

Hello,
I'm using FreeBSD/i386 to easily build the lastest Wine version. It is for test purposes, I want to improve FreeBSD compatibility in Wine but I saw that FreeBSD/i386 will only use 2 GB of RAM which is clearly not enough for Wine. Every big games like Skyrim, DOOM 3, Quake 4 and so much more will crash complaining about memory allocation. This problem doesn't exists in GNU/Linux because their i386 will allocate around 3,9 GB which is clearly enough because Windows 64 bits also offer that amount of memory for their 32 bits applications.

I could add PAE support but :
1. It will not change the amount of memory one application can use (so still stuck with 2 GB).
2. It seems to be broken. I tried but I have endless reboots.

How much memory FreeBSD/amd64 can allocate to an i386 binary ? Also 2 GB ? Or can it offer more ?


----------



## SirDice (Aug 25, 2017)

Adrien2002 said:


> It is for test purposes, I want to improve FreeBSD compatibility in Wine but I saw that FreeBSD/i386 will only use 2 GB of RAM which is clearly not enough for Wine.


I don't know where you got this from but the limitation is a little under 4GB, just like any other OS. The limitation is due to processor registers, not the OS. 



Adrien2002 said:


> How much memory FreeBSD/amd64 can allocate to an i386 binary ? Also 2 GB ? Or can it offer more ?


4GB. The limitation is on the CPU, not the OS.


----------



## Adrien2002 (Aug 25, 2017)

I got this information from my installed FreeBSD ! So yes, I suppose you are right but then it means that since 2013, I played all these games on Wine like Skyrim with only 2 GB when I was on GNU/Linux when people say it might not work correctly ? I mean, now it doesn't work, it's sure, it crashes like almost every other big games.

I can quickly show you the output of Quake 4 : wineserver: file_set_error() can't map error: Cannot allocate memory

So like other games, it crashes.

Where do I get this information by the way ? I got it from my dmesg

$ dmesg |grep memory
real memory  = 12884901888 (12288 MB)
avail memory = 2024976384 (1931 MB)

I am running a Core i7 4700HQ @ 2.4 GHz

PCSX2 too is crashing with this message : GSdx: out-of-memory, texturing temporarily disabled

I opened a bug report about it and they all say that it must be a lack of memory. When I am on Windows 10, I can play for hours with PCSX2 without even one crash. This information about FreeBSD only having 2 GB, I made it by myself after seing so many things crashing on FreeBSD due to a memory issue when it works everywhere else on the same architecture. Whatever it is true or false, I just try to find out WHY everything is crashing that way !

And I tried, I tried HARD to get informations, to get help from other people but it seems no one give a damn to that issue, I don't know where to look now, if I always had 2 GB of RAM on everything 32 bits since 4 years now without to ever noticing it, where's the problem then ?


----------



## skywhi (Aug 25, 2017)

Hi,

I don't know if this helps but maybe you can check if the right amount of memory is detected with:
`# sysctl hw.machine_arch hw.ncpu hw.physmem hw.usermem`


----------



## SirDice (Aug 25, 2017)

Is this perhaps a machine with memory being shared with the videocard? That might account for the "missing" memory. 32 bit applications can only address 4GB but some some of this memory space is mapped to I/O and/or expension cards. Which is why you typically have 3.8-3.9 GB available. But I'm not sure what happens with those shared video memory cards. I would suspect it's going to eat up some of that 4GB space too.


----------



## Adrien2002 (Aug 25, 2017)

skywhi → 
# sysctl hw.machine_arch hw.ncpu hw.physmem hw.usermem
hw.machine_arch: i386
hw.ncpu: 8
hw.physmem: 2065039360
hw.usermem: 1890734080

SirDice →
It's an ASUS ROG with a GTX 765m, I doubt it shares the memory between the video one and the RAM. And, really, it always worked like a charm on GNU/Linux using Wine 32 bits and on Windows. If my hardware is the problem, I should have this issue since the beginning, not only since I am using FreeBSD, no ?

I have another computer, an old machine I made myself with only 4 GB and I booted the FreeBSD/i386 installed, dropped into a shell and check the available memory and, yes, it says I have 4096 MB of real memory and 3175 MB of available memory so you are right.
Then... Why ? I can't understand  why on this machine with 12 GB, I can only see 2 ! It's driving me crazy


----------



## Adrien2002 (Aug 26, 2017)

I'm really trying to understand...

I downloaded a Debian/i386 and the same, it only sees 2 GB...
I'm really impressed, it means I always ran my 32 bits binaries with 2 GB in the past ?

I want to come back to the initial question then because it's the first time I run an i386 operating system on this machine so it's the first time I see that it uses 2 GB but can it be different if I use an 64 bits operating system ? Can it allow more ? I know it's the CPU who manage this but maybe the CPU can allow more in the case of running an amd64 ? (I have no idea, it's a technical question and I don't know how the answer to that question).
Maybe the Core i7 will allow up to 4 GB if you are running an amd64 OS and only 2 GB if you are running an i386 OS (in some way).
Maybe if I try FreeBSD/amd64, I'll be able to get things working as it should !


----------



## Ordoban (Aug 26, 2017)

The CPU can handle 4GB in 32Bit mode, but not every software can.
Memory related variables are often declared as 32bit SIGNED integer, and this signed integer can handle values between 2147483647 and -2147483648. This value limitation is the "magic" 2GB.

The 2GB seems a very common crash point for memory-leaking programs. I have seen programs crashing around 2GB on Windows 32, Windows 64(32Bit mode) and Linux too.

I would suggest you to use 64bit Versions of OS and programs to avoid memory limitations.
If you are need to use 32bit programs on a 32bit FreeBSD, you can try a PAE enabled kernel version.
With PAE the programs itself can use only 4GB each one, but the kernel itself can handle more.


----------



## Adrien2002 (Aug 26, 2017)

Ordoban said:


> The CPU can handle 4GB in 32Bit mode, but not every software can.



Yes but, here, FreeBSD itself only use 2 GB so whatever they can use 4 GB or 256 GB, it will only be able to use 2 GB in every cases which cause all these crashes.



Ordoban said:


> I would suggest you to use 64bit Versions of OS and programs to avoid memory limitations.
> If you are need to use 32bit programs on a 32bit FreeBSD



Well, you are right and that's why I said it was the very first time I use an i386 OS in 4 years, I just wanted to use it to be able to build easily Wine. Wine will allow FreeBSD to run Windows executables but Windows applications are almost all in 32 bits. If I can simply build Wine 64 bits and use it for 32 bits applications, I won't create this thread but, on FreeBSD/amd64, I'll have to chroot to build Wine 32 to be able to test 32 bits applications from Windows on FreeBSD. Clearly, it was way easier for me to install FreeBSD/i386 and build as is Wine than instal FreeBSD/amd64 and find a solution for 32 bits rpath which is a bit more annoying to do.

Same with PCSX2 which doesn't have any package in repositories. This software is 32 bits ONLY and, because I only have 2 GB of memory when on i386, PCSX2 crashes.
Even if my CPU offers me 2 GB, it is OF COURSE better to use amd64 to run i386 applications because when FreeBSD is running in 32 bits, it will run inside those 2 GB available so games will have to push the OS to have space to sit but if in amd64, the 2 GB will be "empty".



Ordoban said:


> you can try a PAE enabled kernel version.
> With PAE the programs itself can use only 4GB each one, but the kernel itself can handle more.



I invite you to try PAE now, it doesn't work for me, it boots then reboot then reboot etc... I can not use PAE because it causes kernel panic at boot, that's why I created this topic. By the way, is it the same when using PAE than when using amd64 about 32 bits binaries ?
Using PAE will allow the OS to use more than 4 GB of memory BUT will still be able to give up to ~ 4 GB to applications.
Using amd64 will do the same, no ? I can use my 12 GB of memory but will only be able to allocate 4 GB to 32 bits applications, that's right ?

By the way, thank you for your technical answers, guys !


----------



## Ordoban (Aug 26, 2017)

Adrien2002 said:


> I invite you to try PAE now, it doesn't work for me, it boots then reboot then reboot etc... I can not use PAE because it causes kernel panic at boot, that's why I created this topic.


PAE is a way to work around the 32bit system limitations, but PAE is still a crutch.


Adrien2002 said:


> By the way, is it the same when using PAE than when using amd64 about 32 bits binaries ?
> Using PAE will allow the OS to use more than 4 GB of memory BUT will still be able to give up to ~ 4 GB to applications.


Yes, and PAE requires chipset support (forgot to say that, sorry).


Adrien2002 said:


> Using amd64 will do the same, no ? I can use my 12 GB of memory but will only be able to allocate 4 GB to 32 bits applications, that's right ?


Right. 4GB only if everything is made right, 2GB in every else case. Seems in your case is something wrong. Where? I don't know.
Maybe BIOS, FreeBSD, wine, or clang. 

By the way you can use more memory in a application than the whole system have. The missing ram will be taken from swap space. Extremely slow, but works.


----------



## Terry_Kennedy (Aug 27, 2017)

Adrien2002 said:


> I just wanted to use it to be able to build easily Wine. Wine will allow FreeBSD to run Windows executables but Windows applications are almost all in 32 bits. If I can simply build Wine 64 bits and use it for 32 bits applications, I won't create this thread but, on FreeBSD/amd64, I'll have to chroot to build Wine 32 to be able to test 32 bits applications from Windows on FreeBSD. Clearly, it was way easier for me to install FreeBSD/i386 and build as is Wine than instal FreeBSD/amd64 and find a solution for 32 bits rpath which is a bit more annoying to do.


You really want to use the amd64 FreeBSD version. I have never built / used wine, but amd64 FreeBSD provides 32-bit compatibility "out of the box" - if you want to turn 32-bit compatibility off, you need to go out of your way to do it. amd64 FreeBSD is capable of building programs in 32-bit mode as well as executing them. I have a number of large 100K to several MLOC packages that need to be built in 32-bit mode. The "magic" on the compiler command line (for gcc, clang may be slightly different) is: 
	
	



```
-m32 -DCOMPAT_32BIT -L/usr/lib32 -B/usr/lib32
```
I don't know if wine takes this into account or not, but feel free to mention the above code to the appropriate developer(s) if it will help.


----------



## Adrien2002 (Aug 27, 2017)

The "best" solution is still to use a chroot. Not because I have nothing to set to build in i386 but because Wine will require multiple packages to build (way more than the files in /usr/lib32), so did I.

I installed FreeBSD/amd64, I fetched the base of i386 to extract it somewhere, I chrooted in it, I used pkg to install everything Wine needs to build and run, I built Wine.

From there, I have a Wine built and installed in the chroot. Of course, if I just try to run it, it won't work because it will look for the libs in /usr/local/lib which is my amd64 installation si I checked the i386-wine-staging to see what do they do and they use a variable I never saw before : LD_32_LIBRARY_PATH

I was trying with LD_LIBRARY_PATH and it wasn't working at all but with LD_32_LIBRARY_PATH, it perfectly works.

SO ! I went to try Skyrim which is known to be quite big !
I can run it now, it runs as it should and it uses 1700 MB in my memory.

The problem ? (yes, there's one), my machine is still limiting 32 bits applications to 2 GB and, IF Skyrim reaches 2 GB, it will simply die. It usually stay between 1500 and 1750 MB but it can go a little bit higher. If it does, the game just freeze and I have to kill wine 

Is it the fault of FreeBSD ? Absolutely not, FreeBSD is doing its amazing job as it should, nothing wrong with this operating system  If i have 2 GB, it's NOT the fault of FreeBSD because Debian too is only having 2 GB. I just suppose it always worked in the past because I was lucky to never reach 2 GB, that's all !

Quake 4 is running without trouble too, I'm going to try out PCSX2 but I suppose the problem is just entirely solved.

I just have two questions :

Can a swap help Wine ? I know it will refuse to receive more memory from my machine, it will stick to its 2 GB and crash if it arrives to it but CAN IT use swap to avoid crashes (at the price of slow down in loading at some moments I guess)

In "top", the memory used by a software is in "RES", right ?


----------



## Ordoban (Aug 27, 2017)

After become curious, I made some investigations.
Memory limitations on wine to 2GB is well known: https://ubuntuforums.org/showthread.php?t=2233124
I have written a small 32 bit windows program who just allocate memory:

```
stefan@Desktop3 ~/winec $ uname -a
Linux Desktop3 4.10.0-28-generic #32~16.04.2-Ubuntu SMP Thu Jul 20 10:19:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
stefan@Desktop3 ~/winec $ wine memtest.exe 1940
Allocate 1940MB Ram
Done. Press ENTER to exit
stefan@Desktop3 ~/winec $ wine memtest.exe 1950
Allocate 1950MB Ram
EOutOfMemory: Zu wenig Arbeitsspeicher
```
And under windows 2003 32bit:

```
C:\>ver

Microsoft Windows [Version 5.2.3790]

C:\>memtest 1870
Allocate 1870MB Ram
Done. Press ENTER to exit
C:\>memtest 1880
Allocate 1880MB Ram
EOutOfMemory: Zu wenig Arbeitsspeicher
C:\>
```
Even worse under Windows 7 64bit:

```
C:\>ver

Microsoft Windows [Version 6.1.7601]

C:\>memtest 1810
Allocate 1810MB Ram
Done. Press ENTER to exit
C:\>memtest 1820
Allocate 1820MB Ram
EOutOfMemory: Zu wenig Arbeitsspeicher
C:\>
```

So what is this? Windows itself cannot handle more than 1,8GB per application?
Finally i found out: the 4GB address space is divided into 2GB for the application (minus some headers) and 2GB for the kernel image.
see https://msdn.microsoft.com/en-us/library/windows/desktop/aa366912(v=vs.85).aspx


----------



## Adrien2002 (Aug 28, 2017)

Do you have a Core i7 too ?

Every Core i7 I tried today were limiting the 32 bits to 2 GB but every Core i5 I tried were not. It was as it should be (more than +3 GB).
I remember that Bethesda made a patch, back in time, for Skyrim to allow the game to use more than 2 GB of memory on Windows (it was unusual at that time to have such big games) so I suppose it means it is normal to have more than 2 GB in 32 bits, no ? Maybe the Core i7 are just doing wrong ?


----------



## Ordoban (Aug 28, 2017)

Yes, it's an older i7. But I think it's not the fault of i7. It is one Windows tuneable. See https://msdn.microsoft.com/en-us/library/windows/desktop/bb613473(v=vs.85).aspx and https://msdn.microsoft.com/en-us/library/windows/desktop/aa366912(v=vs.85).aspx for more info.
Maybe this tuneable is enabled by default on i5. I will try this out after work.

The more interesting question is: Does this kind of tuneable also exist in wine?


----------



## Ordoban (Aug 28, 2017)

Jea I've learned many new:
- If an 32bit windows application needs more than 2GB ram it must have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag. (Added to my test program)
- We cannot get all the memory in one piece. Me test program needs to split the memory allocation in small chunks.
- the /3GB flag works on Win2003, let me get 2992MB.
- on Wondows 7 (64bit) this is not required, let me get 3964MB.
- on WINE (Linux Mint 64bit) I can get 3944MB.
- on WINE (FreeBSD 64bit) I can get 2988MB. The wine version is the "i386-wine-2.0.1" package.


----------



## Adrien2002 (Aug 19, 2018)

Ordoban said:


> Jea I've learned many new:
> - If an 32bit windows application needs more than 2GB ram it must have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag. (Added to my test program)
> - We cannot get all the memory in one piece. Me test program needs to split the memory allocation in small chunks.
> - the /3GB flag works on Win2003, let me get 2992MB.
> ...



That 1GB missing is a real problem on big games. It causes crashes when GNU/Linux won't. Why is it like that ? PCSX2 is not an usable emulator on FreeBSD because of that issue and so many games are crashing because of a "can't allocate memory" issue. That's annoying !


----------



## shkhln (Aug 19, 2018)

DOOM 3 / Quake 4 were released when a typical desktop computer contained somewhere around 512 MB or 1 GB RAM, so even 2 GB per process should be plenty.


----------



## Crivens (Aug 19, 2018)

Question: does wine die when it is allowed more than 2GB? Or does it die when it does not get more memory? Say you ulimit wine to 2GB, so it can not allocate more? Is someone using signed addresses?


----------



## kpa (Aug 19, 2018)

Running windows games on a FreeBSD system using Wine makes roughly 0.1% or less of FreeBSD use cases so it's no wonder that it's not a priority at all in the FreeBSD community. I'm at a loss of thinking a situation where you must use FreeBSD to run your windows games and there are no other reasonable solutions available.


----------



## shkhln (Aug 19, 2018)

Actually, I think I'm familiar with this "can't allocate memory" message. It usually looks like (excerpt):


```
wine: Unhandled page fault on read access to 0x00000000 at address 0x623427ed (thread 0126), starting debugger...
Fontconfig warning: "local.conf", line 1093: saw number, expected matrix
Fontconfig warning: "local.conf", line 1093: saw number, expected matrix
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
wineserver: file_set_error() can't map error: Cannot allocate memory
```

It has nothing to do with crash per se, it just follows the actual crash reason. Not sure what wine tries to do there.


----------



## shkhln (Aug 19, 2018)

kpa said:


> I'm at a loss of thinking a situation where you must use FreeBSD to run your windows games and there are no other reasonable solutions available.



There is nothing reasonable about FreeBSD desktop anyway.


----------



## Ordoban (Aug 19, 2018)

For such "high performance" games this memory issue seems just the *first* issue.
Last time I tried playing Lineage II on FreeBSD9.0+Wine the 3D performance was very bad.
This game runs pretty well on Linux+Wine, so I use Linux for desktop.
By the way, the server of this game (imoriath.com) runs very well on FreeBSD. 
So: *BIG THANKS* to the FreeBSD community!


----------



## Phishfry (Aug 19, 2018)

shkhln said:


> There is nothing reasonable about FreeBSD desktop anyway.


I am editing video on Xfce4 for a website right now with multimedia/kdenlive
Nothing unreasonable about it at all. It is what you make of it. I hated all the kde dependancies but I am using a 64GB SSD.
No shortage of space as my FreeBSD desktop install only takes around 4 Gigabytes.
Everyday I find more useful software for both the desktop and command prompt.
Who needs bluetooth? The little things missing aren't a big deal. Otherwise we would have them.
I would like to have AutoCAD 32bit running under Wine. I just haven't found the time. Plus I have a Windows box for that task.


----------



## Adrien2002 (Aug 19, 2018)

shkhln said:


> There is nothing reasonable about FreeBSD desktop anyway.



It's more about mixing passions together than for a real need. I still have my Windows on the side to play video games but I'm just so curious about to run video games on FreeBSD and the performances on my machine are really good ! FreeBSD is way better than what people think when speaking of video games. Or is it because I'm lucky and have a well supported hardware... Can we still say that in 2018 with FreeBSD ?


----------



## phoenix (Aug 20, 2018)

A 32-bit process can access 4 GB of virtual memory map. However, that map is then split into 2 pieces: 2 GB for the kernel, 2 GB for the process itself. It's the same setup on just about every PC-based OS (Windows, Linux, MacOS X, the BSDs, etc).

If a process needs to access more than 2 GB of the virtual memory map, then it needs to resort to various tricks like PAE, memory thunking, bounce buffers, or reconfiguring the kernel/user split (aka large memory support).

The first and last options are things the user can do. The others are things only the program devs can do.

Not sure how to change the kernel/user split in FreeBSD, but it is possible to use a 1/3 split instead of the standard 2/2 split. As you discovered, it's possible to add a flag to the Windows boot loader to change this.


----------



## Crivens (Aug 20, 2018)

phoenix - sorry, but that is not entirely correct if my memory serves me right.

MacOS gives you almost 4GB, the "kernel" mapped in is a 64KB address map switcher. 

PAE can only allow the kernel to have MMU tables which contain more than 4GB. One process tops out at an address range of 4 GB no matter what, but you can now have several of these on your system. I would be happ to be corrected on this one.


----------



## Ordoban (Aug 20, 2018)

So there are 2 operation modes for 32bit applications in 64bit OS:
- 2G/2G or 1G/3G split with "real" kernel in address space
- 4194240K/64K split with address map switcher

This leads to a question:
What is the memory limit for native 32bit FreeBSD applications?
What split model uses FreeBSD for its 32bit compatibility mode?

Wine is nothing else than an application who offers its "guest" the windows API's and translates the API calls to the native kernel.
So wine is bound to the OS limitations too.


----------



## Crivens (Aug 20, 2018)

The 64KB switch code is special to how Mach (and thus MacOS X) go about things. You can not configure that on FreeBSD or any Unix I know of.


----------



## Ordoban (Aug 20, 2018)

I have written some max memory values earlier in this thread.
Based on this values I guess both linux and windows 64bit doing some similar thing.

Don't get me wrong. I don't think FreeBSD should have such mechanism to squeeze the last byte out of the 32bit memory model.
If an application needs huge amounts of memory - it should go to 64bit.


----------



## phoenix (Aug 22, 2018)

Ordoban said:


> So there are 2 operation modes for 32bit applications in 64bit OS:
> - 2G/2G or 1G/3G split with "real" kernel in address space
> - 4194240K/64K split with address map switcher
> 
> ...



Not sure about 32-bit apps on 64-bit OS (would assume it's the same as for a 32-bit OS, but you know what they say about assumptions), but a 32-bit install of FreeBSD uses a 2G/2G mapping.  There's been talk on the mailing lists this year about switching to a 1G/3G mapping.  Not sure if that change was ever made or not.


----------



## shkhln (Jan 13, 2019)

phoenix said:


> 32-bit install of FreeBSD uses a 2G/2G mapping. There's been talk on the mailing lists this year about switching to a 1G/3G mapping.  Not sure if that change was ever made or not.



Looks like https://reviews.freebsd.org/rS332489. Curiously enough, I don't see it in 12.0 release notes.


----------



## olli@ (Jan 13, 2019)

shkhln said:


> Looks like https://reviews.freebsd.org/rS332489. Curiously enough, I don't see it in 12.0 release notes.


That change (plus a bunch of follow-up fixes) was committed before _current_ was made into _stable/12_, so it clearly is part of FreeBSD 12.
It was probably just an oversight not to include it in the release notes. It also affects only very few users, I guess.


----------



## shkhln (Jan 13, 2019)

Part of i386 FreeBSD to be precise. I've actually found a few 32-bit games running into address space exhaustion on FreeBSD amd64 with wined3d roughly at 3.3-3.5 Gb allocated virtual memory (according to htop), which is both annoying and amusing. The program from https://unix.stackexchange.com/questions/388156/how-can-i-enable-4g-4g-split-in-linux/418551#418551 prints 3556352000. Anybody want to check this on i386?


----------



## shkhln (Jan 24, 2019)

shkhln said:


> The program from https://unix.stackexchange.com/questions/388156/how-can-i-enable-4g-4g-split-in-linux/418551#418551 prints 3556352000.



This actually doesn't include RLIMIT_DATA (rather generous 512 Mb by default), RLIMIT_STACK and jemalloc's accounting overhead. Overall it's pretty close to 4 Gb (on FreeBSD amd64).


----------



## shkhln (Jan 24, 2019)

Ordoban said:


> - on WINE (Linux Mint 64bit) I can get 3944MB.



How did you manage to get this number? Wine _itself_ limits applications to 2 or 3 Gb depending on LARGE_ADDRESS_AWARE flag.


----------



## Ordoban (Jan 25, 2019)

shkhln said:


> How did you manage to get this number? Wine _itself_ limits applications to 2 or 3 Gb depending on LARGE_ADDRESS_AWARE flag.


I just run the test program who allocates memory until it gets an "out of memory" exception.
The limit of 2 or 3 or 4 Gb depends on set compiler switches:

```
#ifdef __i386__
...
static void *address_space_limit = (void *)0xc0000000;  /* top of the total available address space */
static void *user_space_limit    = (void *)0x7fff0000;  /* top of the user address space */
static void *working_set_limit   = (void *)0x7fff0000;  /* top of the current working set */
...
#elif defined(__x86_64__)
...
static void *address_space_limit = (void *)0x7fffffff0000;
static void *user_space_limit    = (void *)0x7fffffff0000;
static void *working_set_limit   = (void *)0x7fffffff0000;
...
#else
...
static void *address_space_limit;
static void *user_space_limit;
static void *working_set_limit;
...
#endif  /* __i386__ */
...

...
#if !defined(__i386__) && !defined(__x86_64__)
...
#ifdef _WIN64
    address_space_limit = (void *)(((1UL << 47) - 1) & ~page_mask);
#else
    address_space_limit = (void *)~page_mask;
#endif
    user_space_limit = working_set_limit = address_space_limit;
#endif
```
I do not know what switches were set during the wine compilation.

0xc0000000 = 3Gb if set __i386__  (wine32 on 32 bit OS ?)
0x7fffffff0000 = 128Tb if set __x86_64__ (wine64 on 64 bit OS ?)
~page_mask = calculated from current memory model (maybe 4Gb), if none switch set. (wineXX on 64 bit OS ?)


----------



## Crivens (Jan 25, 2019)

To clearify: that is about an i386 binary on an amd64 host, yes?


----------



## shkhln (Jan 25, 2019)

Crivens said:


> To clearify: that is about an i386 binary on an amd64 host, yes?



Yes, nothing there makes sense otherwise.



Ordoban said:


> I do not know what switches were set during the wine compilation.



I believe -m32 compiler option also defines __i386__, so `address_space_limit = (void *)~page_mask` line isn't relevant for 32-bit Wine apps on amd64. I don't know what it is supposed to do, that would require some digging through commit history. Presumably it will be compiled on ARM, but I think it actually predates Wine's ARM port.


----------



## Crivens (Jan 25, 2019)

Wine in particular has the problem of emulating some of the windows process memory layout, so the GB must have holes for wine and reserved areas anyway, right?


----------



## shkhln (Jan 25, 2019)

I'm not quite sure whether Wine reserves anything for heap immediately on start. One thing I noticed is that recompiling Wine with user_space_limit set to 3 Gb reduced allocated virtual memory by roughly 1 Gb for two problematic applications (I complained earlier in this thread about them), while actual memory usage is unaffected. Didn't expect that at all.

Note also that Wine's heap is a distinct entity from jemalloc's heap, so there is some possibility for pathological interactions. Most likely it is just Wine misbehaving though.


----------



## Ordoban (Jan 25, 2019)

Crivens said:


> To clearify: that is about an i386 binary on an amd64 host, yes?


Yes, 32bit Windows binary on amd64 Linux system. Is wine itself a 32 or 64 bit binary? I don't know! I am also not remember the Wine version.


shkhln said:


> Yes, nothing there makes sense otherwise.
> I believe -m32 compiler option also defines __i386__, so `address_space_limit = (void *)~page_mask` line isn't relevant for 32-bit Wine apps on amd64.


That's the drawback of precompiled binary packages. It's hard to say what options the packager used.


shkhln said:


> I don't know what it is supposed to do, that would require some digging through commit history. Presumably it will be compiled on ARM, but I think it actually predates Wine's ARM port.


Wine on ARM? Sounds funny. Since there is no real Windows on ARM (except Win10 for RPi3), there are no windows programms for ARM.


Crivens said:


> Wine in particular has the problem of emulating some of the windows process memory layout, so the GB must have holes for wine and reserved areas anyway, right?


Sure. But that's the case on real Windows too. You can't do a `malloc(3*1024*1024*1024);` on real Windows nor on Wine.


shkhln said:


> I'm not quite sure whether Wine reserves anything for heap immediately on start. One thing I noticed is that recompiling Wine with user_space_limit set to 3 Gb reduced allocated virtual memory by roughly 1 Gb for two problematic applications (I complained earlier in this thread about them), while actual memory usage is unaffected. Didn't expect that at all. Note also that Wine's heap is a distinct entity from jemalloc's heap, so there is some possibility for pathological interactions. Most likely it is just Wine misbehaving though.


If you force Wine to 3Gb, it takes 3Gb. I am not surprised. You are talking about virtual memory - is that just address space or allocated real physical memory?


----------



## shkhln (Jan 25, 2019)

Ordoban said:


> You are talking about virtual memory - is that just address space or allocated real physical memory



Address space, the memory which application requested but didn't necessarily try to read or write. In other words, if you request too much memory _mmap_ refuses to work because the system is simply out of valid pointers. It's kinda funny, since it's usually considered a theoretical problem.



Ordoban said:


> If you force Wine to 3Gb, it takes 3Gb. I am not surprised.



You are not paying attention: setting limit to 3 Gb makes Wine process request 2 Gb _instead_ of 3-3.5 Gb on start with two separate games. Actual memory usage never exceeds 1.5 Gb, so there is no reason to think they require LARGE_ADDRESS_AWARE flag and, of course, it's not set out-of-the-box — it's not an issue on Windows.


----------



## Ordoban (Jan 25, 2019)

shkhln said:


> Address space, the memory which application requested but didn't necessarily try to read or write. In other words, if you request too much memory _mmap_ refuses to work because the system is simply out of valid pointers. It's kinda funny, since it's usually considered a theoretical problem.


Do I understand right? Wine gets the full 4Gb address space from host for later ram usage, and if the guest need address space for everything else (maybe memory mapped files) this fails, because wine can not get more address space.
Sounds strange!


shkhln said:


> You are not paying attention: setting limit to 3 Gb makes Wine process request 2 Gb _instead_ of 3-3.5 Gb on start with two separate games. Actual memory usage never exceeds 1.5 Gb, so there is no reason to think they require LARGE_ADDRESS_AWARE flag and, of course, it's not set out-of-the-box — it's not an issue on Windows.


Sorry if I misunderstand you. It's a heavy topic.


----------



## shkhln (Jan 25, 2019)

Ordoban said:


> Wine gets the full 4Gb address space from host for later ram usage, and if the guest need address space for everything else (maybe memory mapped files) this fails, because wine can not get more address space.



The issue is that some Wine component, most likely wined3d, requests much more memory than it actually cares to use to the point it prevents OpenGL driver from allocating memory for its own needs (like loading textures or compiling shaders). The driver eventually returns GL_OUT_OF_MEMORY error and crashes with nonsensical stack trace. _Why_ this happens is a whole other question. I can't offer you any explanation because I do not know that in the first place.


----------



## shkhln (Jan 25, 2019)

shkhln said:


> One thing I noticed is that recompiling Wine with user_space_limit set to 3 Gb reduced allocated virtual memory by roughly 1 Gb



Looks like it's a relatively well known workaround: https://bugs.winehq.org/show_bug.cgi?id=34658#c24, https://bugs.winehq.org/show_bug.cgi?id=45045#c11.


----------

