# Microsoft Strikes Again: Memory compression in Win 10. Useful in FreeBSD?



## Maelstorm (Oct 7, 2015)

Read this to get a handle on what Microsoft has announced:

http://www.tenforums.com/windows-10-news/17993-windows-10-memory-compression.html

Now, being a system programmer type myself, I understand how this is implemented.  System maintains two memory lists.  When a page hasn't been used for awhile, it gets compressed and moved off the active list onto the inactive list and is marked swapped out.  If the program needs it, a page fault occurs which causes the swapper to check the inactive list, uncompress the page, and give it to the program.  If the page has been moved off the inactive list, then it has been swapped to disk so swapper reads it from disk, uncompresses it, and then gives it to the program.  The only big difficulty is the compression algorithm.  Microsoft claims that their algorithm is fast and compresses on average 40%.  This means that on average, 5 pages can fit into 2, depending on the content of the data.

How useful would something like this be for FreeBSD and would we be able to implement it (given legal considerations)?


----------



## Beastie7 (Oct 7, 2015)

There's a wishlist on the FreeBSD Ideas page of taking OS X's implementation of memory compression using the WKdm algorithm. Don't know the status on it, but it appears no one has attempted to do this yet. The only thing I can think of as a benefit is to further maximize memory density for virtual machines after a motherboard is maxed out.


----------



## protocelt (Oct 7, 2015)

I could see this being useful in FreeBSD. One thing that comes to mind is in-memory databases.


----------



## SirDice (Oct 7, 2015)

Didn't they try something like that before? Ram doubler on DOS/Windows 3.1. As I recall it introduced more problems than it tried to solve. When memory got a lot cheaper people just stuck more memory in the machine. And with Gigabytes of RAM of these days I don't really see a need for it. Maybe it could be useful on small, embedded, systems where memory is scarce. But on big high-end servers and desktops, would it really be useful?


----------



## cpm@ (Oct 7, 2015)

That sounds similar to zswap for me.

See also: https://www.reddit.com/r/sysadmin/comments/3lkoph/memory_compression_in_windows_10_or_how_windows/


----------



## drhowarddrfine (Oct 7, 2015)

That sounds like a lot of work for a machine to be doing; and it would be doing it constantly. I haven't dealt with system programming in many years but there have been lots of methods for dealing with this sort of thing and I'm sure FreeBSD figured out the best way long ago.


----------



## lme@ (Oct 11, 2015)

There was a GSoC project for implementing memory compression on FreeBSD: https://wiki.freebsd.org/SummerOfCode2015/MemoryCompressionDeduplication


----------



## Maelstorm (Oct 12, 2015)

From what I understand in the article, Microsoft's reasoning for this is that it is cheaper (and faster) to keep a compressed page in memory and decompress it on the fly when it is needed than having to read it from swap.  With the CPU clock speeds I see today and with multiple cores, memory compression management may not add that much overhead to the system.  Pages are compressed only when more memory is needed, so it is only invoked if malloc can't find enough RAM to fill the request.  Pages and entire processes are swapped out when even more memory is needed.

At least that's the thinking that I got from reading the article.

EDIT:

As with lists, you can get high performance lookups using either hash tables or binary search.  So the machine is not doing as much work as one might think.


----------



## rootwyrm (Dec 5, 2015)

gpatrick said:


> AIX from 6.1 TL4 has offered Active Memory Expansion. It has been around in AIX since at least 2010, if not before.



Here's the thing though - Active Memory Expansion is borderline black magic. It's not simple compression, there's hardware involvement, and it has some pretty steep costs. To the point where IBM actually provides several tools (e.g. amepat) to analyze your system to see if AME will actually buy you anything.

In many cases, the answer from AME is that you'll get maybe 5-10% but it'll cost you 10-15% CPU to get there. The CPU utilization got better when the memory controller path got optimized compression routines. But you won't see AME being run most places. Usually large WebSphere setups where depending on the app, it might buy you 70%. For stuff like Oracle, I usually see no real gain. 
Also, AMS plays a part. AMS is a P6 and later feature that basically allows overprovisioning of partition memory from a pool. So you can say, have 4 LPARs each given 128GB from a total physical pool of 256GB. As long as they aren't all actually using 128GB at once, that's a 100% gain basically.

Here's the thing though - AMS + AME works because you're talking about a fundamentally different architecture. No I don't mean x86 vs POWER, I mean whole-system versus partitioned. If all my LPARs are 128GB and I'm saving 25% with AME per LPAR, that means I can resize to get 10 LPARs and 4 VIOS in a 1TB system instead of 7. With AMS I can potentially increase that to 12-14. In contrast, on FreeBSD with 128GB, it just means I get higher effective system memory. 

I'm not saying it'd necessarily be a bad thing to have on FreeBSD - but I don't see it being a significant benefit for most users.


----------



## RedShift1 (Jan 14, 2016)

Linux has Kernel Same-page Merging (KSM), which merges duplicate memory pages into one COW page to save memory.


----------



## grahamperrin@ (May 25, 2016)

lme@ said:


> … GSoC project … https://wiki.freebsd.org/SummerOfCode2015/MemoryCompressionDeduplication



Thanks, that _Memory compression and deduplication_ project page refers to a repository https://reviews.freebsd.org/diffusion/DEFGSOC/ that is no longer found, and a review https://reviews.freebsd.org/project/profile/52/ entitled _memory deduplication_.

Please:

can anyone tell whether that review encompasses both deduplication _and_ compression?


----------



## Crivens (May 25, 2016)

I use swap on a compressed zvol. When memory has to go out, it is compressed by zfs. Since the compressed pages should take up less space than the original area, there is suddenly memory free again, so the pressure to actually write out the compressed pages is reduced. This is a poor mans version of this memory compression, I know. It would be interesting to see what the GSOC project resulted in and how it compares to the simple solution.

Also, doesn't apple do this compression for some time now?


----------



## grahamperrin@ (May 29, 2016)

In-kernel memory compression [LWN.net] (2013-04)



Beastie7 said:


> There's a wishlist on the FreeBSD Ideas page of taking OS X's implementation of memory compression using the WKdm algorithm. Don't know the status …



It would be good to know … I'll ask in IRC.



Crivens said:


> … doesn't apple do this compression for some time now?



June 2013: Apple - OS X Mavericks - Advanced Technologies

WebKit bug 120143 https://bugs.webkit.org/show_activity.cgi?id=120143 (resolved 2013-08-23) was _Support in memory compression of rarely used data_.

October 2013: OS X Mavericks Core Technologies Overview

osx - vm_compressor_mode (vm.compressor_mode) values for enabled compressed memory in OS X - Ask Different – includes links to *vm_compressor.c*, *vm_fault.c* and *vm_pageout.h* in *xnu-2422.1.72* in Apple open source for OS X 10.9. 

https://opensource.apple.com/source/gnuzip/gnuzip-31/gzip/bits.c

```
…
*  PURPOSE
*
*      Output variable-length bit strings. Compression can be done
*      to a file or to memory. (The latter is not supported in this version.)
…
```
https://opensource.apple.com/source/gnuzip/gnuzip-31/gzip/TODO

```
… 
- Support in-memory compression for arbitrarily large amounts of data
  (zip currently supports in-memory compression only for a single buffer.) 
…
```
http://opensource.apple.com/source/gcc_os/gcc_os-1823/zlib/zlib.h

```
…
/*
     The 'zlib' compression library provides in-memory compression and
  decompression functions, including integrity checks of the uncompressed
  data.  This version of the library supports only one compression method
  (deflation) but other algorithms will be added later and will have the same
  stream interface.

     Compression can be done in a single step if the buffers are large
  enough (for example if an input file is mmap'ed), or can be done by
  repeated calls of the compression function.  In the latter case, the
  application must provide more input and/or consume the output
  (providing more output space) before each call.

     The library also supports reading and writing files in gzip (.gz) format
  with an interface similar to that of stdio.

     The library does not install any signal handler. The decoder checks
  the consistency of the compressed data, so the library should never
  crash even in case of corrupted input.
*/
…
```


----------



## Matty (Apr 4, 2019)

So what did become of it? 
There was a https://wiki.freebsd.org/SummerOfCode2015/MemoryCompressionDeduplication but it seems this never produced any code.

To be honest I think it's still a viable idea given that all other major OS have it for years


----------



## Sevendogsbsd (Apr 4, 2019)

Is it that FreeBSD is more efficient with memory and doesn't need it, unlike the other OSes? That's probably a stupid question from a non-programmer  . I am with SirDice though: I buy as much ram as I can afford to stuff in a box so I never run out. Then again I have a workstation so never run out anyway...


----------



## rufwoof (Apr 4, 2019)

Assuming 50% reduction, fast (de)compression works well with slower mechanical disks. Where the time to (write)read half as much data and (de)compress can be significantly faster than just reading 100% of standard data. Squashed filesystems access can be much quicker than conventional, at least for mechanical disks. For faster/solid state disks however the benefit can narrow right down. For memory, in a low cost memory era however the benefits of memory compression, if any, just gut feels very narrow.


----------



## Crivens (Apr 5, 2019)

Just put your swap on a compressed zvol.


----------



## grahamperrin@ (Apr 5, 2019)

[GSOC idea] virtual memory compression


----------



## pave (Apr 10, 2019)

grahamperrin said:


> [GSOC idea] virtual memory compression



That's me. Here's the final application.



Crivens said:


> Just put your swap on a compressed zvol.


There's no benefit if the page size and disk's block size are both 4kb.


----------



## Wozzeck.Live (Apr 10, 2019)

Sevendogsbsd said:


> Is it that FreeBSD is more efficient with memory and doesn't need it, unlike the other OSes? That's probably a stupid question from a non-programmer  . I am with SirDice though: I buy as much ram as I can afford to stuff in a box so I never run out. Then again I have a workstation so never run out anyway...



YES
Ram memory under windows IS CRITICAL because this is A VORACIOUS os.
This doesn't mean Microsoft if faulty.... I don't like troll. This is linked to the fact that modern Windows 10 Desktop have many more services running compared to his ancestors. The graphic interface use widely the hardware acceleration.

As today, Windows 10 64 bits needs at a minimum 6 GB of RAM. It can run with less memory, but this generally sollicitate the swap drive very much. And in fact, as many Linux distro tend to be a "destktop" alternative to Windows, modern linuces have the same problem. Unbuntu 64 bit is not known to be very fast, but if you run it with only 2 GB of RAM.......

This proves that Microsoft isn't faulty. This is simply linked to moderns desktop : people want beautiful desktop and visual effects.... it needs power, people want a lot of services...

The memory management of FreeBSD is known to be the one the best of the industry.
FreeBSD amd64 can run with just 512 MB of RAM. Impossible for modern Windows and most Linux distros (or we should revert to Windows 2000/XP).

I wouldn't say that memory compression is not useful.... I would say that many other things are far more important :
- updating smbfs client to SMBv2 + protocol
- developping a REAL support of bluetooth which is now widlely used for Mouse, Keyboard, Speakers, Phones. Modern laptops have very few usb ports, so connecting devices through bluetooth becomes the prefered solution in order to keep free at less one usb port
- More drivers for WIFi. This is critical. FreeBSD is known to have one of the best IP layer, but support of WifI card and Wifi protocol is very poor so you can't use FreeBSD to build wireless routers... you must use Linux. (basic support of Wifi AC just came out with FreeBSD 12.... it was already supported by Linux since a long time).
I think FreeBSD makes a mistake not considering Wifi as something urgent because it misses a big market.... those integrating wireless routing in various appliances.


----------



## hukadan (Apr 10, 2019)

Wozzeck.Live said:


> As today, Windows 10 64 bits needs at a minimum 6 GB of RAM.


As I type this message, my Windows 64 with Iridium open uses less than 3Go out of the 16Go available.


----------



## Vull (Apr 10, 2019)

Typing this message, I'm using 903 MiB out of the 3GiB RAM and ~6GiB swap available on my 32-bit Dell Dimension mini-tower, with FreeBSD 11.2-RELEASE running Firefox on a customized plasma5-plasma desktop. Konsole, top, Apache server, and PostgreSQL server are also running concurrently:
	
	



```
(del@dellfp27 /home/del)# top -n all
last pid:  2477;  load averages:  0.39,  0.40,  0.34  up 0+03:27:08    17:05:49
71 processes:  1 running, 70 sleeping

Mem: 513M Active, 390M Inact, 12M Laundry, 113M Wired, 67M Buf, 1956M Free
Swap: 5967M Total, 5967M Free


  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1983 del           3  21    0   151M    99M select  1   0:58   3.47% Xorg
 2346 del           5  20    0   176M    98M select  0   0:08   1.76% konsole
 2043 del           5  20    0   193M 66260K select  1   0:05   0.20% kwin_x11
 2057 del          46  20    0   677M   346M select  1   0:41   0.00% firefox
 2047 del          12  20    0   334M   206M select  1   0:26   0.00% plasmashell
 2073 del          16  20    0   495M   211M select  1   0:26   0.00% firefox
 2255 del          15  20    0   404M   121M select  0   0:04   0.00% firefox
 2045 del           7  20    0   203M 94964K select  0   0:03   0.00% krunner
 2020 del           6  20    0   121M 52892K select  1   0:01   0.00% kdeinit5
 2041 del           4  20    0   110M 47316K select  0   0:01   0.00% ksmserver
 2031 del           3  20    0   103M 42576K select  1   0:01   0.00% kglobalaccel5
 2019 del           4  20    0   102M 41948K select  0   0:01   0.00% klauncher
  385 root          1  20    0  6140K  1888K select  0   0:01   0.00% moused
 2059 del           6  20    0 99932K 45032K select  0   0:01   0.00% org_kde_powerdevil
  739 pgsql         1  20    0 15104K  7420K select  1   0:01   0.00% postgres
  733 pgsql         1  20    0   154M 16452K select  1   0:01   0.00% postgres
 2023 del           3  20    0   101M 41840K select  1   0:01   0.00% kaccess
 2051 del           5  20    0   107M 44152K select  1   0:01   0.00% polkit-kde-authenti
 2165 del           3  20    0   102M 47968K select  0   0:01   0.00% kuiserver5
 2000 del           1  20    0  6724K  2912K select  0   0:00   0.00% dbus-daemon
 2461 del          14  20    0   384M   105M select  0   0:00   0.00% firefox
 2062 del           7  20    0 90420K 33676K select  1   0:00   0.00% kactivitymanagerd
 1992 polkitd      10  20    0   198M 19508K select  0   0:00   0.00% polkitd
  753 root          1  20    0 22816K 13804K select  0   0:00   0.00% httpd
  738 pgsql         1  20    0   154M 16896K select  1   0:00   0.00% postgres
 2018 del           1  20    0 84000K 35896K select  0   0:00   0.00% kdeinit5
  655 messagebus    1  20    0  6724K  2976K select  1   0:00   0.00% dbus-daemon
 1990 root         15  20    0 27048K  6336K select  0   0:00   0.00% console-kit-daemon
 2033 del           3  20    0 63940K 21596K select  1   0:00   0.00% kscreen_backend_lau
 2053 del           3  20    0 69688K 23700K select  1   0:00   0.00% xembedsniproxy
  856 www           1  31    0 24864K 15420K lockf   1   0:00   0.00% httpd
 2055 del           4  20    0 72748K 24636K select  0   0:00   0.00% gmenudbusmenuproxy
 2070 del           4  20    0 20424K  6532K select  1   0:00   0.00% at-spi-bus-launcher
 2037 root          3  20    0 17400K  5824K select  0   0:00   0.00% upowerd
  737 pgsql         1  20    0   154M 16500K select  1   0:00   0.00% postgres
  736 pgsql         1  20    0   154M 16456K select  0   0:00   0.00% postgres
  759 root          1  20    0  6008K  1932K nanslp  1   0:00   0.00% cron
  960 root          1  20    0  6484K  2368K wait    0   0:00   0.00% login
 2412 del           1  20    0  6484K  2360K wait    0   0:00   0.00% su
  520 root          1  20    0  5980K  1876K select  0   0:00   0.00% syslogd
 2348 del           1  52    0  6500K  2372K wait    1   0:00   0.00% login
 2167 del           1  20    0 11416K  6184K select  0   0:00   0.00% gconfd-2
 2040 del           1  24    0 47436K 10524K sbwait  0   0:00   0.00% kwrapper5
 2035 del           3  24    0 14676K  5148K select  0   0:00   0.00% dconf-service
 2350 del           1  20    0  6456K  2400K wait    1   0:00   0.00% sh
 1996 del           1  23    0  6456K  2268K wait    0   0:00   0.00% sh
  735 pgsql         1  20    0   154M 16836K select  0   0:00   0.00% postgres
 1909 del           1  52    0  6456K  2368K wait    0   0:00   0.00% sh
 1986 del           1  29    0  9028K  3252K wait    0   0:00   0.00% ck-launch-session
 2414 root          1  20    0  6456K  2380K wait    0   0:00   0.00% sh
  398 root          1  20    0  6944K  3812K select  1   0:00   0.00% devd
 1969 del           1  52    0  6456K  2248K wait    1   0:00   0.00% sh
 1982 del           1  30    0  7300K  2428K wait    0   0:00   0.00% xinit
  959 root          1  52    0  5960K  1740K ttyin   0   0:00   0.00% getty
  958 root          1  52    0  5960K  1740K ttyin   1   0:00   0.00% getty
  956 root          1  52    0  5960K  1740K ttyin   0   0:00   0.00% getty
 2477 root          1  20    0  7340K  2532K CPU1    1   0:00   0.00% top
  955 root          1  52    0  5960K  1740K ttyin   1   0:00   0.00% getty
  957 root          1  52    0  5960K  1740K ttyin   0   0:00   0.00% getty
  954 root          1  52    0  5960K  1740K ttyin   1   0:00   0.00% getty
  857 www           1  20    0 22816K 13860K kqread  0   0:00   0.00% httpd
  318 root          1  52    0  6084K  1936K select  0   0:00   0.00% dhclient
  727 root          1  20    0 10820K  5584K select  1   0:00   0.00% sshd
  953 root          1  52    0  5960K  1740K ttyin   1   0:00   0.00% getty
  767 root          1  52    0  6776K  2528K select  0   0:00   0.00% ftpd
  863 www           1  52    0 22816K 13860K lockf   0   0:00   0.00% httpd
  858 www           1  52    0 22816K 13860K lockf   0   0:00   0.00% httpd
  861 www           1  52    0 22816K 13860K lockf   1   0:00   0.00% httpd
 2075 www           1  20    0 22816K 13860K lockf   0   0:00   0.00% httpd
 1999 del           1  20    0  7832K  3104K select  0   0:00   0.00% dbus-launch
  380 _dhcp         1  20    0  6084K  2000K select  1   0:00   0.00% dhclient
```


----------



## Spartrekus (Apr 14, 2019)

Wozzeck.Live said:


> YES
> Ram memory under windows IS CRITICAL because this is A VORACIOUS os.
> This doesn't mean Microsoft if faulty.... I don't like troll. This is linked to the fact that modern Windows 10 Desktop have many more services running compared to his ancestors. The graphic interface use widely the hardware acceleration.
> 
> ...



This is because FreeBSD avoids bloatware as good as possible, when other computer science areas have rapid, non hardware resource caring, software development.

Using Microsoft programming environments and having an IDE mean to not readily care of simple programmnig solutions.. but also to be willing to have a sort of magic, beautiful, animated, resource heavy software - using a mouse.


----------



## D-FENS (Apr 14, 2019)

Spartrekus said:


> This is because FreeBSD avoids bloatware as good as possible, when other computer science areas have rapid, non hardware resource caring, software development.
> 
> Using Microsoft programming environments and having an IDE mean to not readily care of simple programmnig solutions.. but also to be willing to have a sort of magic, beautiful, animated, resource heavy software - using a mouse.


Yeah, the kernel and the base system should stay as slim as possible. The rest should be in user packages. Of course, you could bloat your FreeBSD system if you want to. (You know what? I think it would be quite funny to have a machine with all possible packages installed. I wonder how much disk space it would take? And how about enabling all possible services at once? I need to try this!)

The problem with softwares from some other vendors is that they think they know better than you and then they install all kinds of weird programs by default. Like Cloud accounts, search engines, browsers full of preconfigured favorites, etc.


----------

