# Understanding memory management



## Korger (Apr 6, 2022)

Hi Everyone,

I'd like to learn more about how FreeBSD manages memory, specifically about the types of memory that *top* shows.  I've read a few articles on the topic, also Chapter 7 in the Architecture Handbook.  The following is my current understanding on the subject.  Please clarify if I got something wrong:



> FreeBSD manages memory in 4kB units called pages.  These pages exist either in RAM or on the swap device.  Each page can be in one of the following states, showed in the output of top:
> 
> 
> 
> WiredPages that must be kept in RAM at all times.  Kernel memory is always wired.    ActiveAllocated memory which has been accessed recently by a process.  Since a recent access makes future accesses more likely, the memory manager tries to keep Active pages in RAM, but they may be moved out to swap when RAM is filling up.  Doing so will likely incur heavy performance penalties.    InactiveAllocated memory which has not been accessed recently by any process.  These pages may be moved out to swap, and the expected cost is much lower than for active pages.    BuffersPages that represent the content of a file on the hard disk or other non-volatile storage device.  This redundant information exists only to speed up filesystem access if enough RAM is available.  When RAM becomes scarce, then buffers can be deallocated any time, therefore they are regarded as free memory.    ARCARC replaces Buffers for systems using ZFS.  The *Total* field shows the number of wired bytes in ARC.  *MRU* stands for _Most Recently Used_, *MFU* stands for _Most Frequently Used_ data. There's also the *Anon*, *Header* and *Other* fields, which I have no clue about.LaundryPages assigned to a file whose content has been modified since it was read from the disk, or it has not been stored on the disk yet. Laundry pages must be written back to the filesystem before they can be deallocated.  They may count as free memory, though their availability may depend on how quickly they can be backed up.    FreeFree memory.





> This description implies that determining which memory pages are considered free at any time is not that straightforward: inactive pages are allocated, but they may be moved out to swap, whereas laundry pages should become free soon, but they have to be written back to disk first.  In general, we should regard Wired, Active, Inactive and ARC pages as used memory, whereas Buffers, Laundry and Free comprise free memory.
> 
> Top also shows the allocated memory for each process.  RES shows the number of bytes which is currently present in physical RAM on behalf of the process.  SIZE shows the total memory size that the process is able to address.  This memory does not need to reside in physical RAM, and it can be shared among different processes; for example shared libraries are counted for each process that links to them.



Does this summary line up with reality?  I'm particularly concerned about Buffers and ARC.  My understanding is that Buffers and Laundry belong to the filesystem cache under UFS, both backed by vnodes; Laundry being pages whose content have changed in memory.  But on my ZFS system I still have Laundry in addition to ARC.  If ARC replaced Buffers, what is Laundry doing?  

Please make any comments/corrections to the above.  Thanks!


----------



## grahamperrin@ (Apr 7, 2022)

Korger said:


> … If ARC replaced Buffers, what is Laundry doing? …



<https://wiki.freebsd.org/Memory> might help, if you haven't already seen it.


----------



## Korger (Apr 7, 2022)

Yes, that's where I got the idea that a Laundry page refers to a sector in the filesystem or the swap, whose content has changed, and it needs to be written back before the page can be repurposed.  I still don't know how this works with ARC, which has seemingly done away with Buffers.

It's also confusing that it writes that inactive pages which are dirty are moved to the tail of the laundry queue.  Based on the Handbook I thought inactive pages don't contain filesystem cache, only memory allocated by processes?  Why would process memory need to be laundered, if it's not tied to the filesystem?


----------



## thindil (Apr 7, 2022)

As far I understand, Laundry is a short term cache for modified data, which will be saved to a filesystem soon. It isn't tied to any filesystem, it is a part of memory management in FreeBSD, not a filesystem. ARC is a long term cache for any data read from a ZFS filesystem. And it is a part of ZFS, not memory management.

By the way, as far I see, `top` command reports that FreeBSD allocate a small buffer (40k) even with ZFS.


----------



## PMc (Apr 9, 2022)

Korger said:


> Yes, that's where I got the idea that a Laundry page refers to a sector in the filesystem or the swap, whose content has changed, and it needs to be written back before the page can be repurposed.  I still don't know how this works with ARC, which has seemingly done away with Buffers.


That is a common misconception. The trick to understand it is:
In Berkeley-style Unix every memory is considered some place in the filesystem, mmap()ed to memory. If the memory get's dirty, it points to the paging area, but the logic doesn't change: a memory page is either clean (then it has a place in the filesystem) or dirty (then it can be put into swapspace). 
What the system does, is move these pages around between different queues to make best use of the memory, meanwhile keeping track of which pages can or should be updated to file or need to be fetched back from file.
And the ARC is just a cache in between.


----------



## Korger (Apr 9, 2022)

Thanks, this is really interesting.  But I'm getting even more confused.  I thought putting pages into swapspace depended on the amount of memory and new allocation requests, and not on some inherent quality of each page, like being "dirty".  So what does "dirty" mean exactly?  What triggers this condition?  Does it have to do anything with running out of free RAM?


----------



## PMc (Apr 10, 2022)

Korger said:


> Thanks, this is really interesting.  But I'm getting even more confused.  I thought putting pages into swapspace depended on the amount of memory and new allocation requests, and not on some inherent quality of each page


Yes and yes. Both is right. The question *if* and *when* pageout happens, depends on available memory. The question *how* and *where* the pageout goes, depends on the page.
Lets get to that:


Korger said:


> , like being "dirty".  So what does "dirty" mean exactly?  What triggers this condition?


Dirty means changed. There are different cases:
If an executable is read from disk, it will usually not be changed in memory, so it does not need to be paged out, and can simply be done away with, as it can be loaded again on demand.
If your program creates a variable, that needs memory, This memory is certainly changed, so it must point to swapspace - and this is clarified beforehead, so when memory gets low, it is already clear where to move the various pages, and -most important- what cost is involved. This doesn't mean that the page might ever be paged out, but the system keeps track on the potential amount of swapspace needed (visible in `sysctl vm.swap_reserved`).
More interesting are other cases: imagine two applications copy the same data-file into memory. As long as they do not change it, the pages are clean and can simply be done away with when memory gets low (and reloaded when the application accesses them again). Since the pages point to the same file space, they do not need to exist twice in memory. But then one application changes the data in memory - now the page has to be copied into a dirty one and a clean one, and the dirty one must point to swapspace. Until the application decides to copy the data back to disk, then the page is clean again.


Korger said:


> Does it have to do anything with running out of free RAM?


Not necessarily. It is just proper housekeeping, and having a contingency plan when when no more ram is available.


----------



## Erichans (Apr 10, 2022)

Memory management is quite a complicated topic. ZFS memory management even more so because ZFS extensively acts and interacts on so many levels of the OS. ZFS is a COW (Copy On Write) filesystem that combines a more or less traditional file system with a volume manager and provides extensive data protection. ZFS acts on part of main memory (=RAM) besides secondary memory (=disks) and, somewhere in between (L2ARC and SLOG).

When you're looking at the separate parts of memory management, a more detailed look at the basics and inner workings of memory management might be useful (without focussing on ZFS). For a good internal understanding of memory management in FreeBSD, I'd suggest you have a look at some of the main resources in hand and where memory management interacts or extends to the file system (secondary memory); that's where UFS (FFS in its origins) come into play. The book _The Design and Implementation of the FreeBSD Operating System, 2nd Edition_ has a whole chapter on it: _Chapter 6. Memory Management_ (see: FreeBSD Development: Books, Papers, Slides). In addition to that fine chapter:

 Design elements of the FreeBSD VM system or _pdf version_
An easy to follow description of the design of the FreeBSD virtual memory system
 Memory Management in FreeBSD 12.0 by Mark Johnston, BSD Taiwan in 2017 - _slides_*
 Exploring Swap on FreeBSD by Mark Johnston (written for Klara Systems), January 14, 2021**
In the last entry, Mark Johnston gives detailed insight—without discussing it at the code level—of the operation of three queues (the _active_, _inactive_ and the _laundry_ queue) that are important for memory management where swapping is concerned. As programs are loaded, run and finish, memory pages are continuously created, deleted and moved from one queue to another. When pages need to be swapped out to disk however, they are taken from the laundry queue. This process is handled by the laundry thread. The laundry queue and its handler were both added to FreeBSD 12 as a better way to manage memory and more reliably predict the use of memory pages likely not needed in the future.   

As filesystems go, ZFS will take you to the next level. Where UFS extends somewhat into main memory management, ZFS has a prominent presence in main memory and control of its components there. When you look at how things interact in ZFS internally, it may help you to better understand the influence of individual memory (kernel) parameters:

 The Design and Implementation of the FreeBSD Operating System, 2nd Edition: _Chapter 10. The Zettabyte Filesystem_
 ZFS Internals Overview by Kirk McKusick - 21 Oct 2015
 OpenZFS Basics by Matt Ahrens and George Wilson - 15 May 2018
 ELI5: ZFS Caching (2019) - 18 Oct 2019, by Allan Jude
An in-depth look at how caching works in ZFS, specifically the Adaptive Replacement Cache (ARC) 
 ELI5: ZFS Caching (2019) - 18 Oct 2019, by Allan Jude  >_slides_<
Also, have a look at it from the top :

 Explaining top(1) on FreeBSD - October 18, 2021, by Klara Systems
___
* unfortunately only slides; a small description of the presentation by Allan Jude in BSD in Taiwan | BSD Now 221 (from ca. 26:34 min).
** [edit: added] this article is also referred to in Explaining top(1) on FreeBSD but it deserves to be mentioned explicitly. grahamperrin had mentioned it explicitly before me.


----------



## mark_j (Apr 10, 2022)

Ah, Laundry, an often misunderstood concept. Please see here.


----------



## Korger (Apr 10, 2022)

Does this mean that swap is mandatory in FreeBSD?  I don't like how slowly pages are loaded from swap, so I was thinking of investing more in RAM and disabling swap entirely.  So when you say



PMc said:


> If your program creates a variable, that needs memory, This memory is certainly changed, so it must point to swapspace



what happens if I either don't have swap at all, or it's much smaller than the actual RAM size, so not all pages in RAM can point to swap?  Do I lose any functionality?  Does this also mean that laundry exists only when swap exists?


----------



## grahamperrin@ (Apr 10, 2022)

Korger said:


> Does this mean that swap is mandatory in FreeBSD?



No.



> I don't like how slowly pages are loaded from swap, so I was thinking of investing more in RAM and disabling swap entirely. …



More RAM is good. 

I don't have the usual links handy, but I'm fairly certain that it's usually recommended to enable swap. From Exploring Swap on FreeBSD | Klara Inc. (2021): 



> … Although swap is tremendously slower than RAM, it can still be a valuable tool. …



With or without swap enabled: it's commonplace to use the swap _partition_ for crash dumps. dumpon(8).


----------



## jbo (Apr 10, 2022)

Korger said:


> Does this mean that swap is mandatory in FreeBSD?


It is not. After all, you can also disable/enable swap on the go. See swapctl(8).



Korger said:


> I don't like how slowly pages are loaded from swap, so I was thinking of investing more in RAM and disabling swap entirely.


Don't forget that unused RAM is wasted RAM.
While more RAM is often better/desirable, personally, I'd recommend not spend money on hardware upgrades unless you're actually running into a performance issue/bottleneck. Unused RAM is wasted RAM.
Lets also not forget that modern non-volatile storage media (eg. SSDs) are pretty fast. So swapping in and out of the system isn't as "painful" as it used to be on a rotating disk where you're severely I/O limited (compared to a modern NVMe SSD). The kernel's job is to make good decisions on when to swap in or out. Unless you're actually running into a bottleneck you'd not expect the kernel to swap-out data that is frequently or soon accessed again.

I don't think that this is relevant here but let's not forget that in theory (and also in practice), more RAM makes a system less robust from a hardware perspective. This is especially true for non-ECC RAM.


----------



## grahamperrin@ (Apr 10, 2022)

If you like:









						GitHub - Freaky/swapflush: Flush swap devices on FreeBSD
					

Flush swap devices on FreeBSD. Contribute to Freaky/swapflush development by creating an account on GitHub.




					github.com
				





```
% date ; uptime ; uname -KU ; swapinfo
Sun 10 Apr 2022 20:17:19 BST
 8:17p.m.  up  1:16, 5 users, load averages: 4.34, 6.78, 6.64
1400056 1400056
Device          1M-blocks     Used    Avail Capacity
/dev/ada0p2.eli     16384      354    16029     2%
% sudo time /usr/home/grahamperrin/dev/swapflush/swapflush -a
grahamperrin's password:
        7.87 real         0.00 user         0.14 sys
% swapinfo
Device          1M-blocks     Used    Avail Capacity
/dev/ada0p2.eli     16384        0    16384     0%
%
```

Later:


```
% date ; uptime ; swapinfo ; sudo time /usr/home/grahamperrin/dev/swapflush/swapflush -a && swapinfo
Mon 11 Apr 2022 04:35:35 BST
 4:35a.m.  up  9:34, 5 users, load averages: 1.43, 1.75, 2.25
Device          1M-blocks     Used    Avail Capacity
/dev/ada0p2.eli     16384     1097    15286     7%
swapflush: swapoff: Cannot allocate memory
        0.00 real         0.00 user         0.00 sys
% date ; uptime ; swapinfo ; sudo time /usr/home/grahamperrin/dev/swapflush/swapflush -a && swapinfo
Mon 11 Apr 2022 04:37:07 BST
 4:37a.m.  up  9:36, 5 users, load averages: 1.83, 1.79, 2.21
Device          1M-blocks     Used    Avail Capacity
/dev/ada0p2.eli     16384     1081    15302     7%
       69.28 real         0.00 user         0.72 sys
Device          1M-blocks     Used    Avail Capacity
/dev/ada0p2.eli     16384        0    16384     0%
%
```

in response to `swapoff: Cannot allocate memory` I chose to close two Firefox windows, one of which included the FreeBSD Handbook.


----------



## PMc (Apr 10, 2022)

Korger said:


> Does this mean that swap is mandatory in FreeBSD?


Today not anymore. Traditionally (until ~2000) it was mandatory to have at least the amount of swap as of ram.
But them memory got always bigger, and in such sizes most of the swap was not practically useful anymore. So the systems were tuned to cope with the situation of no physical swap existing. They will nevertheless consider a dirty page as _logically _residing in swapspace, but, if there is none and as long as memory does not get seriously exhausted, this will be of no consequences.



Korger said:


> what happens if I either don't have swap at all, or it's much smaller than the actual RAM size, so not all pages in RAM can point to swap?  Do I lose any functionality?


Much smaller is not a problem, because the mapping is logical, not physical: the swap is filled as available. No swap at all will likely engage the OOM-killer when things get too big, and simply kill the biggest processes. But this is the default nowadays anyway - if you run with little memory and want slow swapping to happen, you must explicitely tune the OOM-Killer down (sysctl vm.pageout_oom_seq=_larger_value )_



Korger said:


> Does this also mean that laundry exists only when swap exists?


I don't know, I never tried that. I build base+ports in bhyves, and I throw a couple of them in as demand arises, and they can get very swappy when e.g. building llvm13, but it doesn't matter because the whole builds take a day or so anyway - but I definitely don't want things being killed away in midflight and the system idling and the stuff not ready to install when I come back in the morning.

The general rule of best-practice is: give the system some amount of swap, say, 5-10 GB, on SSD. The advantage is: you can quickly see what is going on and when something is exhausting memory. The downside is, if you have a bunch of browser tabs linger for a day unused, the system will consider them idle any may page them out, e.g. in the night when the periodic daily runs and needs lots of inode space. So the next access to these tabs will then be noticeable slower.


----------



## bakul (Apr 11, 2022)

Have a swap device at least as large as the RAM in your machine in case you want to save and later poke at kernel crash dumps. But in general any swapping will slow things down (much less so on an nvme device but still noticeable) so put in more RAM. When suddenly the systems seems to slowdown, you can use ps to check that your firefox or some other program has been using up memory! Otherwise the kernel will kill a random process when all physical memory is used up. Here you at least have a chance to do something more useful.


----------



## grahamperrin@ (Apr 11, 2022)

Korger said:


> … thinking of investing more in RAM and disabling swap entirely. …





grahamperrin said:


> … I don't have the usual links handy, …



I still can't find the reference that I want, the closest I can find is this (from Mark Johnston's article):



> … many applications allocate RAM in advance which they never actually use. If you have swap available, those mallocate requests are initially fulfilled by simply reserving swap space. Without swap, on a system without free RAM, the kernel must evict potentially valuable page cache to honor the mallocate call—even if that allocation is never actually _used_ by the app which requested it. …



Discussion of the article: <https://news.ycombinator.com/item?id=25789809>


More …

tuning(7)

<https://old.reddit.com/r/freebsd/comments/2k6hq0/swap_virtual_machines/clj2ic1/> is old (2014), but covers some of what's in the manual page.

<https://unix.stackexchange.com/a/146166/13260> is similarly old, but helped to remind me of these two things:

<https://wiki.freebsd.org/SystemTuning> – originally based on the manual page for tuning(7)
Design elements of the FreeBSD VM system | FreeBSD Documentation Portal
FreeBSD bug 263212 – Author (Matthew Dillon) and date of publication (2000) for 'Design elements of the FreeBSD VM system'

Chapter 8: Beyond Processes in Forensic Discovery (2004)


----------



## PMc (Apr 11, 2022)

grahamperrin said:


> I still can't find the reference that I want, the closest I can find is this (from Mark Johnston's article):


Well, elaborate on that. Explain Your point. 

Or, let people just do what they want. There is a tendency for simple explainings - so if over time some count on the swap, like here:


```
470 processes: 1 running, 469 sleeping
CPU:  0.1% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.7% idle
Mem: 409M Active, 5759M Inact, 789M Laundry, 12G Wired, 1174M Buf, 12G Free
ARC: 6543M Total, 5351M MFU, 587M MRU, 3660K Anon, 415M Header, 178M Other
     5514M Compressed, 10G Uncompressed, 1.87:1 Ratio
Swap: 36G Total, 17M Used, 36G Free
```

... then that must be the obvious reason why the machine gets slow.


----------



## grahamperrin@ (Apr 12, 2022)

(Nit: I don't imagine _perceptible_ slowness from just 17 M use of swap.)


----------



## jbo (Apr 12, 2022)

grahamperrin said:


> (Nit: I don't imagine _perceptible_ slowness from just 17 M use of swap.)


I agree on that. After all, the system also reports 12G of free memory.
Unfortunately, I don't yet have an in-dept understanding of FreeBSD's memory management design but in general (i.e. from an academic/abstract point of view), a kernel is allowed to use swap space when available. Even if there's plenty of primary memory available. I have no idea what these 17M of swapped data exactly is but given that it's swapped-out while there is still _plenty (!)_ of available primary memory would suggest that the kernel decided that those 17M are just not "needed" in primary memory. In general, this is not an indication for a badly designed memory management system nor that the machine would benefit from additional RAM being installed.


----------



## PMc (Apr 12, 2022)

jbodenmann said:


> I agree on that. After all, the system also reports 12G of free memory.
> Unfortunately, I don't yet have an in-dept understanding of FreeBSD's memory management design but in general (i.e. from an academic/abstract point of view), a kernel is allowed to use swap space when available. Even if there's plenty of primary memory available. I have no idea what these 17M of swapped data exactly is but given that it's swapped-out while there is still _plenty (!)_ of available primary memory would suggest that the kernel decided that those 17M are just not "needed" in primary memory.


There may have been an earlier time when memory was filled by something. That something disappeared again, memory was freed, but a couple unused pages had gone into swap - and were never fetched back from there because these pages weren't touched. 
There is quite a lot of pages that are used once at process start for some initialization, and then not again - because you rarely use all the options of some command.

Swap is not like a railway train where the swap wagons come at the end and all is filled (and later freed again) in sequence. With earlier systems (say Linux as of 1994) you could observe memory getting full, and only then the machine would come to a halt while moving memory to swap - you could not even login, but you could watch the drive light. This is easy to understand what's happening - but it is not very useful for networked systems and multiprocessor systems. So we need an approach that acts earlier and more probabilistic.


----------



## Deleted member 70435 (Apr 12, 2022)

you can get info `vmstat -m` `vmstat -z`


----------



## grahamperrin@ (Apr 12, 2022)

PMc said:


> … Explain Your point.  …



I vaguely recall reading that some types of application can or will use less RAM with swap enabled, than with swap disabled. Words to that effect. Maybe something that I picked up in my Mac OS X days.


----------



## grahamperrin@ (Apr 12, 2022)

PMc said:


> … No swap at all will likely engage the OOM-killer when things get too big, and simply kill the biggest processes. …



In my experience (with swap _enabled_), killings are not always so blunt. 



grahamperrin said:


> … if you haven't already seen it:
> 
> The out-of-swap killer makes poor choices | <https://markmail.org/message/siorx6pswhpncluf>



(The linked text is verbatim, not me _agreeing_ about poor choices – I don't know enough.)


----------



## PMc (Apr 13, 2022)

Dimitri Chuikov said:


> you can get info `vmstat -m` `vmstat -z`


Yes, but they did never line up.
And I had this large lump of "Wired", half of the installed mem or more, and it gave me headaches since I use ZFS. I know what it is, most of it comes from kernel but it can in part also be user processes, and there is no means to sort it out and get a precise quantity structure of what is actually in there right now.
Then at some point my ZFS stalled at arc_min after a day or so because of mem fragmentation (still i386), and then UMA appeared, and I began to understand what it is all about - what the problem actually is with mem allocation, what the solution is, etc. And then I figured why the "Wired" may bloat into more than 60% of installed mem (a lot more than the actual ARC size), but then may magically shrink again when memory gets scarce - and this works by signalling - there is not a code function I might find within arc.c (yes, I searched for such).

And now You come along and explain all this, just as if it were the daily news. Yeah, I had great fun reading that article.


----------



## PMc (Apr 13, 2022)

grahamperrin said:


> In my experience (with swap _enabled_), killings are not always so blunt.


It depends on how fast your swap is. 
And on sysctl vm.pageout_oom_seq.



grahamperrin said:


> (The linked text is verbatim, not me _agreeing_ about poor choices – I don't know enough.)


Yes, I have perceived exactly this as described here, when building my ports.  I searched for solutions and found the hint to pageout_oom_seq. This can be set to 1000 or higher, and then the oom-killer will not engage, even on slow swapspace. (I don't know what happens when actually running out of swap - maybe this does then not get detected - but should never happen anyway.) 
But then, building llvm13, with the fortran stuff, at some point creates compile processes around 4 GB each. So running this in a 10core vm, one needs 40GB mem size. Forcing this to swap will then not crash, but will never come to an end, because these 10 processes continuousely rival against each other, and none gets enough RSS to really get someway further. 
That means: swapping, which worked somehow successful in single CPU systems, may not work at all in multi processor when multiple big processes are created at the same time.


----------



## Deleted member 70435 (Apr 13, 2022)

here is some knowledge, well some parts are in russian but i translate for you


----------



## grahamperrin@ (Apr 13, 2022)

Random, this morning, with `if ( -x /usr/bin/fortune ) /usr/bin/fortune freebsd-tips` in my ~./login:



> To learn more about what your system is doing, take a look at systat(1). For example, to get various statistics related to virtual memory usage, process scheduling, device interrupts, system name translation caching, and disk I/O, enter the following:
> 
> `systat -vmstat`
> 
> ...



`systat -vmstat`


```
8 users    Load  5.85  3.79  4.85                  13 Apr 08:43:36
   Mem usage:  98%Phy 16%Kmem                           VN PAGER   SWAP PAGER
Mem:      REAL           VIRTUAL                        in   out     in   out
       Tot   Share     Tot    Share     Free   count     3                 59
Act  8020M    310M    584G     911M     345M   pages    19                663
All  8044M    334M    585G    1039M                     3 ioflt  Interrupts
Proc:                                                4688 cow    4082 total
  r   p   d    s   w   Csw  Trp  Sys  Int  Sof  Flt   18K zfod        atkbd0 1
  2      20  962       32K  27K 138K  589       24K       ozfod       acpi0 9
                                                         %ozfod    95 ehci0 ehci
13.9%Sys   0.0%Intr 34.8%User  0.1%Nice 51.2%Idle     13K daefr   919 cpu0:timer
|    |    |    |    |    |    |    |    |    |    |   19K prcfr   850 cpu1:timer
=======>>>>>>>>>>>>>>>>>                              44K totfr   854 cpu2:timer
                                           dtbuf      946 react   870 cpu3:timer
Namei     Name-cache   Dir-cache    348278 maxvn          pdwak       hdac0 32
   Calls    hits   %    hits   %    348278 numvn      27K pdpgs   195 xhci0 33
  502668  431485  86                160606 frevn       40 intrn     1 em0:irq0
                                                    2976M wire        hdac1 36
Disks  ada0   da0   da1 pass0 pass1 pass2 pass3     8242M act      43 iwn0 37
KB/t  59.13  6.74  5.64  0.00  0.00  0.00  0.00     3228M inact   188 ahci0 38
tps     188    13     7     0     0     0     0      885M laund    67 vgapci0 39
MB/s  10.87  0.08  0.04  0.00  0.00  0.00  0.00      345M free
%busy    51     2     0     0     0     0     0         0 buf
```

`:swap`

… a few minutes later:



Spoiler: 2,968 K used





```
/0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   |||||||||||

Device/Path       Size  Used |0%  /10  /20  /30  /40  / 60\  70\  80\  90\ 100|
ada0p2.eli         16G 2968K

Pid    Username   Command     Swap/Total Per-Process    Per-System
 19592 grahamperrin firefox    3M /  14G  0%              0%
 20766 grahamperrin firefox  284K /   3G  0%              0%
```




… later, in xterm:




… bugged after a few hours in xterm:



– so I `:quit` then ran `systat -swap`:



Spoiler: 844 M used





```
/0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   ||||||

Device/Path       Size  Used |0%  /10  /20  /30  /40  / 60\  70\  80\  90\ 100|
ada0p2.eli         16G  844M XX

Pid    Username   Command     Swap/Total Per-Process    Per-System
 19592 grahamperrin firefox  163M /  15G  1%              0%
 22023 grahamperrin firefox  155M /   9G  1%              0%
  3185 grahamperrin mysqld   127M / 472M 26% XX           0%
  3203 grahamperrin java      82M /   6G  1%              0%
  3089 grahamperrin plasmashe 17M / 797M  2%              0%
  3212 grahamperrin python3.8 16M / 222M  7%              0%
 33861 grahamperrin firefox   15M /   3G  0%              0%
  3229 grahamperrin python3.8 15M /  67M 22% XX           0%
  3230 grahamperrin python3.8 14M /  54M 25% XX           0%
  3118 grahamperrin kgpg      13M / 349M  3%              0%
  3085 grahamperrin kwin_x11   7M / 758M  0%              0%
  3107 grahamperrin xterm      5M /  25M 20% XX           0%
 20766 grahamperrin firefox    5M /   3G  0%              0%
  3084 grahamperrin kded5      5M / 233M  2%              0%
 57873 grahamperrin recollind  4M / 128M  3%              0%
  3086 grahamperrin ksmserver  3M / 145M  2%              0%
  3198 grahamperrin xterm      3M /  25M 11% X            0%
  3139 grahamperrin korgac     3M / 215M  1%              0%
  3061 grahamperrin startplas  3M / 117M  2%              0%
  3074 grahamperrin plasma_se  2M / 117M  2%              0%
  3076 grahamperrin kdeinit5   2M / 118M  2%              0%
  3095 grahamperrin DiscoverN  2M / 169M  1%              0%
 82438 grahamperrin dolphin    2M / 415M  0%              0%
  3091 grahamperrin org_kde_p  2M / 144M  1%              0%
 81314 grahamperrin code-oss   2M /  49G  0%              0%
  3136 grahamperrin kscreen_b  2M /  90M  1%              0%
  3178 grahamperrin akonadi_c  2M / 139M  1%              0%
  3077 grahamperrin klauncher  2M / 134M  1%              0%
  3179 grahamperrin akonadise  2M / 188M  0%              0%
  3224 grahamperrin gvfs-udis  1M /  60M  2%              0%
  3092 grahamperrin kaccess    1M / 282M  0%              0%
 81312 grahamperrin code-oss   1M /  41G  0%              0%
  3088 grahamperrin kglobalac  1M / 141M  0%              0%
  3102 grahamperrin kactivity  1M / 144M  0%              0%
  3143 grahamperrin at-spi-bu  1M /  47M  2%              0%
 81302 grahamperrin code-oss   1M /  49G  0%              0%
  3094 grahamperrin xembedsni  1M /  91M  1%              0%
  3111 grahamperrin dconf-ser  1M /  31M  3%              0%
  3226 grahamperrin gvfs-mtp-  1M /  36M  3%              0%
  3099 grahamperrin gmenudbus  1M / 101M  1%              0%
  3096 grahamperrin kdeconnec  1M / 268M  0%              0%
  3148 grahamperrin at-spi2-r  1M /  36M  2%              0%
  3186 grahamperrin gkrellm  944K / 143M  0%              0%
 81898 grahamperrin firefox  564K /   3G  0%              0%
  3016 grahamperrin ck-launch520K /  16M  3%              0%
  3134 grahamperrin gstat    424K /  14M  3%              0%
  3218 grahamperrin ibus-daem420K /  51M  0%              0%
  3138 grahamperrin gvfsd    412K /  45M  0%              0%
  3059 grahamperrin dbus-run-396K /  13M  2%              0%
  3216 grahamperrin autorandr388K /  13M  2%              0%
  3144 grahamperrin dbus-daem340K /  14M  2%              0%
  3206 grahamperrin java     304K /   6G  0%              0%
  3060 grahamperrin dbus-daem296K /  16M  1%              0%
  3331 grahamperrin akonadi_e188K / 617M  0%              0%



Disks  ada0   da0   da1 pass0 pass1 pass2 pass3
KB/t  14.88 21.50  4.12  0.00  0.00  0.00  0.00
tps      18     1     1     0     0     0     0
MB/s   0.26  0.01  0.00  0.00  0.00  0.00  0.00
%busy     8     0     0     0     0     0     0
Showing swap, refresh every 5 seconds.
```




The xterm view is incomplete, compared to Konsole:



Uptime 16:30,



Spoiler: 3,544 M of 16 G used





```
/0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   ||||

Device/Path       Size  Used |0%  /10  /20  /30  /40  / 60\  70\  80\  90\ 100|
ada0p2.eli         16G 3544M XXXXXXXXXX

Pid    Username   Command     Swap/Total Per-Process    Per-System
 19592 grahamperrin firefox  465M /  16G  2%              2%
 22023 grahamperrin firefox  349M /   9G  3%              2%
  3203 grahamperrin java     258M /   6G  4%              1%
  3185 grahamperrin mysqld   167M / 472M 35% XXX          1%
 11247 grahamperrin firefox  155M /   3G  5%              0%
  9786 grahamperrin firefox  132M /   3G  4%              0%
  3089 grahamperrin plasmashe115M / 807M 14% X            0%
 81312 grahamperrin code-oss 104M /  41G  0%              0%
  9940 grahamperrin firefox  100M /   3G  3%              0%
 81898 grahamperrin firefox   98M /   3G  3%              0%
 11060 grahamperrin firefox   92M /   3G  3%              0%
  3206 grahamperrin java      87M /   6G  1%              0%
  3085 grahamperrin kwin_x11  66M / 757M  8%              0%
 81308 grahamperrin code-oss  63M /  59G  0%              0%
 81302 grahamperrin code-oss  59M /  49G  0%              0%
 10957 grahamperrin firefox   39M /   3G  1%              0%
 62485 grahamperrin gam_serve 37M /  55M 68% XXXXXX       0%
 81314 grahamperrin code-oss  30M /  49G  0%              0%
 57873 grahamperrin recollind 28M / 128M 21% XX           0%
 82438 grahamperrin dolphin   26M / 415M  6%              0%
 81304 grahamperrin code-oss  26M / 626M  4%              0%
  3212 grahamperrin python3.8 25M / 222M 11% X            0%
 77377 grahamperrin spectacle 23M / 523M  4%              0%
  3229 grahamperrin python3.8 22M /  67M 33% XXX          0%
  3389 grahamperrin konsole   19M / 378M  5%              0%
  3230 grahamperrin python3.8 19M /  54M 34% XXX          0%
 25772 grahamperrin firefox   16M /   3G  0%              0%
  3287 grahamperrin ibus-exte 14M /  96M 14% X            0%
  3381 grahamperrin kwalletma 14M / 343M  4%              0%
  3118 grahamperrin kgpg      14M / 349M  3%              0%
 20766 grahamperrin firefox   13M /   3G  0%              0%
  3233 grahamperrin kwalletd5 13M / 341M  3%              0%
 81316 grahamperrin code-oss  13M /  40G  0%              0%
 81318 grahamperrin code-oss  11M /  40G  0%              0%
  3179 grahamperrin akonadise  9M / 188M  4%              0%
  3211 grahamperrin evolution  9M / 245M  3%              0%
  3084 grahamperrin kded5      7M / 233M  2%              0%
  3186 grahamperrin gkrellm    6M / 143M  4%              0%
  3337 grahamperrin akonadi_m  6M / 653M  0%              0%
  3328 grahamperrin akonadi_a  6M / 647M  0%              0%
  3342 grahamperrin akonadi_u  6M / 640M  0%              0%
  3332 grahamperrin akonadi_f  6M / 644M  0%              0%
  3338 grahamperrin akonadi_m  6M / 640M  0%              0%
  3341 grahamperrin akonadi_s  6M / 640M  0%              0%
 27897 grahamperrin firefox    6M /   3G  0%              0%
  3107 grahamperrin xterm      5M /  25M 20% XX           0%
  3331 grahamperrin akonadi_e  5M / 617M  0%              0%
  3357 grahamperrin evolution  5M / 130M  3%              0%
   883 grahamperrin gtk-mixer  5M / 640M  0%              0%
  3351 grahamperrin evolution  5M / 132M  3%              0%
  3092 grahamperrin kaccess    4M / 282M  1%              0%
  3360 grahamperrin evolution  4M / 149M  2%              0%



Disks  ada0   da0   da1 pass0 pass1 pass2 pass3
KB/t  14.65  0.00  0.00  0.00  0.00  0.00  0.00
tps      19     0     0     0     0     0     0
MB/s   0.27  0.00  0.00  0.00  0.00  0.00  0.00
%busy     8     0     0     0     0     0     0
Showing swap, refresh every 5 seconds.
```


----------



## grahamperrin@ (Apr 18, 2022)

Retrospective (2015): 

Out of Memory Handler Rewrite
– Konstantin Belousov (kib@), project sponsored by the FreeBSD Foundation.


----------



## _al (Jun 10, 2022)

Additionally (as a supplement), classic explanation of "How BSD memory works " (on example of a stack of network protocols)  contained in "TCP/IP Illustrated Volume 2,  by W. Richard Stevens and Gary R. Wright".


----------

