# swap - memory



## fernandel (Jan 14, 2018)

Hi!

I have FreeBSD 11.1 - Release (amd64).
I am using `portmaster` for building ports. The "problem" is why it use for building swap if is a lot of memory available still.
I am using GNOME 3 at this momemnt and rebuilding www/iridium and
there are 3.2 GM of 8.0 GB used and 25.7 MB of Swap. Size of the swap partition is 3.6 GB.
I am running System Monitor and used of RAM is between 3.1 GB to 3.7 GB.

Do I need to do some settings, please?
Thank you.


----------



## ralphbsz (Jan 15, 2018)

You did notice that the amount of swap used is microscopically small?  You have 8G of RAM, and 0.025G of swap were used.  At the typical speed of a hard disk (about 100-200 MByte/s), writing that swap took less than 1/4 second.

For some reason that I don't understand, some operating systems use swap, even when they are never out of memory.  OK, that' a fact of life.  As long as the swap usage is tiny, it should not be a reason to worry.


----------



## SirDice (Jan 15, 2018)

Modern operating systems will always use a bit of swap. This is a tiny amount however. Swap usage in and of itself is never a problem, it's excessive swapping that will cause performance issues.


----------



## Snurg (Jan 15, 2018)

To find how much swap my pc uses atm, I did `man swapctl` and then tried `swapctl -h` according to the info there.
It only puts out the usage information. However, `swapinfo -h` works.
Is it me, my computer, or an incorrect man page?

Semi-OT: Regarding SirDice's  point of excessive swapping, I made an interesting observation.
On Windows computers, I regularly notice much swap-in when I switch between different apps.
Even with lots of spare memory, Windows appears to swap out gigabytes, bogging down computer's performance.
This is the reason why I configure Windows usually that way that it only gets the absolute minimum possible swap partition size to avoid performance loss.
I wonder what's the reasoning behind Windows' swap strategy...


----------



## SirDice (Jan 15, 2018)

Snurg said:


> I did  man swapctl and then tried  swapctl -h according to the info there.
> It only puts out the usage information.


Because you need to tell it to list something, `swapctl -l -h`


----------



## Crivens (Jan 15, 2018)

The reason the system uses swap when memory is "free" simply is that that memory is not really free. The free memory is used to cache file data, so there is some benefit from pushing unused content to the swap partition to make more cache memory available. Windowa, IIRC, does swap coloring and prefetching on a diferent level, which means it will swap out cached data from Word when Photoshop is in front to make more room for that one. So you always profit from giving the system some swap space.


----------



## achix (Feb 9, 2018)

SirDice said:


> it's excessive swapping that will cause performance issues.


Which is exactly what is happening with FreeBSD 11.1. After I upgraded to 11.1 I noticed significant drop in performance. So I told the admin ppl to just order another 16GB for me. After I installed the extra mem, the system went OK for about 1-2 days, then it started swapping again. Basically we spent the money on the extra mem to still have a slow system.
Disabled the swap and lived happily thereafter (even if having a system not set up "officially" correct).


----------



## SirDice (Feb 9, 2018)

achix said:


> So I told the admin ppl to just order another 16GB for me. After I installed the extra mem, the system went OK for about 1-2 days, then it started swapping again. Basically we spent the money on the extra mem to still have a slow system.


I have several 11.1 machines with only 8GB. They rarely touch the swap, even after running non-stop for a month or so. So I suspect you have an application that's using up the memory. Might even have a memory leak.


----------



## Snurg (Feb 9, 2018)

achix' post about turning off swap motivated me to turn off swap.

*And the change in responsiveness was prompt and very noticeable*.
And at this moment the workstation session was very fresh. Booted up just a few minutes ago. Free memory was 43GB at that moment!

Another interesting observation about memory usage is this:

Recently I backed up my laptop's HDD, I plugged it into the running computer, turned the drive's bay power on, imported the pool and copied the data (>200GB) onto the workstations' drives.
After that I noticed that Firefox had become extremely sluggish, had to reload every tab that was previously loaded. Like if it was started freshly. But it was a session several days old.

When I examined memory usage, I found that ZFS ARC had taken over all memory.
*ARC cache had grown to 43GB, leaving only 5GB for kernel+programs.
Swap usage was little, about 80MB of 4GB swap area.

Firefox apparently had to drop all memory usage on demand of ZFS ARC, leaving it with only 57 MB!!!*
Computer had become very very sluggish. I had to reboot to fix that.
(Just now, this session is very fresh, only 2 hrs. And so Firefox just now uses only 5GB for 10 windows with maybe 30, 40 tabs of >100 loaded. On long sessions it takes >10GB).

After that experience I restricted ZFS ARC usage to 9GB to avoid having to reboot after backing up things.

And... until I discovered that, I always believed Firefox were the culprit... I then restarted it. But I hate to restart it, because all tabs then need to be reloaded when I access them.


----------



## achix (Feb 9, 2018)

SirDice said:


> I have several 11.1 machines with only 8GB. They rarely touch the swap, even after running non-stop for a month or so. So I suspect you have an application that's using up the memory. Might even have a memory leak.



Will have to let the system run for a while and let you know. For the moment it is running better than ever, with everything up (both bhyve guests, jboss, eclipse, postgresql , KDE, thunderbird, lots of konsoles, firefox windows, etc) and having swapoff'ed the swap disk. I have no ZFS.

Currently top shows :
`last pid: 10099;  load averages:  0.52,  0.51,  0.49                                up 29+23:13:44  10:51:50
268 processes: 1 running, 267 sleeping
CPU:     % user,     % nice,     % system,     % interrupt,     % idle
Mem: 2703M Active, 25G Inact, 3186M Wired, 1436M Buf, 558M Free
Swap:`

When swap was on, it went to a point of using it all, while still having lots of Inactive memory. Would love to enable back swap , let it run and report back, but this would mean killing all services to free mem and then disable it again, and today is not a good day to do so.


----------



## SirDice (Feb 9, 2018)

achix said:


> but this would mean killing all services to free mem and then disable it again, and today is not a good day to do so.


That's quite understandable.

Machines haven't been running long, a few days now, had to reboot for updates. But this is my firewall, has 4GB, but not much running on it:

```
last pid: 54876;  load averages:  0.01,  0.00,  0.00                                                       up 10+07:09:28  10:40:45
67 processes:  2 running, 64 sleeping, 1 waiting
CPU 0:  0.0% user,  0.0% nice,  0.4% system,  0.8% interrupt, 98.8% idle
CPU 1:  0.0% user,  0.0% nice,  1.6% system,  0.4% interrupt, 98.0% idle
Mem: 21M Active, 109M Inact, 2304M Wired, 1217M Free
ARC: 1043M Total, 481M MFU, 274M MRU, 32K Anon, 14M Header, 273M Other
     518M Compressed, 873M Uncompressed, 1.69:1 Ratio
Swap: 8192M Total, 8192M Free
```
Another machine, 16GB (recently upgraded from 8GB):

```
last pid: 38497;  load averages:  0.17,  0.24,  0.23                                                        up 4+19:42:10  10:42:15
93 processes:  2 running, 90 sleeping, 1 waiting
CPU 0:  2.4% user,  0.0% nice,  0.8% system,  0.4% interrupt, 96.5% idle
CPU 1:  3.1% user,  0.0% nice,  1.2% system,  0.0% interrupt, 95.7% idle
CPU 2:  2.7% user,  0.0% nice,  2.0% system,  0.0% interrupt, 95.3% idle
CPU 3:  1.9% user,  0.0% nice,  1.6% system,  0.0% interrupt, 96.5% idle
Mem: 2115M Active, 700M Inact, 29M Laundry, 12G Wired, 730M Free
ARC: 8182M Total, 2955M MFU, 4567M MRU, 256K Anon, 101M Header, 558M Other
     7103M Compressed, 8067M Uncompressed, 1.14:1 Ratio
Swap: 16G Total, 755M Used, 15G Free, 4% Inuse
```
There is some swap usage on this machine, I suspect it's because of Java running on it. Swap is used but there's very little going in or out so it's not problem and doesn't affect performance. This machine does most of the work on my home network, torrents, Samba, a Java app, a jail and a bhyve VM. I also use this machine to build packages with poudriere on it.

One last example:

```
last pid:   635;  load averages:  0.68,  0.73,  0.61                                                        up 4+19:20:04  10:50:19
107 processes: 2 running, 104 sleeping, 1 waiting
CPU 0:  0.8% user,  0.0% nice,  0.4% system,  0.0% interrupt, 98.9% idle
CPU 1:  0.8% user,  0.0% nice,  1.1% system,  0.0% interrupt, 98.1% idle
CPU 2:  1.1% user,  0.0% nice,  0.0% system,  0.4% interrupt, 98.5% idle
CPU 3:  1.5% user,  0.0% nice,  0.0% system,  0.0% interrupt, 98.5% idle
Mem: 185M Active, 314M Inact, 1632M Wired, 5796M Free
ARC: 1161M Total, 322M MFU, 761M MRU, 23M Anon, 7813K Header, 47M Other
     983M Compressed, 1972M Uncompressed, 2.01:1 Ratio
Swap: 2048M Total, 2048M Free
```
This is a bhyve VM, 4 cores, 8GB memory. It runs Zabbix 3.4.


----------



## fernandel (Feb 9, 2018)

On the FreeBSD version before 11.1 I use:
`sysctl vm.defer_swapspace_pageouts=1`
and it help me but it doesn't work on 11.1.


----------



## Minbari (Feb 9, 2018)

My machines are swapless (ram 8GB, fs=ZFS) for years and everything is working ok considering the fact that I'm always building every program from sources. Check to see if u don't have a memory leak.


```
last pid: 49408;  load averages:  0.30,  0.49,  1.01                                                                                   up 0+09:34:26  12:48:56
50 processes:  4 running, 46 sleeping
CPU:  5.3% user,  0.0% nice,  1.4% system,  0.0% interrupt, 93.2% idle
Mem: 577M Active, 244M Inact, 86M Laundry, 2176M Wired, 4675M Free
ARC: 1112M Total, 182M MFU, 753M MRU, 16K Anon, 11M Header, 165M Other
     689M Compressed, 1511M Uncompressed, 2.19:1 Ratio
Swap:
```


----------



## achix (Feb 9, 2018)

Reverted back to using swap, but turned off kern.ipc.shm_use_phys . Seems ok since I did it 1-2 hours ago. I noticed some minimal swap used at the first minutes, but then went to zero. Will keep an eye to it.


----------



## Snurg (Feb 9, 2018)

Like Minbari reported his experiences, I had for years FreeBSD (older versions) running on an 8GB machine with no swap partitions, never experienced any non-obvious problem.

It was about 3 years ago when I decided to have the system use the default swap allocated by the installer, because (almost) all people said one should not disable swap.

For my part, I will not experiment any more in that direction.
With me, swap is gone forever. Not going to activate it again.
(I don't care about being perceived as heretic.)


----------



## Nicola Mingotti (Feb 9, 2018)

Guys, I disabled the swap, my FreeBSD-11.1 running inside a VMWare (1 core) on a 2009 Macbook is now fast and furious !!!  ... I know there will be drawbacks but I want to enjoy the moment now

BTW, I think there are memory problems in Firefox (package version). Memory free does not increas when I close tabs or windows, only when I close all Firefox windows !


----------



## Snurg (Feb 9, 2018)

Nicola Mingotti said:


> Memory free does not increas when I close tabs or windows, only when I close all Firefox windows !


My impression is that FF invokes its garbage collection only when free system memory decreases below some threshold.

This is why one can observe FF's memory usage shrinking when free system memory gets lower, even without closing any window/tab, up to the point that it keeps in memory only the topmost active window/tab.
It also would explain its behavior described in my post #9 above, when it released a dozen gigabytes of memory back to the system, as said without closing any window/tab, only to satisfy ZFS ARC's desire to gobble up every byte of RAM it can get hold of.

Conclusion:
Probably every desktop user can profit by restricting ZFS memory usage to reasonable limits, and prevent shared memory being swapped out, either by a) completely disabling swap or b) changing `kern.ipc.shm_use_phys` as achix suggested. Method a) has proven to work reliably, method b) still needs long-term verification.


----------



## Deleted member 45312 (Feb 9, 2018)

Snurg said:


> Method a) has proven to work reliably.


Not for me: under high memory pressure (like building packages with poudriere), the kernel kills some process to get free memory. So some important process could be suddenly killed. That's not the case with enough swap space. That's why I now have 12G of swap which is used at almost 30-40% when building packages with poudriere while using my desktop computer.


----------



## Eric A. Borisch (Feb 9, 2018)

achix said:


> Reverted back to using swap, but turned off kern.ipc.shm_use_phys . Seems ok since I did it 1-2 hours ago. I noticed some minimal swap used at the first minutes, but then went to zero. Will keep an eye to it.



To be clear, you returned kern.ipc.shm_use_phys to its _default_ (0) state and things worked better?


----------



## Snurg (Feb 9, 2018)

dlegrand said:


> Not for me: under high memory pressure (like building packages with poudriere), the kernel kills some process to get free memory. So some important process could be suddenly killed. That's not the case with enough swap space. That's why I now have 12G of swap which is used at almost 30-40% when building packages with poudriere while using my desktop computer.


May I ask how much memory your build box has total, and how much was free when starting poudriere?

Edit: I wonder if it might be sensible to allocate the kernel a generous minimum of reserved memory? (vm.kmem_size="...", vm.kmem_size_max="...")


----------



## topcat (Feb 9, 2018)

I don't have this problem on any of my 11.1 machines, and they have been up for weeks (only reboots for kernel updates). I run Firefox, Chrome with hundreds of tabs open, with 16 GB total RAM.

I wonder if something is touching a lot of pages in RAM quickly. This should result in a big inactive queue, and will also sometimes swap stuff out. It's probably reading and not writing, because you don't seem to have much in the laundry queue. `systat -vmstat` should be interesting to observe with swap on. Look at laundry list, soft and hard page faults.

This is a perfect opportunity to learn how the memory manager works


----------



## Deleted member 45312 (Feb 10, 2018)

Snurg said:


> May I ask how much memory your build box has total, and how much was free when starting poudriere?
> 
> Edit: I wonder if it might be sensible to allocate the kernel a generous minimum of reserved memory? (vm.kmem_size="...", vm.kmem_size_max="...")


It's a 6 cores CPU (FX6300) with 8Gb memory and separate graphic board with its own 1Gb memory. It's my desktop machine so I am running Xorg, Xfce, Firefox, Emacs, before launching poudriere. I am doing parallel build with sometimes 6 big builds like libreoffice, thunderbird, firefox ...
That's why sleeping processes are going to swap space, but my desktop is still responsive with running processes. I am happy like that, FreeBSD works great for me.

```
last pid:  1309;  load averages:  0.40,  0.31,  0.26                                                    up 0+01:23:10  09:01:30
87 processes:  1 running, 86 sleeping
CPU:  0.2% user,  0.0% nice,  0.2% system,  0.2% interrupt, 99.4% idle
Mem: 1105M Active, 712M Inact, 69M Laundry, 2542M Wired, 3474M Free
ARC: 1885M Total, 587M MFU, 1224M MRU, 316K Anon, 12M Header, 62M Other
     1534M Compressed, 3204M Uncompressed, 2.09:1 Ratio
Swap: 12G Total, 12G Free
```


----------



## achix (Feb 10, 2018)

Eric A. Borisch said:


> To be clear, you returned kern.ipc.shm_use_phys to its _default_ (0) state and things worked better?



it seemed so. I just logged in my workstation, remotely, and now it see it is using some 1.7GB swap out of the total 8GB. Not quite promising. All sysctl settings are back to default btw. I'll revert back in Monday, after some real usage.


----------



## Nicola Mingotti (Feb 10, 2018)

You all know for sure, there are two threads in which we are discussing the same things,
"Gereat UI/X preformance gains by turning off sapw" by achix 
"swap - memory" by fernandel 

We should migrate all the discussion into one. This is a very important
subject IMHO.


----------



## achix (Feb 10, 2018)

I agree. I vote for this thread, since the other's subject might end up misleading in the end.


----------



## Snurg (Feb 10, 2018)

I vote for this thread, too.

Iirc marino wrote there is a rule of thumb of about 8GB memory usage per Synth parallel build.
What @legrand wrote seems about the same size (8GB plus minus a few GB) when running poudriere building.
Is it then safe to assume that, with a machine with, say, 16GB free RAM one can do builds without having to expect memory going critically low?


----------



## Nicola Mingotti (Feb 10, 2018)

Great achix ! 

I am convinced you right person to take the decision because you opend the thread.

So, I think you could write a final post into  "Gereat UI/X preformance gains by turning off sapw"
and say something like: 
----
This discussion has moved to thread "swap - memory" by fernandel, 
we are discussiong there the same things. Please write there further
considerations, experiences and ideas.
-----


----------



## Nicola Mingotti (Feb 10, 2018)

Report of experiment: n#1

This post refers to a previous question I wrote as message #17 into thread "Gereat UI/X preformance gains by turning off swap" by achix. There is described my machine hardware.

*Problem*: My machine is small, runs FreeBSD-11.1. Removing swap it became damnt faster, i would like to keep swap-off but I am scared on consequences (mainly on the disk).

*Proposed Sulution*: Limit the main user "foo" moemory usage in such a way that system can not go out of memory.

*Actions Taken*:

```
-] Enable resource control writing in /boot/loader.conf
kern.racct.enable=1

-] Reboot

-] login as user "foo", start X, Firefox, Emacs, 2-3 terminals

-] Remove swap
#> swapoff -a

-] Limit user "foo" maxiumu memory use
#> rctl -a user:foo:memoryuse:deny=2g

-] turn on "top" and see what happens
#> top

-]  Run a script as user "foo" that eats memory ad see what happens
---- Ruby code, run from Emacs + Inf-ruby ----
[NOPARSE]
counter = 0
longString = ""
(1..20).each do |i|
  longString = longString + ("x" * 100_000_000)
  counter += 1
  puts "#{i} -- longString takes now ~ #{counter * 100} MBytes"
  sleep 2
end
[/NOPARSE]
-------------------------------------------------------------------
```

*Results*:
-] Looking at "top" I saw that memory resctions on user "foo" was not honoured.
The script in Ruby ate more than 2GB.
-] At some point the script ate too much memory and what happend surprised me a lot
--] The script did not stop and in order:
----] I lost control of Emacs
----] Some Firefow windows died
----] I lost control of the system
----] I had to power off
----] On reboot there was some complains about disk status, but I could complete the, luckily.

-] What suprises me most is that I expected the Ruby script should have been
killed by the system, with an error message like "Out of core memory".
Instead the whole system started to do unexpected stuff and eventually froze.


----------



## Snurg (Feb 10, 2018)

Wouldn't the same have happened after filling up the swap?
When the system is thrashing onto the swap and barely (if at all) responds to input, can one stop it safely?

As I understand (and you probably too) the rctl() man page, wouldn one expect to just not be allowed more than 2gigs... so what is wrong?


----------



## Snurg (Feb 10, 2018)

An utility like rctl would imho be the solution, if it worked...


----------



## Nicola Mingotti (Feb 10, 2018)

Exactly Snurg,
-] It crashes almost in the same way when i fill up the swap
-] I don't know why the "2gigs" limit is not honoured 

... further experiments are needed, but each time i risk to trash the machine ;P
I am writing the "memEater" in C now, I want to run it out of Emacs and Ruby GC control.


----------



## Snurg (Feb 10, 2018)

Maybe you can clone the VM you are using?
When the clone gets fscked up, it won't matter much I guess.
I will verify these experiments on a dedicated PC with ZFS. If the filesystem gets shot, no problem, just reinstall


----------



## Nicola Mingotti (Feb 10, 2018)

Yes, I could do another little vm which would be expendable, i will try leter in afternoon

Now i must leave the office, if you want to do some experiments here is the code in C for "memEater" .


```
[NOPARSE]
// WHAT:
// -] eats 100 Megybtes each second (or two, three) for "n" times
// -] "n" must me contained in ARGV[1]
//
// Compile
// $> cc -o memEater memEater.c
//
// Example, eats 1 GB in mem
// $> memEater 10
//
// Check
// $> top -w -o size

#include<stdlib.h>
#include<stdio.h>
#include<unistd.h>
#include<locale.h>

int main(int argc, char **argv) {
  void *ptr = NULL;
  int step = 100E6;
  int count = 0;
  int nstep = 0;
  // check there is one necessary argument
  if (argc != 2) { printf("Error, read code \n"); exit(2); }
  // no checks ! write a number in ARGV[1] !
  nstep = (int)strtol(argv[1], NULL, 10);
  // for comma seprated long integer
  setlocale(LC_NUMERIC, "en_US.UTF-8");
  for(int i = 0; i < nstep; i++) {
    ptr = malloc(step);
    if (ptr == NULL) {
      printf("Error, in allocating memory\n");
      exit(1);
    } else {
      count += step;
      printf("Allocated space : %'d MBytes \n", (int)(((float)count)/1E6));
    }
    sleep(3);
  }
  exit(0);
}

[/NOPARSE]
```


----------



## Deleted member 30996 (Feb 10, 2018)

My FreeBSD 11.1-RELEASE-p4 Thinkpad X61 .mp3 player with 4GB RAM is currently at 62.5 days uptime and showing 3979Mb of swap with 3943Mb free. I don't use any sysctl settings.

I listen to it every day for hours, never close multimedia/xmms, use graphics/gimp to manipulate images, etc. and never have a problem with it.


----------



## Nicola Mingotti (Feb 10, 2018)

Trihexagonal , if you don't push it to the limit of eating all memory you will not have troubles.  When it reaches that limit it behaves in an unexpected way. (I still did not understand well, i need more tests)

If you feel like experinmenting try this:
-] Disable the swap and you will see, X stuff becomes much faster.
-] [extra, but related] Firefox has an ugly way to manage memory (IMHO), keep open "top -w" then open 4-5 Firefox windows and many tabs in them. Browse around, let Firefox eat a lot of memory. Then, now, start to kill tabs one by one and keep an eye on "top". What happens in my system is that Firefox frees memory only when i close ALL its windows.


----------



## Deleted member 45312 (Feb 10, 2018)

Nicola Mingotti said:


> -] Disable the swap and you will see, X stuff becomes much faster.


What do you mean by much faster ? All my X stuff are already fast.


----------



## Deleted member 30996 (Feb 10, 2018)

Nicola Mingotti said:


> Trihexagonal , if you don't push it to the limit of eating all memory you will not have troubles.  When it reaches that limit it behaves in an unexpected way. (I still did not understand well, i need more tests)
> 
> If you feel like experinmenting try this:...



As a general rule, if it's not broke I don't fix it. 

I did think it looked like it was starting to use swap a few days ago (I can monitor it through sysutils/gkrellm2), and worried I might have to reboot and lose my uptime, but it seems to have released it and is purring along nicely.

I keep a terminal open running `top` most of the time on my other machines.


----------



## Nicola Mingotti (Feb 10, 2018)

dlegrand said:


> What do you mean by much faster ? All my X stuff are already fast.



If you have every tried changing your disk from ordinary mechianical dish to a solid state and remembre the "WOW, now it s really faster". Ok, that is the feeling when i changed to "swap-off" ! [ Maybe 70% of the disk change offect ] 

I can see the difference expecially in opening mails in Thunderbird, and read one of the many GoogleDocuments I need to keep open all the time in Firefox.

I could make a video with my phone if you are really curious and you don't want to test


----------



## Nicola Mingotti (Feb 10, 2018)

Trihexagonal said:


> As a general rule, if it's not broke I don't fix it.
> 
> I did think it looked like it was starting to use swap a few days ago (I can monitor it through sysutils/gkrellm2), and worried I might have to reboot and lose my uptime, but it seems to have released it and is purring along nicely.
> 
> I keep a terminal open running `top` most of the time on my other machines.



Ohhh, the "uptime", in a period of time i remember i was also very proud of my uptimes 

I will try gkrellm2 ASAP. 

it is true, I also belive in rule "not broken => don't change it !". 
But, if it can be faster, why not ?


----------



## Deleted member 30996 (Feb 10, 2018)

Nicola Mingotti said:


> I will try gkrellm2 ASAP.



You can monitor all kinds of things with it, including /var/log/pflog to see if it starts showing activity. I keep it visible at all times.


----------



## Nicola Mingotti (Feb 10, 2018)

Report of experiment: n#2

ok, guys I made more tests, I create a toy vm and boldly went where many probably have been before

*Objective*: discover how to limit user "foo" to eat to much memory and avoid system crash when in "swap-off".

What I discovered is :
-] Restricting "memoryuse" with rctl is not enough
-] To see what resources must be limited I used this command:
`#> rctl -hu user:foo`
-] There I saw, that to stop "memEater" bofore it eats all i must restrict
"swapuse" and/or "vmemoryuse". If I do that "memEater" gets killed properly
and I receive a message in console. Don't be fooled by the name "swapuse", that is positive
in "rctl" output even if you are into "swap-off" ! 

My *conclusion* is, to be able to use the system safely with swap-off
I would need a restrction on "swapuser + memoryuse < 3500M" for user "foo".
But, reading the "man rctl" I don't think that is possible. 

To understand more I think I would need to read a few chapters of "The design and implementation of the FreeBSD Operating System". But now I'm deep into a web project, I can't do that. So, someone else more experienced must push the cow forward


----------



## Deleted member 45312 (Feb 10, 2018)

Nicola Mingotti said:


> If you have every tried changing your disk from ordinary mechianical dish to a solid state and remembre the "WOW, now it s really faster". Ok, that is the feeling when i changed to "swap-off" ! [ Maybe 70% of the disk change offect ]
> 
> I can see the difference expecially in opening mails in Thunderbird, and read one of the many GoogleDocuments I need to keep open all the time in Firefox.
> 
> I could make a video with my phone if you are really curious and you don't want to test


FreeBSD Host AMD64 ZFS 8Gb RAM + FreeBSD AMD64 UFS Guest in Virtualbox. I tried with and without swap space in guest and didn't notice any meaningful difference, sorry.


----------



## Snurg (Feb 10, 2018)

Just got up and about going shopping, then looking into that more while breakfast.

Imho this is a more-than-critical security and stability issue.
It can break every machine running FreeBSD if some attacker runs an user level process that gobbles up memory.
So this would be an ultra-easy DOS attack vector.


----------



## achix (Feb 12, 2018)

Nicola Mingotti said:


> To understand more I think I would need to read a few chapters of "The design and implementation of the FreeBSD Operating System". But now I'm deep into a web project, I can't do that. So, someone else more experienced must push the cow forward



That's the problem. No one has time to do dedicated testing. Having about 10 projects to run at the same time, I don't feel I can help much here.
Anyways, this morning swap usage was again back to almost 3GB. I *thought* the system was slow, so disabled it, and then decided to actually measure something e.g. web page loading, and so I enabled it again.


----------



## Snurg (Feb 12, 2018)

achix said:


> That's the problem. No one has time to do dedicated testing. Having about 10 projects to run at the same time,


I feel like you. But this issue is critical to me. I have no other OSes as back-up I could use if my FreeBSD installations break due to some stupid program eating up memory and fscking up the filesystem. Must get these `rctl` things work and verify them.
I have written a small malloc based memory stresser now, going to test that on a dedicated test PC after a naplet.
Will have to add SysV SHM then and do some more tests.


----------



## Nicola Mingotti (Feb 12, 2018)

To Snurg and achix, we will do what we can


----------



## PacketMan (Feb 12, 2018)

Is there a way to tell FreeBSD OS to not swap any X based processes (thus all that stays in memory and X stays snappy fast) yet OS can swap remainder of processes?  And if you think a program is using way too much memory then document and submit a bug report I say.


----------



## Nicola Mingotti (Feb 12, 2018)

PacketMan said:


> Is there a way to tell FreeBSD OS to not swap any X based processes (thus all that stays in memory and X stays snappy fast) yet OS can swap remainder of processes?  And if you think a program is using way too much memory then document and submit a bug report I say.



1] If you read "man rctl", and then do some experiments, you will see it is not self evident how to reach this : "not swap any X based processes". BTW, IMHO, the easiest way is to limit a specific user.
Untill now I have not found a way.

2] Yes, in my opinion Firefox memory management is inapproriate, how do I fill "bug?" report to the person who manages the FreeBSD port ? (sorry i'm new)


----------



## ronaldlees (Feb 12, 2018)

IMO FreeBSD could use a real _swappiness_ setting like Linux has.  But, in some ways I see why it doesn't.  For the same reason that FreeBSD doesn't come with a GUI, it doesn't have "_swappiness_".  Actually, it has _swappiness_ in the form of many `sysctl`values that one changes at one's own peril, if one is not a FreeBSD VM expert.

FreeBSD has been optimized as a server system.  So, that's where the _swappiness_ is set, via myriad `sysctl`values.  Do a `sysctl -a | grep ^vm` and see how many there are. They are mostly undocumented, because they're dangerous to set.  You easily end up with an unstable system when you play, so play on an unimportant system.  Linux's swappiness goes from 0-100, with _100 favoring servers_, and _0 favoring desktops_. The default favors servers because it's set to 60 (barely favors).  FreeBSD can be altered to some extent, in the same way, but only by being a VM expert or by taking chances.  As with the GUI,  FreeBSD will let you on your own in this area, as it's already set up for servers, and they're happy with that.

I reread some of the Design and Implementation over the weekend.  It doesn't explicitly correlate sysctl values to how the VM system actually works, so that has to be inferred by understanding the terms and descriptions.  Do we all want to have that level of knowledge about BSD internals?  Even tho that'd be great, it seems we could use a 0-100 thrust control and a couple of idiot lights, for those of us who run GUIs and fluff like that


----------



## Nicola Mingotti (Feb 12, 2018)

Umm, I think this thing of "swappiness" could be useful. Or also a documented roadmap to reduce swapping.

This is my rationale. 
-] FreeBSD is server oriented, no problem, I like it like that by default.
-] IMO, to know well the system you must live with it, so, use it for servers and personal desktop as well. Each day of use on the desktop i discover something that i can re-use on the 
servers. 
-] => There should be a documented path to make it more "desktop friendly", we 
can write this document all together if we discover how to.

For the moment I'm trying to limit a user *total* memory use. In such a way that, even if i would set "swap off", a general user "foo" could not put the system out of memory and crash all.


----------



## ronaldlees (Feb 12, 2018)

Nicola Mingotti said:


> For the moment I'm trying to limit a user *total* memory use. In such a way that, even if i would set "swap off", a general user "foo" could not put the system out of memory and crash all.



+1 - Let us know how your experiments work out.

I think that FreeBSD is very conservative, even for the "server" setting that is currently "stiffly" soft-wired.  They want "hot" processes to get the juice, and to make sure that filesystem stuff is never reloaded, and the paging to other  queues is kept down for IO throughput.  So - it's a server with a network bias.

But, would be nice to have _vm.server_bias=0_ or some such (bad name) - switch for us GUI guys.


----------



## ronaldlees (Feb 12, 2018)

Nicola Mingotti said:


> 1] If you read "man rctl", and then do some experiments, you will see it is not self evident how to reach this : "not swap any X based processes". BTW, IMHO, the easiest way is to limit a specific user.
> Untill now I have not found a way.
> 
> 2] Yes, in my opinion Firefox memory management is inapproriate, how do I fill "bug?" report to the person who manages the FreeBSD port ? (sorry i'm new)



Look at the _vm.overcommit _`sysctl` value_._  I think if you want RLIMIT_SWAP to work, then you need bit one of _vm.overcommit_ set, but by default it is set to zero.  Of course you could still lose data if your user app crashes. 

See:
https://people.freebsd.org/~kib/overcommit/

There may be other side effects of this setting, so you'll have to be wary. I see that literally nobody is doing this (including me), based on web searches, so you'd be on your own relative to the risks, etc.


----------



## Nicola Mingotti (Feb 12, 2018)

ronaldlees, thank you ! I will read carefully later.

I just finished a couple of experiments which I would really like some of you
to confirm because I found interesting news (news at least for me) but also scary news.

NOTE: there experiments where made WITH swap-on. WITHOUT X, only 3 terminals
in a small virtual machine running FreeBSD-11.1.

1] The previous version of "memEater.c" I posted is completely harmless,
I saw that I can (m)allocate more then double of the phisical memory I have.
The system here is smart, he sees that I allocate memory and I am not using it
so it lets me do whatever I want.

1.1] The memory "memEater.c" uses does not appear in "top", but you can see it in `rctl -hu user:foo` under the name "swapuse" ==> temporary conclusion, it is useless to limit "swapuse" because it can grow out of measure, well beyond system capabilities.

2] I wrote a second version of "memEater.c", called "memEater2.c" (it is attached ) which not only allocates big chunks of memory but also [great stuff] fill it with 'a' ! This makes a lot of difference !

2.1] Using "memEater2.c" you will see memory usage growing in `top`

2.2] Using "memEater2.c" you will see "memoryuse" to grow in `rctl -hu user:foo` output under the name "memoryuse"

2.3] => "memoryuse" it is probably only thing which has some sense to constrain.

3] Using "memEater2.c" at the point of eating more memory than it is phisically available in Ram + Swap  an ugly thing happened. "memEater2.c" kept trying to eat stuff, it was not terminated by the system and I lost control of the machine. There appeared an error message in the "root" console saying something about memory, but the teminal was lost.

4] At this point I can confirm Snurg worries: if a malicious (non root) user wants to hang the system he can, simply by allocating a lot of memory and writing garbadge into it.

5] my conclusion is, it is imperative to limit all non root users "memoryuse". I don't know if it is possible but *even better would be if it was possible to reserve some memory out only for root* to use. So if system go into stress, root can still log in and kill who needs to be killed.


----------



## Snurg (Feb 12, 2018)

Just got up and had my coffee.
Nicola Mingotti exactly pointed out a thing what I am worrying about.

A malicious drive-by javascript could just eat up memory and potentially kill the desktop system.
(As I am allergic against Javascript, I use it only when I have no alternative, say, for web pages. So I am not fluent in it.
Maybe somebody else could write a memeater script to check out what happens when such a page gets called. A local file:// page would suffice for that...)

Other issue is that running embedded systems without memory buffer stop it is just crazy and irresponsible, if any serious things depend on them to work reliably.
This gets even more aggravated if they are connected to a IoT network.
Furthermore, embedded things usually have no swap device. On these, it must be possible to turn off swap _entirely_ and replace it with sensible settings restricting memory usage by, for example, `rctl`.

Another consequence of this is that one probably has to add a toor user for FreeBSD, who has all rights as root, except of some memory usage restrictions.

And now I am going to play with my test PC to find out myself how totally disabling swap could be achieved in a safe manner. Will report back...


----------



## Snurg (Feb 13, 2018)

Just played around a bit to see how FreeBSD handles swap. Set up the test PC (40GB RAM, 550GB swap partition, of which FreeBSD apparently uses only 256GB) and played a bit with my quickndirty swapstresser utility to see how the system behaves when memory gets used.

```
% ./swapstresser
swapstresser: Unixoid out-of-memory situation evaluation tool
Usage: swapstresser [-bmdafstrc]
  -b   block size in megabytes to malloc every time (default 1)
  -m   max block count to allocate
  -d   cycle duration in milliseconds
  -a   allocate new block every n cycles
  -f   free a block every n cycles
  -s   if specified, stress memory every nth cycle,
       by accessing each allocated 4KiB page of chosen blocks
  -t   fraction of contiguous blocks to stress (1 = all)
  -r   if given, run this number of cycles, otherwise run infinitely
  -c   continue if allocation failed
%
```
If you want to play with it, download the source and its makefile from Github.

For example, when I allocate memory, I see that this memory usage does not show up in top, until it actually has been accessed. (use sufficiently long cycle times to have time to check this). Nicola Mingotti's according observation is correct. It is easily recognizable by the time it takes when the memory gets first accessed in "stressing" cycles, as these are much slower than the subsequent "stressing" cycles (when the memory is actually allocated and in use).

What I also do not understand yet is, why `top`-displayed RESident memory grows until it hits the full available RAM and then drops to zero, but `swapctl -hgl` only displays swap usage of what exceeds actual physical memory. I.e. the latter displays about 10GB swap usage when `swapstresser` uses 50GB, even when displayed RESident memory hits zero.

From reading the handbook chapter on security restraints, it looks like that *restricting memory usage per user seems only possible by confining the user to a jail*, as it seems to be possible to restrict memory usage per-jail.
This in return means that programs on embedded hardware must be run in jails, if one wants to have some sort of buffer stop protection.

However, I am still having difficulties to get `rctl` work like the handbook and man page suggests.
There are only very few mentions in the forums, creating the impression that `rctl` might be still not very widely known and used. (For example I didn't know about it until Nicola Mingotti mentioned it)
Maybe the default [PMAN=]login.conf[/PMAN] must be changed for making rctl() work?

Edit:
As a compromise for unjailed desktop usage, it then would be interesting to *find out how to restrict the OS that way, that swapping is done only when all physical memory has been used up*.
This way one could avert disaster when swapping sets on, initiated possibly by a mere drive-by malicious ad on a web site.
Then one could save the computer from imploding by using the ctrl-alt-bksp handbrake to zap the X server...

Edit 2: 
The stroke-out statements seem to be incorrect, see my next post below.


----------



## Nicola Mingotti (Feb 13, 2018)

Yeah, the *jail* !  Great Snurg, I guess you discovered the right way to go !

-] I guess I can easily write a memEater running in web browser, in Javascript, no problem.

-] about `rctl`, yes, to have it working it is necessary to put in "/boot/loader.conf"
"kern.racct.enable=1" .

-] An example of swapless system by default. I recently installed a FreeBSD-11.1 into a BeagleBone Black to use it as a wirelss access point. Well, that system by default is swapless [no swap in "top"] ! (i did not make it like that, I chose the default config when i installed AFAIR)

-] For ronaldlees  , I tryed to read the page you found but it is really techincal, I can understand ~ 50% of it. I guess I need to read those few chapters of "The design and implementation of the FreeBSD o.s." before it.

What's next ? The next things I will try are the "jail way", I will write a JS memEater and I will try Snurg "swapstresser". I guess i can do it in the weekend.


----------



## Snurg (Feb 13, 2018)

What is strange now:
I had set `kern.racct.enable` set according to the warning I got when I ran `rctl` for the first time.

There seems to be either some misunderstanding on my side, or the documentation is outdated...
I tested with multiple instances of `swapstresser` with the `-c` option activated, and they together do not exceed the memory usage I set in the setting shown here.
(I apologize for any confusion my misunderstanding caused.)

*Thus it does not seem to be necessary to use jails to make per-user memory restriction work...*

Will have to add SHM support for swapstresser later today to test whether this also applies to SHM.
If this is the case, I'd be very happy, as it won't be necessary to make the system safe against excessive memory usage without having to use jail.


----------



## Nicola Mingotti (Feb 13, 2018)

Snurg said:


> *Thus it does not seem to be necessary to use jails to make per-user memory restriction work...*



Is it bothersome to use/configure "jail" ? I never tried.


----------



## Snurg (Feb 13, 2018)

OT:


Nicola Mingotti said:


> Is it bothersome to use/configure "jail" ? I never tried.


Not really, when you have understood the concept and are good in system administration, as every jail is in some way a separate FreeBSD installation.

The handbook is a bit outdated, it describes the now-deprecated rc.conf jail configuration. And it shows that there is a bit of work involved.
I recommend to read jail(8) and jail.conf(5) as next, they describe the modern methods.
After you have built the base jail, it's easy creating more jails, some copying, setting links and a bit configuration file editing. Thus I think it's no mistake to try the handbook setup first, and use jail.conf if one want to learn/understand dealing with the jails themselves and not with a wrapper...

Many people prefer to use jail wrappers like iocage or ezjail. However, in my impression these are not easier/quicker to grasp, the learning curve seems practically the same. They kind of make you dependent of them, as they have their own configuration files. And, there is some extra work involved which is not described in these docs, like setting up PF etc, and the device+socket communication stuff which desktop apps need, which you have to do by hand anyway.
And, sadly I am too retarded to understand the jail managers' documentation. it was much easier for me understanding the handbook and the man pages.

This all is the reason why I have the impression that only a few gurus in this forum seem to use jails for their desktop apps.

As I am no guru but instead a lousy sysadmin, I want the PF handling automated also, and so I am working on a simple-to-use jail manager that does use only jail.conf and offers an optional curses-based UI in addition to the command line. The last four weeks I have been busy with other things, though. I needed some change. Playing a bit with C was really refreshing after doing only Perl for months  After the thing with removing swap use from my computers I will continue with that project


----------



## swegen (Feb 16, 2018)

I wrote a JavaScript memory allocator to play with. Browsers however have their own memory management and thus may kill the offending page before it can crash the whole system.

https://five.ml/lab/allocm.html


----------



## Eric A. Borisch (Feb 16, 2018)

Here's my little interactive python memory eater...

 `*` doubles memory use
 `/` halves
 `+` increases by 10%
 `-` decreases by 10%
Ctrl-c to exit.


```
#!/usr/bin/env python

import curses
import sys

tmem = 1

curses.initscr()
curses.savetty()
curses.raw()
curses.cbreak()

try:
    while True:
        s = 'x'*tmem
        sys.stdout.write(" {:<32d}\r".format(tmem))
        prev = tmem
        while prev == tmem:
            res = sys.stdin.read(1)
            if res == '*':
                tmem *= 2
            elif res == '+':
                tmem += tmem // 10
            elif res == '-':
                tmem -= tmem // 10
            elif res == '/':
                tmem = max(1, tmem // 2)
        s = None
except KeyboardInterrupt:
    curses.resetty()
```

It does not try to force memory to stay active after allocation, so you will see (once you start to build some pressure) allocation followed by swapping out, and then it should settle (swapped) until you change it again.

Edit:NOTE: This is intended for stressing memory handling on your system and can cause crashes/freezes. Do not use on a production system.


----------



## Nicola Mingotti (Feb 16, 2018)

swegen said:


> I wrote a JavaScript memory allocator to play with. Browsers however have their own memory management and thus may kill the offending page before it can crash the whole system.
> 
> https://five.ml/lab/allocm.html



Well done ! I tested it on my system and it works.

I was able to hang X running it. I did not wait to much and I was able to open a console out of X and kill Firefox, so I could recover the system.

I attach picture of my system hanged. There you can see all the Swap is gone and the firefox process eating all.
I guess this is the demo case Snurg was waiting: *a malicious web application can hang the system. *

Observe that I don't receive any complaints from Firefox, and FreeBSD (by default) does not try to kill the process even if it eats all out.

How do we procted from this is still work in progress


----------



## swegen (Feb 16, 2018)

At least in your case the kernel kept a small amount of free memory so the kernel itself didn't run out of memory. But it might have if you had tried to allocate even more memory.

I have also tested these low-memory scenarios with a C program. When the available memory becomes too cramped the kernel seems to start randomly killing processes and this is the point where the system may no longer recover without a hard reset.

Recovery is much more likely if the swapspace resides in a freebsd-swap partition instead of a ZVOL.


----------



## Nicola Mingotti (Feb 16, 2018)

swegen said:


> At least in your case the kernel kept a small amount of free memory so the kernel itself didn't run out of memory. But it might have if you had tried to allocate even more memory.
> 
> I have also tested these low-memory scenarios with a C program. When the available memory becomes too cramped the kernel seems to start randomly killing processes and this is the point where the system may no longer recover without a hard reset.



Actually I killd Firefox when it was still allocating stuff, more it goes near to the limit of eating all memory more it becomes slow in allocating. So, I knew I had an handfull of seconds available from X first cranks to total machine hang and random killing of processes.

I also tested in C, and in Ruby as well, Eric A. Borisch posted one in Python ... it is fun to write a memory allocator

CAVEAT. Maybe you all know already, but, don't arrive at the limit of eating all memory on a production machine. You can make a big mess into the disk. Try this always in expendable machines.


----------



## Snurg (Feb 17, 2018)

Thank you very much swegen, Eric A. Borisch and Nicola Mingotti for your great input!
In particular, the Javascript allocator is a jewel... good to learn more about Javascript (I should know more about it, for web page stuff...).
(I am not intending to abuse it for crashing client's machines, but will use it to see how different browsers behave  )

With the settings I posted in this thread, I found out that the settings apparently seem to work actually _per user_. It didn't exceed the memory usage, just failed to allocate memory. (tested with -c option using multiple login sessions and conventional and shared memory combined). So either I misinterpreted the manual pages and the handbook or the information there is actually incorrect.

After doing some more tests with swegen's and Eric A. Borisch's memory allocator/stressers on a throwaway installation, I think I'll be confident enough to test this soon on my main PC (which is still on FreeBSD) whether this will actually protect that swapless machines from easily being crashed.
(I had a bad accident a few years ago when I had in the file manager a lot of files marked for copying, and accidentally hit enter. The computer then attempted to opened hundreds of applications, each one for a file, HDD LED became constantly lit and it became unresponsive and had to be reset. Afterwards filesystem showed a lot of errors, but apparently no data lost.)

By the way, when I added shared memory options to swapstresser.c and did some testing, I found out that the defaults for shared memory are quite low for contemporary software use. That could have been the main cause for my problem with KDE a while back ago...)
If you want to test shared memory, too, please download the updated version from above github link. (Btw, There is still at least one bug in the app, it crashed once when I started it when no memory was left to allocate)
When you do so, you'll want to use ipcs() and ipcrm() commands to examine shared memory usage and maybe clean it up afterwards.

Have a nice weekend 

P.S.: Eric A. Borisch, can I use any Python version, or do I need to install a particular version for your script? (Please excuse that stupid question, I have practically no Python knowledge...)


----------



## Eric A. Borisch (Feb 17, 2018)

Snurg said:


> P.S.: Eric A. Borisch, can I use any Python version, or do I need to install a particular version for your script? (Please excuse that stupid question, I have practically no Python knowledge...)



It should work on 2 or 3. To be pedantic, you can either mark it executable and run it directly (will grab python in PATH) or pass it as the first argument to your python executable of choice.


----------



## swegen (Feb 19, 2018)

Snurg said:


> In particular, the Javascript allocator is a jewel... good to learn more about Javascript (I should know more about it, for web page stuff...).
> (I am not intending to abuse it for crashing client's machines, but will use it to see how different browsers behave  )


Thank you. I tried to write it in a self-explanatory manner and included comments in the source to make out what it does. It's easy to test mobile browsers with it too.

I noticed the tuning(7) manpage has explained some `sysctl` settings regarding to swap and memory usage. Some of those could be worth testing to see if they improve low-memory conditions.


----------



## lefsha (Feb 19, 2018)

Few people including me believe, that SWAP is an evil. It's like buying house you don't have money for
or like to paint old VW with gold color pretending you are rich.

The simple truth is, that at some point in life you have to be honest to yourself, when memory is over.
With or without SWAP. Using swap you mostly downgrade your PC by huge factor.

It's only acceptable if app in swap is mainly sleeping ie not in use. It's better to kill it or stop it.
If just 2 apps are concurrently running and do really need memory you end up
with constant load/unload swap pages effectively making PC unusable.

Your PC's memory is your budget - live according your budget. If it's not enough  - buy more memory
or limit your usage.

Just a reminder. RAM is >10x slower than CPU. SSD is >100 slower, than CPU. HDD >1000 times slower.
Plus add huge HDD latency, where you wait 10+ms just to start reading your data.

Think of developers who optimize CPU 1st level cache usage to make your program running faster and now
you put your data into SWAP...  Think of production grade In-Memory Databases broadly available.

...No wonder the new generation is writing programs in Python. Users do not care.

You need SWAP? - Consider to buy Abacus.

P.S. _Modern_ computers and SWAP is a pure joke.  It did only make sense 20+ years ago.

P.P.S. I do not use SWAP more than 15 years. No single issue because of that.


----------



## Nicola Mingotti (Feb 19, 2018)

swegen, "tuning" man has entered my "FreeBSD notes" file a couple of weeks ago

I can not work on this right now, I used the weekend to attach a nasty^3 
network problem it was necessary to solve. 

I propose we could start to write a document to summarize our advancements, or new people will come and will try to re-do all experiments we already made during the weeks.

Do you have any suggestion ? I use a lot Google Documents (requires a gmail account). Other tools for groups editing ? I am not a fan of wiki(s), they are nore a pleasure to edit them.


----------



## achix (Feb 20, 2018)

Hey I got reading to do, you guys have been doing a great job.
Just wanted to say that after 1 week, about 4GB or 6GB out of total 8GB are used (depending on whether this akonadi_notes_agent eats up memory or not).

Talking about Server vs Desktop swapiness, whats the take of PCBSD (now TrueOS) ?


----------



## Deleted member 9563 (Feb 20, 2018)

lefsha said:


> Your PC's memory is your budget - live according your budget. If it's not enough - buy more memory or limit your usage.



I think you misunderstand some of the uses of swap - post 1998.   It is not just to replace ram if there is not enough. In fact I think that is rarely what it is used for any more. It has many other purposes.


----------



## Snurg (Feb 20, 2018)

For some reason the rctl vmemoryuse user limiting settings seems not to cooperate with xorg. It always fails to run xkbcomp then.
And sadly with no restriction on vmemoryuse programs are not restricted to waste memory as they like.

I suspect that the reason is that the racct/rctl stuff was paid work, sponsored by the FreeBSD foundation, for usage on servers to avoid them thrashing swap, resulting in bad response times.
On servers swapping out is usually somewhat restricting memory usage, because at some point the timeout sets in, resulting the threads (and their memory) getting discarded just because of disk activity blocking processes. (You know when you have sometimes to wait looong for a page to respond, or finally failing)
So it looks to me that the racct/rctl might have primarily been not intended for desktop usage.

If you can confirm my tests result, X always failing to start even if sufficiently high limit set on vmemoryuse (I tested with these values at 10GB), what do you think about making a PR?


----------



## lefsha (Feb 20, 2018)

OJ said:


> I think you misunderstand some of the uses of swap - post 1998.   It is not just to replace ram if there is not enough. In fact I think that is rarely what it is used for any more. It has many other purposes.



You might be joking Mr. OJ. Am I right?
Many other purposes for which RAM is not good enough??? WOW! Tell me that secret. I promise not to tell any one else.

P.S. Well, unfortunately I know the secret... The sole purpose of SWAP is to get more money from the customer pocket.
99% people on earth have no clue how computers work. The rest 1% makes money with that business.
I am from semiconductor industry Mr. OJ. So you might guess to what percentage I do belong.

N.B. The proof, that SWAP is not necessary can be done purely mathematically with very simple assumptions.


----------



## forquare (Feb 20, 2018)

lefsha said:


> You might be joking Mr. OJ. Am I right?
> Many other purposes for which RAM is not good enough??? WOW! Tell me that secret. I promise not to tell any one else.



Unless I am mistaken, doesn't the contents of RAM get dumped to a SWAP device in the event of a kernel panic?


----------



## Snurg (Feb 20, 2018)

forquare said:


> Unless I am mistaken, doesn't the contents of RAM get dumped to a SWAP device in the event of a kernel panic?


By default, yes. But it's a joke. Unless you are interested in debugging the dumpfile.
And if you are, the default size of 2GB swap is a joke, too.

If you are not interested in jokes, just set `dumpdev="NO"` in rc.conf.

Edit: Another failed joke I still do not understand is why Xorg refuses to start, messaging me it cannot run `xkbcomp` when I have allocated it a maximum of *10GB virtual memory* when there are still about 45GB physical memory (i.e. RAM) free.


----------



## lefsha (Feb 20, 2018)

forquare said:


> Unless I am mistaken, doesn't the contents of RAM get dumped to a SWAP device in the event of a kernel panic?



1. No one knows upfront, what causes the kernel panic. It might be well an issue with a hard drive.

2. If a hard drive IS working, nothing prevents any program including kernel to dump RAM on disk in a form of file, which can be easier recognized by admin, in case the machine does have an admin attached to that particular computer. On most (99%) of user laptops it is not the case.
The same can be said about any embedded things. So there is nobody who will look at it. And even fewer people are able to understand what it means.

3. Any causal program which is running on a computer from time to time saves its data on disk. None of them requires swap for that purpose.

4. Last but not least. What does word "swap" means? It doesn't mean a permanent data storage, neither a database and nothing else as well.
It does mean swap - an outsource of memory pages based on certain algorithm to another storage space, which is a hard drive for example!
The same page goes back into memory once there is enough room for it OR (unfortunately) it is required for a running program.

Nothing prevents any one to save his favorite movie on swap partition. Nothing prevents a man to shave with a knife. And many did in the past. But it is a wrong usage of the instrument which has been developed for a different purpose.

Am I wrong?


----------



## forquare (Feb 20, 2018)

lefsha said:


> Am I wrong?



I don't think you are wrong, but you asked what the other uses for SWAP were, so I gave you one.  It may not be a useful one to you, or to many other people, but it is a use


----------



## lefsha (Feb 20, 2018)

forquare said:


> I don't think you are wrong, but you asked what the other uses for SWAP were, so I gave you one.  It may not be a useful one to you, or to many other people, but it is a use



If you understood me correctly my message was, that using SWAP for something else means using a wrong tool for a different purpose it was not made for. I could easily create a swap partition and use it for database server as raw storage. It is no way different than your proposal.
It has nothing to do with swap. I could go further, create a swap partition and mkfs on it. That would be my secret storage  everyone else would think it is a swap, but not  it's a special thing... I tell nobody about. I could write a simple program to save on swap partition a blockchain sequence and that way excuse the huge size of that space. If you are creative there may be plenty of possibilities to use swap for something it was not made for.

What I have said at the very beginning is - SWAP as swap is not necessary, more over it is a bad idea, bad instrument serving no purpose.
I can't discuss multiple other crazy ideas what people might use that partition for. It's actually off-topic.

By other usage I was thinking of something related to deal with not sufficient RAM. Instead it's just another security hole. By getting access to swap, I could manage what content will be loaded into memory and... possibly executed. Moreover my virus could survive the reboot...


----------



## Snurg (Feb 20, 2018)

lefsha said:


> That would be my secret storage  (...) It's actually off-topic.


Actually swap, kernel and application dumps are highly interesting forensic stuff.
You might find sweet things inside them the user didn't intend at all to make `strings`-able for "others".

So it might be a good idea to turn dumps off by default unless you are into post-mortem necromancy.
And, if you are really into swapping, turn encryption on.


----------



## lefsha (Feb 20, 2018)

Snurg said:


> Actually swap, kernel and application dumps are highly interesting forensic stuff.
> You might find sweet things inside them the user didn't intend at all to make `strings`-able for "others".
> 
> So it might be a good idea to turn dumps off by default unless you are into post-mortem necromancy.
> And, if you are really into swapping, turn encryption on.



Exactly.


----------



## achix (Feb 20, 2018)

Just upgraded the system with new KDE, firefox, etc. Will keep an eye on how it goes.


----------



## gnath (Feb 20, 2018)

Snurg said:


> So it might be a good idea to turn dumps off by default unless you are into post-mortem necromancy.
> And, if you are really into swapping, turn encryption on.


I as a user turn off core dumps. But SirDice and others said low use of swap in normal use. In early days we used to set swap partition as twice of memory. Now probably we can add swap file as required. Here in freebsd there hardly any surge requirement of swap (my case). Initially there was no swap partition witout any problem. Some body said that physical partition required by OS. I added linux partition for freebsd. But noticed never used during compilation.
I do not know swap allocation is monitored & done by only OS or the running programs/process or both. Freebsd's
foot print is very low Firefox require proportionate to open tabs and also open instances. I think running no. of process
as required will be guiding factor for swap for a particular OS.


----------



## Snurg (Feb 20, 2018)

achix said:


> Talking about Server vs Desktop swapiness, whats the take of PCBSD (now TrueOS) ?





gnath said:


> But SirDice and others said low use of swap in normal use.


Yes. And I find it highly annoying when there is always a small latency because there is some swap access when I switch windows, screens or the like.
Dunno whether there is some configuration on PCBSD resp. TrueOS which avoids this annoying latency.
I mean, swapping out is just idiotic when >40GB RAM are shown unused by `top`.

Anyway, back to the experiments on setting limits to memory usage:


```
user:snurg:memoryuse:deny=10g/user
# if swapuse is set to 0, then user cannot log in anymore
# if it is set to 1M one at least gets "out of memory" message
# if it is set to 1G X fails to start
# user:snurg:swapuse:deny=1G/user
# after disabling swapuse option X still does not start.
# So trying to disable the last three parms:
#user:snurg:shmsize:deny=2g/user
#user:snurg:datasize:deny=3g/user
#user:snurg:stacksize:deny=3g/user
# finally I found that X does not start when this option is set, no matter how high the value. Isn't this crazy?
#user:snurg:vmemoryuse:deny=10g/user
```

When using no X, then these settings work fine and get honored. (except swapuse, but when there is no swap, it cannot be used anyway, so probably no need to set it)


----------



## achix (Feb 21, 2018)

After one day swap seems stabilized. 1.7GB out of total 8GB. Seems like the previous version of KDE had some nasty mem leaks.


----------



## achix (Feb 22, 2018)

Still good after 1 day. *Snurg* Xorg is setuid, so it runs as root. Might have something to do with.


----------



## Nicola Mingotti (Feb 24, 2018)

I add a *reason to run FreeBSD in "swap-off"*, which I just discovered.

It is a few days that I am not switching  off FreeBSD at night,
there are too many windows and desktops open. Re-open, Re-position
all of them on the new day would be a bit boring. So I run a screenlock and
let the system run all night.

On the next day, what I see is that, when I log into the
screenlocker, which is snappy, I find an extremely slow
system. After 5-10 minutes of use the speed becomes regular.

I guess (I don't have data on this by now) that at night
FreeBSD is moving basically all my programs to the swap.

ronaldlees  already pointed out in this thread that FreeBSD puts unused stuff in swap
proactively, to be ready to answer to fresh new requests. [this not a quote, i am rephrasing]
So, Its behaviour makes sense, as is it a "server with a network bias".

But, from a desktop machine perspective, there will be not
new unexpected requests, so it makes sense to turn swap off,
and have the system focus on what the main user whats to run.


----------



## Nicola Mingotti (Feb 25, 2018)

I went I little forward on this subject:

AIM:
1] I want to keep "swap off", but I don't want to loose data or corrupt my system.
2] I don't want to let the system do the guardian of memory limits because, when
those limits are surpassed, AFAI-get it is going to kill
programs choosing them I a way I do not understand.

METHOD.
I wrote a little script (with X GUI) that every second checks the available
free memory using `top`. If memory free memory goes below e.g. 1G
then the script does `swapon -a`, it is allowed by `sudo`.

PROBLEM(s).
1] What is *Inactive Memory* I see in `top` ? I do not undertand.
I found this post but it is very old:
https://forums.freebsd.org/threads/inactive-memory-vs-free-memory.8591/

2] Can I consider "Inactive Memory" a synoim of "dirty memory that can be re-allocated" ?

3] Is there a way I can clean "Inactive Memory" and make it "Free" ?

4] (note) I observed If I leave the computer on all night, in the morning
the amount of "Inactive Memory" is very large.


----------



## Crivens (Feb 25, 2018)

Let me pipe in with some info about why swap is a good idea.

You need some MB to help out when a contiguous memory area is needed but there are some mapped pages in the way. Otherwise the mapping will fail.

The kernel dumps only contain active memory, if my memory serves me right. So a dumpdev of 2GB is enough. Please note that that is not a swap device.

He who trusts a kernel not to wreck the file system after something went terribly wrong needs to take some hard lessons from fate - like finding half your files in /lost+found and containing the string "told you so".

And for Xorg, please check how much virtual address range it maps (framebuffers etc). These also count and grow very fast.

Sorry for being so short here, typing on phone screen sucks a golf ball trough a garden hose.


----------



## Maxnix (Feb 25, 2018)

Nicola Mingotti said:


> I guess (I don't have data on this by now) that at night
> FreeBSD is moving basically all my programs to the swap.
> 
> @ronaldlees already pointed out in this thread that FreeBSD puts unused stuff in swap
> ...


You could try setting *kern.ipc.shm_use_phys=1* to "cause all System V shared memory segments to be mapped to unpageable physical RAM". Look here: https://wiki.freebsd.org/SystemTuning#SYSCTL_TUNING


Nicola Mingotti said:


> PROBLEM(s).
> 1] What is *Inactive Memory* I see in `top` ? I do not undertand.
> I found this post but it is very old:
> https://forums.freebsd.org/threads/inactive-memory-vs-free-memory.8591/
> ...


Help yourself: https://wiki.freebsd.org/Memory


----------



## Nicola Mingotti (Feb 25, 2018)

Thank you for suggestions, 

In short, what I propose is:
1] Turn swap on only when there is less than 1G free memory available. What danger do you see on this procedure ? 

2] In case swap had to be always on I have a second option.
Could I make a memory disk and put the swap there?


----------



## Nicola Mingotti (Feb 25, 2018)

Maxnix said:


> You could try setting *kern.ipc.shm_use_phys=1* to "cause all System V shared memory segments to be mapped to unpageable physical RAM". Look here: https://wiki.freebsd.org/SystemTuning#SYSCTL_TUNING
> 
> Help yourself: https://wiki.freebsd.org/Memory



Thank you for the links. The first one for me is new ! 
I will read it after diner 

The second link no, I know it. That is what raised my doubts actually.
What it is says contrasts with what other people told in this discussion:
https://forums.freebsd.org/threads/inactive-memory-vs-free-memory.8591/


----------



## Maxnix (Feb 25, 2018)

Nicola Mingotti said:


> Thank you for suggestions,
> 
> In short, what I propose is:
> 1] Turn swap on only when there is less than 1G free memory available. What danger do you see on this procedure ?


AFAICT no one, but you could try first setting that sysctl variable.


Nicola Mingotti said:


> 2] In case swap had to be always on I have a second option.
> Could I make a memory disk and put the swap there?


This would be like not having swap, after all. In that case just disable it, and keep everything in RAM automatically.


Nicola Mingotti said:


> The second link no, I know it. That is what raised my doubts actually.
> What it is says contrasts with what other people told in this discussion:


I don't see contrast between them: *Inactive memory: *memory not recently used by userland (your programs), that cannot be used by other programs (I guess each program can access only it's own chunks of it if has some) or used by the filesystem (UFS only?) for caching and is swappable. But maybe someone more knowledgeable will correct me.


----------



## Nicola Mingotti (Feb 25, 2018)

Maxnix, here is the inconsistency I see between the two sources.

---- Wiki states ----
*Inactive* Queue

Inactively (not recently used) used by userland
Not recently used (LRU)
Cannot be re-allocated to other processes
-----------------------

Instead on the (old) thread Beastie says 
----------------------
In inactive memory you may have clean or dirty pages. Dirty pages are no longer in use, may be removed in the near future (after being synced to disk) and reallocated for something else.
----------------------

I have red the first link you posted but I don't think my
issues with swap are strictly related to shared memory. [ to be clear, my phd is not in computer science so I may be boldly wrong here ]. 
----------------------- quoted, only changes in colors, bold, underline.

The* kern.ipc.shm_use_phys *sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on). Setting this parameter to 1 will cause all System V shared memory segments to be mapped to unpageable physical RAM. This feature only has an effect if you are either (A) mapping small amounts of shared memory across many (hundreds) of processes, or (B) mapping large amounts of shared memory across any number of processes. This feature allows the kernel to remove a great deal of internal memory management page-tracking overhead at the cost of wiring the shared memory into core, making it unswappable.
--------------------------------------------

P.S. Colors, bold and underline are for my convenince in finding
important parts. No "shout-effect" intended


----------



## Snurg (Feb 26, 2018)

Crivens said:


> You need some MB to help out when a contiguous memory area is needed but there are some mapped pages in the way. Otherwise the mapping will fail.


So FreeBSD cannot just remap without a swap device??


Crivens said:


> The kernel dumps only contain active memory, if my memory serves me right. So a dumpdev of 2GB is enough. Please note that that is not a swap device.


Is the `dumpdev="NO"` directive unreliable? If so, can I use /dev/null for this? If not, that would be bad for embedded usage.



Crivens said:


> He who trusts a kernel not to wreck the file system after something went terribly wrong needs to take some hard lessons from fate - like finding half your files in /lost+found and containing the string "told you so".


Weren't softupdates introduced to make such less probable?

And, if the kernel wants to crash, does having a dump/swap device save the file system from being corrupted? 
And, if the kernel uses normally, say 5GB, and user memory usage is restricted so that the kernel can grow to, say, 10GB, will a swap of 2GB actually "save" the system instead of just postponing the crash?
*Or will having a swap partition enabled (or even worse, a swap file), just make the system unresponsive while swapping, maybe creating even more mayhem by things like important filesystem metadata updating getting out of sync?*

Of course, on web servers swapping will have a "protective effect" for the system by causing many threads to time out and thus freeing memory by dying off/being removed from memory...  maybe this is the indirect "buffer stop effect" of having swap 



Crivens said:


> And for Xorg, please check how much virtual address range it maps (framebuffers etc). These also count and grow very fast.



X fails to start even the user has 10 gigabytes virtual memory allocated. Apparently nobody knows why. Is the setuid X the problem or is the racct/rctl system broken?

Crivens, do you understand that the statement "you must have a swap partition" still does appears not very convincing to me?

Regarding desktop and laptop usage, I am going to ditch FreeBSD anyway, because its S3 suspend with newer Nvidia drivers meanwhile is broken beyond repair, and with systemDOS even S4 suspend (i.e. hibernate) works just fine.
Additionally the latter's swappiness can be easily configured, without touching undocumented kernel knobs.


----------



## gnath (Feb 26, 2018)

Snurg said:


> (or even worse, a swap file)


+1


----------



## Crivens (Feb 26, 2018)

Okay,  now with some real keyboard...

1) Imagine this, you need some range of physical memory for some DMA activity, but there is one page in the way that is allocated and dirty. With swap, the allocator can drop that one, wait for the laundry, and allocate it when done. The allocation succeeds, but you need to wait some milliseconds. Without this, the allocation might fail. That is why other kernels have things like a memory defragmentation now. No big space is needed, and it might even be a memory disk. But that smells a lot like the duke of münchhausen to me. If the system by now can re-map on the fly - I don't know right now. It has been some time since I last dived into that area and read the source code.

2) A swap file is a last resort, the data has to go trough a file system layer also. Why? Just give it some GB of space in a partition - and maybe do that in all disks you have in the system. Interleaved swap space is better than funneling all IO trough one disk. And even with soft updates you still dice with the dataloss. There is this game where you get handed a revolver with one bullet and you point it at your server and pull the trigger. Now there are models with more than 6 chambers - how many will it take for you to accept the risk? For me, I don't play that game. I prefer the thing to fall over, give me a kernel core to see what screwed up and not mess up the file system any more than it already did.

3) Swap space and crash dump space can be separated. You can run a system without the swap space and still have a crash dump location available. And that needs to be a device, not a file. Otherwise, your swap partition will be used in that case (but not your swap files, if I remember correctly). Savecore will then pick the dump up during reboots and stash them away for later. And once you start developing drivers, this is *very* helpful for you. And one of the first steps in my current job was to enable core dumps in the embedded devices because when the field testers come in and complain about sporadic restarts without any idea of why - then this is what you want.


----------



## Gray Jack (Feb 26, 2018)

There is no way to configure "swapness"? Like, define from, dunno, from 1 to 100,  the level of priority to use swap partition


----------



## Deleted member 9563 (Feb 26, 2018)

Gray Jack said:


> There is no way to configure "swapness"?


The `swappiness` parameter is only for Linux as far as I know.


----------



## Gray Jack (Feb 26, 2018)

OJ said:


> The `swappiness` parameter is only for Linux as far as I know.



That's what I thought, that's why I used quotes, cause there was a very good chance that swapness concept didn't exist on FreeBSD.
But it is a great concept, at least on Desktop area, helps a bunch on responsiveness of the Desktop on Linux, it's maybe the only concept that I miss in other UNIX/UNIX-Like systems


----------



## Nicola Mingotti (Feb 26, 2018)

I made a best effort summary of this thread, it is here:
https://docs.google.com/document/d/1cubX4cRAyk3dyOR5ed4t0dngQrhrUjCec6VXpMl8zC0/edit?usp=sharing

If somebody wants to partecipate in the editing let me know.


----------



## Eric A. Borisch (Feb 27, 2018)

Perhaps of use to this discussion:


----------



## Nicola Mingotti (Feb 27, 2018)

Eric A. Borisch said:


> Perhaps of use to this discussion:



Eric A. Borisch, that discussion is outdated, I have red a few lines and they are talking about "vm.defer_swapspace_pageouts" which does not exist anymore !

I wrote the summary [the post before yours] exactly to prevent people to loose time on already discovered and discussed stuff. It is not a man page, but is a short report of our accumulated knowledge of the last weeks of discussion in this thread.


----------



## Eric A. Borisch (Feb 27, 2018)

Yes, that particular sysctl is gone, but vim.disable_swapspace_pageouts still exists.

I think some of the distinctions drawn in that post (between paging and swapping) are worth noting in this discussion, too.

I personally tune these:

`vm.v_free_target
vm.pageout_wakeup_thresh
vm.v_free_min`

to adjust what amount of free memory the system tries to keep available. Depending on the system, workload, and goal, those may need to go up or down -- test for yourself to see if it impacts the behavior you are concerned about.

Refer to https://wiki.freebsd.org/AndriyGapon/AvgPageoutAlgorithm for what they're doing, or the explore the code itself if you're up for it: https://github.com/freebsd/freebsd/blob/master/sys/sys/vmmeter.h https://github.com/freebsd/freebsd/tree/master/sys/vm

Monitoring vm.stats while the system is “behaving poorly” or `top -SHIz -mio` to see where your IO is coming from can be informative, too.


----------



## Nicola Mingotti (Feb 27, 2018)

Thank you for the links Eric A. Borisch I will give them a shot in the evening,  this stuff become very complex very fast. About code, I can read user-space code (I did it for "top" for the first time a few weeks ago), kernel stuff nope, I never tried, I guess I am not prepared for it.


----------



## Nicola Mingotti (Feb 27, 2018)

New *experimental note*

Today I used the system in swap-off, untill at some moment I was
near to eat all the memory so I turned the swap on.

But surprise, after 3 hours of use the system has not used a single byte of it ! 
I attach a picture for non "belivers"

Since I did not see swap getting used I was a worried it would not have been used at all. So I closed many applications, and kept the minimum open. That is why you see a lot of Inactive memory in the screenshot.


----------



## Snurg (Feb 27, 2018)

Nicola Mingotti, your write-up is really very good 
While looking into whether there is a non-too-complicated way of running X as normal user, I found that I apparently cannot do this.

Because, there is no Nouveau driver available for FreeBSD, and the Nvidia blob seems not to support KMS.

Ubuntu supports non-root X already since 2010, Debian even made non-root X the system default 2015.

So there on FreeBSD there seems no way to use the rctl stuff to limit memory usage if one uses Nvidia, like one can do with cgroups in conjunction with Nouveau on messy systemDOS 

(BTW I had attempted to use a AMD/ATI card listed in the supported graphics cards FreeBSD wiki with the Radeon driver.
But the Radeon driver seems to have quite a lot of nasty issues.
For example it would hang when tearing was turned off in the xorg.conf, fail to resume, fail to switch between xorg and the consoles etc etc.
So switching to Radeon turned out to be no solution.  )


----------



## Eric A. Borisch (Feb 27, 2018)

Nicola Mingotti said:


> But surprise, after 3 hours of use the system has not used a single byte of it !
> I attach a picture for non "belivers"



The _pagedaemon_ won't wake up (writting pages to disk) until `vm.stats.vm.v_free_count < vm.pageout_wakeup_thresh`. If this condition hasn't been met since you've turned on swap, nothing will be _paged out_ into the swap devices. (Unless you have vm.swap_idle_enabled=1, which will _swap out_ (via _vmdaemon_) entire idle processes.)


----------



## Snurg (Feb 28, 2018)

Eric A. Borisch, I am not sure whether this is going the right direction.
The problem is to avoid user programs to use more than a specified threshold of RAM.

Optimally this would include their X memory consumption. But one could just make a generous assumption how much memory X could consume, so that user-available memory can be set accordingly lower.

But X will not start for some reason if there is _any_ setting for vmemoryuse for the user, no matter how big.
achix wrote that it could be related to X being setuid, but this doesn't seem plausible to me.

Why should a setting of 10 gigabytes prevent even a small program like xkbcomp to run, making X fail to start?

Does there exist a reasonable explanation for this behavior, or should it be considered as buggy?


----------



## Eric A. Borisch (Feb 28, 2018)

Sorry, it seemed like there was a more general discussion also going on of how swap/paging is used in FreeBSD, and some ways to influence it.


----------



## Nicola Mingotti (Feb 28, 2018)

Snurg , thank you ! 
From time to time I will try to keep that document updated adding new things emergin here.

About, having X running non-root, I guess you are going face a difficult challenge. When it was achieved in OpenBSD it was commented as "incredible milestone", in 2014, see here:
http://undeadly.org/cgi?action=article&sid=20140223112426

Personally, I am sick of dealing with unsupported consumer hardware. When I use Linux/FreeBSD/OpenBSD for desktop use -- that is all days, since many years, I always run them inside a VMWare virtual machine. And happliy leave those problems to younger programmers

For people, like me, who do not know well how X works and do not know why it needs root privilege here is the explanation:


----------



## Nicola Mingotti (Feb 28, 2018)

Eric A. Borisch , thank you for the links, understanding how virtual memory is managed and make a nice expalanatory document for users (in the style of the HandBook) would be super ! 

Unfortunately, the links you have sent are quite criptic to me, I will need a good amount of study to understand what is written there. I will try in the near future.


----------



## fernandel (Feb 28, 2018)

Nicola Mingotti said:


> Snurg , thank you !
> From time to time I will try to keep that document updated adding new things emergin here.
> 
> About, having X running non-root, I guess you are going face a difficult challenge. When it was achieved in OpenBSD it was commented as "incredible milestone", in 2014, see here:
> ...


Did you ever try HardenedBSD  https://hardenedbsd.org/ ?


----------



## Nicola Mingotti (Mar 1, 2018)

fernandel said:


> Did you ever try HardenedBSD  https://hardenedbsd.org/ ?



No, fernandel, I see its name for the first time now. 
Thank you for letting me know. 

Usually when my maximum concern is security I use OpenBSD.
Indeed, in the base, OpenBSD has important things for me as:
mg (editor, Emacs like), dhcpd, perl [whatever scripting language
for me is fine, I don't like programming in the shell languages]

But I am getting used to FreeBSD ipfw and dummynet so I will
consider also HardenedBSD. 

Moreover, OpenBSD updates very fast and changes (expecially pf)
quite frequently. This is a bit of a nuisance for me. I prefer FreeBSD
approach, from what I undestood till now.


----------



## ronaldlees (Mar 3, 2018)

Nicola Mingotti said:


> I made a best effort summary of this thread, it is here:
> https://docs.google.com/document/d/1cubX4cRAyk3dyOR5ed4t0dngQrhrUjCec6VXpMl8zC0/edit?usp=sharing
> 
> If somebody wants to partecipate in the editing let me know.



I looked at the Google doc.



Nicola Mingotti said:


> 10 FACT. On ARM, especially BeagleBone Black, FreeBSD comes swapless by default.



Thanks for the great outline.  I was looking at #10 on your list.  I think there's another reason, in addition to the one about swap "wearing out" the disks more quickly, for there being no default swap setup on some of the ARM images:

Older USB and even some newer uSD storage devices are so slow that it disturbs the swap system.  If the data can't be swapped fast enough due to slow IO, there is a chance that processes will be "killed off" even though there is technically "unused" swap space left on the slow drive.  So, swap depends on pretty fast drives (hard drive speed).


----------



## Nicola Mingotti (Mar 3, 2018)

ronaldlees said:


> I looked at the Google doc.
> Thanks for the great outline.  I was looking at #10 on your list.  I think there's another reason, in addition to the one about swap "wearing out" the disks more quickly, for there being no default swap setup on some of the ARM images: ...



Thank you ronaldlees, I was actually asking myself why it was so.

I just updated the write up with the latest contributions.


----------



## Nicola Mingotti (Mar 3, 2018)

I wrote a little (in geometry and code) GUI to keep memory always under control while
using X. It is very easy to customize it. 
https://github.com/nmingotti/smem


----------



## zbsd (Mar 4, 2018)

Eric A. Borisch said:


> Yes, that particular sysctl is gone, but vim.disable_swapspace_pageouts still exists.
> 
> I think some of the distinctions drawn in that post (between paging and swapping) are worth noting in this discussion, too.
> 
> ...



Could you give an example of how this works please? Like the method used to configure the values you listed for those parameters. Does it have any effect on processes running as root? This does not involve the zfs file system.


----------



## Snurg (Mar 5, 2018)

Eric A. Borisch said:


> Sorry, it seemed like there was a more general discussion also going on of how swap/paging is used in FreeBSD, and some ways to influence it.


No need to say sorry. Eric A. Borisch. Your input is extremely helpful!
I have to apologize to have been too much oriented in making work safe on desktop...


----------



## Eric A. Borisch (Mar 5, 2018)

zbsd said:


> Could you give an example of how this works please? Like the method used to configure the values you listed for those parameters. Does it have any effect on processes running as root? This does not involve the zfs file system.



You use these with sysctl:
`sysctl vm.pageout_wakeup_thresh=16384` or in /etc/sysctl.conf to set on boot.

These values are in pages (4K), so 16384 = 64MB. Adjust as needed for your system.

My understanding is that the wakeup threshold is when the pagedaemon will be woken up to try to page out (and mark as cache (ready to be freed)) inactive memory. The target is what it shoots for before sleeping again. The _min setting determines when some allocations are delayed, if I recall. Check the wiki page I linked to.

It impacts your entire system. Root, ZFS, etc. I bump them all up from the defaults to try to start recovering memory sooner rather than later (closer to 0.) As with any VM tuning, YMMV; test under your usage case & hardware.


----------



## Nicola Mingotti (Mar 20, 2018)

New data available.

After a couple of weeks of daily use I can state with near to certanity level of
confidence that : almost all my problems with swap were caused by Firefox.

Another factor I recognized is, when OSX is up since days the overall performance
of the VMWare virtual machine (in this case FreeBSD) deteriorates badly. Switching the VM on/off makes no effect. An OSX reboot is necessary. 

Finally, the only other thing I toggled is, I increased swap space from 1G to 4G.
I don't know if this may had an influence, I had before and now 5G Ram for FreeBSD VM.

It seems all my swap problems are gone !  

P.S. my OSX is 10.11.6, Apple does not let me upgrade.


----------



## ralphbsz (Mar 21, 2018)

Nicola Mingotti said:


> almost all my problems with swap were caused by Firefox.


Buggy software 1, FreeBSD 0.

Next match please ...


----------



## Mage (Feb 11, 2019)

I’ve been using FreeBSD for many years. From day one, I used only ZFS and swap on ZVOL.  It used to be okay.

Lately, as my projects were growing on the servers, I started running into the problem when the ARC is like 28G, but FreeBSD starts swapping (not only a percentage, also "in" and "out" all the time). The system slows down. I lowered the vfs.zfs.arc_max. The result is disturbing. I either have 4-10G free (not used) RAM on the servers or swapping occurs once in a while.

I decided to test turning off the swap before I do it on production. I’m still running the rest while writing the post. What I have seen so far, it surprised me. My super scientific test method:


```
dd if=/dev/random of=1.file bs=1m count=2k
dd if=/dev/random of=2.file bs=1m count=2k

screen:
  for i in $(seq 0 100); do echo $i; xz -9e -T 4 -k -f 1.file; done
  for i in $(seq 0 100); do echo $i; xz -9e -T 4 -k -f 2.file; done

irb:
  size = 500; a = [0] * (1024 ** 2 * size); nil

top -o res
tail -f /var/log/messages
cat 1.file 2.file > /dev/null # to grow up ARC
```


The irb is a Ruby console. It allocates a huge object. I can control how much memory it takes by the "size" variable. In the VM I used for the testing when the two xz processes maxed out their memory usage, Ruby would use up the rest of the memory somewhere size = between 400 and 500.

Setup: VM, FreeBSD 11.2, 16G RAM, 8 cores. Most settings are FreeBSD default.

Result: 
When there is a swap partition (8G ZVOL), FreeBSD 11.2 behaves like a mass murderer. It kills processes on sight.  For example:

* Without the xz processes running,  if the irb process’ memory usage would get near (but not that close) to the size of the ram, the kernel kills the process without using a considerable amount of swap, and without lowering the ARC to the minimum. Yeah, why not? He is the boss.

* When the two xz processes were running and used up about 4-5G RAM each, the kernel killed either the irb or one of the xz processes at about 6-7% swap usage. Maybe once I could go up to 60%, but most of the time a process got killed below 10%. (It might have made a difference whether used irb or xz first.)

* At the last test run, the kernel didn’t kill anything for a while. Neither it did anything. It was not frozen as I could type in the ssh terminal. But it couldn’t execute even an ls. When this happened, the top stopped refreshing. In this test run, I added the swap on the ZVOL only after logging in. I turned off the "org.freebsd:swap" on the ZVOL and rebooted. It has been stuck already for half an hour. There is a lot of free swap.

When I turned off the swap:

The kernel became peaceful. It went to the end to not kill anything. It emptied the Laundry (when there was one), the Inact, and most of the Wired. When it reached the wall, it slowed the execution of the processes. Instead of 400% CPU usage per xz process, they went down to 100% or even 0%.

When the above happened, the system still was responsive. The terminal with the top was refresing. It wasn’t usable though. I couldn’t even quit irb to free up the memory. 

Conclusion: 

When there was no swap in the system, it was terrible to run out of memory.

When there was a swap in the system, processes got killed way before the swap filled up. Other times (not in the test but on production servers) the system was swapping when there was a lot of ARC used. Without swap, ARC would have been lowered. 

Once, when I ran out of physical RAM but I used almost zero swap, the system became and stayed unresponsive.  Only a hard reset helped.  It was worse than without swap and running out of memory.

I would say, the swap was used when it shouldn't have been used. It slowed down the system for nothing. Other times, the swap wasn’t used when it could have been used to save the system from a freeze. With swap, the kernel also much more likely killed processes even when, I thought, there was no reason to do so.

After I post this comment, I will turn off swap on all my servers, and remove the limit of the ARC.


----------



## Crivens (Feb 11, 2019)

Could you try to enable compression on the zvol, or swap on a physical device?

And you may need to check the block size of the ZVOL, swapping is done using 4k blocks. I can't look up the settings on my machine as I am not at home.


----------



## abishai (Feb 11, 2019)

I have even worse experience with swap and VMs. After a day or two all VM memory is moved to swap. My server has 64GB and 32 GB swap and usually, the layout looks like 28GB of used swap and everything is ARC. Not only VMs are slow, but I have OOM events as well. Not sure what to do.


----------



## CyberCr33p (Feb 11, 2019)

Maybe locking shared memory into RAM helps:

`sysctl kern.ipc.shm_use_phys=1`


----------



## Mage (Feb 11, 2019)

Crivens said:


> Could you try to enable compression on the zvol, or swap on a physical device?
> 
> And you may need to check the block size of the ZVOL, swapping is done using 4k blocks. I can't look up the settings on my machine as I am not at home.



The block size was 8k. I never touched it. I guess it’s the default. However, I used to have a few recordsize=8k (and even 4k) file systems due to wide-spread advice. It was a horrible mistake. Later, when I tested it, I could not find a single scenario where a non-128k recordsize was better than the default 128k. I tested it with PostgreSQL, Exim, and various logfiles. I tested it with and without compression.

As for putting the swap on a physical device, I have no interest in it since almost all my production servers are running "in the cloud". Even if I wanted to test it (I don’t), there isn’t a single computer on Earth which fulfills all of these: has a physical drive, has FreeBSD on it, I have access to it, it’s not a production machine.

By the way, many years ago, the main reason why I migrated to FreeBSD was ZFS. The main reason I migrated to ZFS was the checksumming. I would never put swap on a physical drive under FreeBSD. It makes zero sense to have checksums on the disk if the swap is not checksummed. I would compare it to putting ZFS on a physical raid. No, it’s worse. ZFS on a physical raid at least would notice if there was an error.


----------



## Mage (Feb 11, 2019)

abishai said:


> I have even worse experience with swap and VMs.



Somehow I could understand Crivens’ desire to make me test it on a physical disk. I like physical servers. Pricing is better. Performance is better. I even like to know that there is somewhere a machine that’s mine. But there is a point when the flexibility worth more than the performance advantage.

The point is, if the production is running in a VM, the best place to run tests is a VM.



abishai said:


> After a day or two all VM memory is moved to swap. My server has 64GB and 32 GB swap and usually, the layout looks like 28GB of used swap and everything is ARC. Not only VMs are slow, but I have OOM events as well. Not sure what to do.



Turning off the swap?


----------



## abishai (Feb 12, 2019)

Swap is absolutely mandatory in case of ZFS filesystem if ARC size is not limited: in case of memory usage spike from userland apps, ZFS subsystem has good chances to die before it shrinks ARC.


----------



## abishai (Feb 12, 2019)

CyberCr33p said:


> Maybe locking shared memory into RAM helps:
> 
> `sysctl kern.ipc.shm_use_phys=1`


Is VM memory considered as System V shared memory ?


----------



## Mage (Feb 12, 2019)

abishai said:


> Swap is absolutely mandatory in case of ZFS filesystem if ARC size is not limited: in case of memory usage spike from userland apps, ZFS subsystem has good chances to die before it shrinks ARC.



I understand what you say. The only question is, why my testing showed me the opposite of this.

It’s a bit similar to the recordsize. I took common advice about ZFS. Later, when I tested it, I experienced the opposite. Okay, this is not limited to ZFS but the life on this planet.

My testing methods were clear. Although there was a bit self-irony in the "super scientific" labeling, two things I often/all the time run on my servers are Ruby and xz. As my production servers are running in a VM, I ran my tests in a VM. I did it multiple times.

Maybe there are cases when you are right. However, I know that at least there are cases when the opposite is true. I don’t get it why unbacked opinions spread on technical forums.

I ran through the five pages of this thread before I posted here. I didn’t read all posts but at least I looked at them. It’s like 70% of the posts are telling that if one runs FreeBSD without a swap partition, the Universe will collapse into itself, or at least the dimensional gates will shatter and release unholy creatures to destroy life on Earth.

And maybe 30% of the posts say they turned off the swap, and it’s either fine or better.

I’m busy too like many people but it’s still possible to spend an hour on testing before I post something that might affect others’ decisions. Besides, I have seen my production servers swapping while the ARC was dozens of GBs.


----------



## Mage (Feb 15, 2019)

Update: after I removed the swap and the limit of the ARC, one of my servers were running near 100% CPU usage all the time. The other servers also exhibited increased CPU usage.

At first attempt, I didn’t turn on the swap but I put back the limit on the ARC. Right after I did it, the average CPU usage went from 98% to 40%. It has been running like that for more than six hours.

Maybe it’s not the swap that has an issue but the ARC. It doesn’t mean that swap is needed. It’s interesting that the impact on the servers varies. It was the worst on the server that has the most memory. These servers run the same software and are under similar workloads. They are running web and PostgreSQL. They have tons of RAM. Most of the RAM is used by the OS for caching either as ARC or buffer cache.


----------

