# Excessive swapping while using synth



## scottro (Dec 27, 2016)

As marino@ has pointed out, the question and information will probably be lost in the 35 plus page synth thread, so I've taken the liberty of copying my original post, and the responses.
pquote]
*Synth using memory till swap is full*



I see that someone had this issue back with FreeBSD-10.x, where synth started using swap at an unexpected rate.  https://forums.freebsd.org/threads/56171/  Lately, this has started happening to me on 2 FreeBSD-11 systems, both with reasonably good processors, zfs, and 8 GB of RAM.  Adding a second swap partition didn't help.  This seems to happen when building large packages, such as libreoffice, however, using portmaster or just doing make install from a port doesn't do this--it will use a lot of CPU, but swap stays at unused or just a little bit used.  This happens when I run synth prepare-system.  I haven't seen any mention of it save for that one thread I linked, so it may be one of those Just Me(TM) things, but I do see it on two different machines.  I see swap gradually climbing till it reaches 100 percent and the machine becomes unresponsive.  Logs show things like

```
09:28:47 s kernel: swap_pager: out of swap space
Dec 26 09:28:47 s kernel: swap_pager_getswapspace(5): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s last message repeated 6 times
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
```
[/quote]
marino@ responded

There are 2 possibilities:

you've set too many builders + jobs for the resources you have for every case
you've got a leak somewhere. Something is using but not releasing swap space. grep log directory for watchdog failures to see if those are happening (they really should never happen).
I answered


For what it's worth, I used defaults, and looking through logs I don't see any watchdog errors. This may mean more work in tracking down a memory leak than I have the ability (or time) to do right now.

@fernadel posted

I had the same problem still, now on FreeBSD 11-RELEASE. Not many times but it happened.
Not so long I use Synth and at the same time it built LibreOffice and Blender. SWAP raise to 36% and after LibreOffice was done dropped to 13.6%. And when was everything done swap stayed on 13%??

abishai posted
8Gb of ram is not enough for synth on my system. When 2 builders start to build Firefox/Thunderbird it starts swapping heavily and eventually system crashes. I disabled tempfs for workarea in Synth config and everything runs smoothly after that.

marino@ replied
how much swap do you have? maybe you find a cheap 64gb or higher SSD and use that (assuming we're not talking about a space constrained machine like a laptop here).

The original problem was likely your *swap* partition was way too small, but if the swap is a spinning disk, it's not much better than just turning off tmpfs for the workspace.

alternatively, you can shift the synth build root to an SSD partition so that building without tmpfs won't be such a big performance hit.

And I posted
Today I started a synth update, but first tried limiting vfs.zfs.arc_max from its default of RAM minus 1 GB to about 4 GB.  I think something is leaking memory, but don't have the expertise to track it down. In my case (on a ZFS mirror in one case, on a single ZFS disk pool in another--haven't tried on that machine yet), it may have fixed the issue, with swap, so far, staying below 30%, though there are still many packages left to go. I'll update when the build is complete.


So, putting this in this new thread (some posts above have been somewhat edited) in case others run into the problem as there are two possible fixes listed, abishai and mine.

TLDR; From the synth thread, I posted about swap getting completely used, and I've posted the responses here, which include two solutions that have worked for two individuals.


----------



## scottro (Dec 27, 2016)

Both machines with the issue have 2 GB of swap.  On the second machine, I did try adding a 4 GB swap partition as well, but had the same problem.   Swap usage during the build I'm doing now (with limitation on ZFS ARC) has risen to 65 percent of the 2 GB, and I'm not sure if there will be a libreoffice build on this run, which seems to frequently be the culprit.  I should also mention (going back to marino@'s last post, that the other machine where this happened is on an SSD, though a fairly low end one.


EDIT:
Well, limiting ARC didn't help.  So, as I saw swap climbing up over 90% I added a swap partition, 8 GB this time, and it seems to be working.  I still think there's a memory leak somewhere, but as a workaround, this will do for now.


----------



## fernandel (Dec 27, 2016)

I have 8 GB of RAM and my swap partition is 3.6 GB. Should I resize swap partition.
As I wrote yesterday swap was used 36% when Synth build LibreOffice and Blender at the same time but I didn't run anything. Usually I am running GNOME 3 or Fluxbox and using GIMP or surfing with Netsurf or Firefox...


----------



## marino (Dec 27, 2016)

those swap partititions are too small for building.  I would say you need a 4:1 ratio between ram and swap (so 4G ram requires 16G swap), etc.   With cheap SSDs, you really could just throw a 64Gb fast swap at the problem as a permanent solution.

Less than 1:1 between ram and swap is not going to work.  Don't be stingy here.

Added: This was directed as scottro@ but applies to fernandel@ even better.


----------



## scottro (Dec 27, 2016)

See my edit, I think you're right.     Again, thanks for your always quick responses on synth issues.


----------



## ASX (Dec 27, 2016)

I have observed that when building some of the mentioned packages (libreoffice, firefox, thunderbird, ...) RAM+SWAP usage reach approx. 4 or 5 GB for each builder (using tmpfs, of course), sometimes more, very approximately half for local base and half for work area.

That's a raw estimate but enough do some basic calculation about how many builder your system can support, with or without swapping.

Also, I noticed that disabling use of tmpfs result in a much increased disk I/O (understandably) and usually a performance penalty, not much because of disk throughput, but more because of disk concurrent access (read & write from each builder and additional I/O if you use ccache, like I do.), that's why an SSD help a lot.

Where possible I would favor swapping over turning off tmpfs, because that is usually faster than ZFS I/O or journaled UFS I/O, also use of tmpfs allow for much faster "deinstall/cleanup" of the work areas.

If you are forced to turn off tmpfs a NON journaled UFS partition will work faster than a journaled one, local base and work area are temporary, you may safely avoid journaling.


----------



## fernandel (Dec 27, 2016)

I decided to enlarge swap file because mine FreeBSD is installed on iMac and add SSD disk is not cheap and iMac is 11,1.
`gpart show`

```
34 1953525101 ada0 GPT (932G)
34 6 - free - (3.0K) 
40 409600 1 efi (200M) 
409640 1216587112 2 apple-hfs (580G) 
1216996752 1269536 3 apple-boot (620M) 
1218266288 1024 4 freebsd-boot (512K) 
1218267312 727710720 5 freebsd-ufs (347G) 
1945978032 7547102 6 freebsd-swap (3.6G) 
1953525134 1 - free - (512B)
```

I never resize partition and my idea is to use OS X boot camp and take sam space from apple-hfs, than use gpart and delete swap file and make the new larger one or I am doing wrong, please?


----------



## marino (Dec 27, 2016)

I would be nervous about resizing partitions too.  
If you can't add a second drive easily (or cheaply) then maybe you just have to keep tmpfs turned off.  

alternatively, you can create a second synth profile that doesn't use tmpfs and just build libreoffice and other monsters with that new profile and use the old profile for everything else.  That would be a good compromise.


----------



## ASX (Dec 27, 2016)

fernandel said:


> I decided to enlarge swap file because mine FreeBSD is installed on iMac and add SSD disk is not cheap and iMac is 11,1.
> gpart show



Or you could just stay with the current disk layout.



fernandel said:


> have 8 GB of RAM and my swap partition is 3.6 GB. Should I resize swap partition.
> As I wrote yesterday swap was used 36% when Synth build LibreOffice and Blender at the same time but I didn't run anything. Usually I am running GNOME 3 or Fluxbox and using GIMP or surfing with Netsurf or Firefox...



It appear your system used approximately 1.2 GB of swap, turning off tmpfs for only local base (and leaving it on for work area) will give you enough free RAM to run the other apps safely. Or, follow the advice from marino@ of course.


----------



## marino (Dec 27, 2016)

ASX said:


> It appear your system used approximately 1.2 GB of swap, turning off tmpfs for only local base (and leaving it on for work area) will give you enough free RAM to run the other apps safely.



Right, but then TMPFS is never used leading to a very noticeable performance penalty.  He has a swap issue < 1% of the time, so removing tmpfs 100% of the time is a big penalty.


----------



## ASX (Dec 27, 2016)

marino@ said:


> He has a swap issue < 1% of the time, so removing tmpfs 100% of the time is a big penalty.



Of course, I agree that where possible tmpfs should be used, but I'm not sure it is only 1% of the time, quite a few packages require similar resources (on a average desktop system)., libreoffice, thunderbird, firefox, virtualbox, the webkit series, texlive-texmf,  ... to mention some.

I can also see some difficult in identifying which one are those packages ... I can remember some of them now, after running Synth several times, but that is only after you already built them and monitored the resources usage.


----------



## marino (Dec 27, 2016)

it's not if swap is used.  It is if the activity of building the package uses 100% of the swap.  Out of 27,000 packages, the number of packages that require that are probably less than 0.1%.  I'd bet it's less than 20 packages.

Added: obviously systems with absurdly low levels of swap are excluded here.  3.6G seems reasonable at first glance but it's simply insufficient for multi-builder/multi-job profiles in all cases.


----------



## fernandel (Dec 27, 2016)

I will leave as is. I have a problem just when I am building LibreOffice otherwise works pretty fast and it doesn't bother me whatever I am doing. I have concurrent builders and jobs per builders 2, 2, tmpfs work area and localbase true and I am using ccache too.
Thank you.


----------



## ASX (Dec 27, 2016)

marino@ said:


> it's not if swap is used. It is if the activity of building the package uses 100% of the swap


Please do not take me wrongly, my previous answer was referred  to fernandel's context that was "building while using some other apps". (to me that imply minimize swap usage).


----------



## scottro (Dec 27, 2016)

Well, even adding gigs of swap (on a ZFS mirror) didn't stop it from eventually, while apparently building firefox, thunderbird, and libreoffice simultaneously, locking up the system.  I may play with the config file to limit the number of builds (I think it's 6 on that workstation.)   I did create enough swap to give me RAM*4 as marino@ suggested, but it wasn't enough. :-(

I may try again tomorrow with a 64 GB swap partition, or, as I said, limit the number of builds.


----------



## abishai (Dec 27, 2016)

On my laptop I have 1.5GB swap and 8GB ram. So, for me, the situation looks normal as now I have 2 builders instead of portmaster's one.
As for tests, it is not nesessary to change partition layout at all. One can create swap file for synth and delete it after it finishes.
I compile FreeBSD kernel on 1 buck cheap 128MB Ram VDS with

```
truncate -s 10G /usr/swap0
mdconfig -a -t vnode -f /usr/swap0 -u
swapon /dev/md0
```
The effectiveness of such swap is lower, but it's swap after all.


----------



## fernandel (Dec 28, 2016)

I had 672 port to rebuild. I run GNOME3. I start Synth before I went to bed and results are:
Time: 5:56:42
Swap: 22.7%
`swapinfo` shows:

```
Device          1K-blocks     Used    Avail Capacity
/dev/ada0p6       3773548   383532  3390016    10%
```
Everything was okay, no errors. They were also LibreOffice, Firefox...webkits.


----------



## scottro (Dec 29, 2016)

While I don't have a complete answer at this point, it seems that if I first build libreoffice with `synth install editors libreoffice` everything else goes fairly smoothly.  I would have to try a few times to be sure, but for now, that's the solution I'll use with synth, as well as temporarily adding a lot of swap.


----------



## ASX (Dec 29, 2016)

scottro said:


> I may play with the config file to limit the number of builds (*I think it's 6* on that workstation.) I did create enough swap to give me RAM*4 as marino@ suggested, but it wasn't enough. :-(


I think 6 builders are beyond the capability of an 8 GB RAM machine, (that's the RAM size you mentioned in your first post).

From a performance point of view you should not make your system swap most of the time. (swap access even on SSD is thousands times slower than RAM, as you surely know).

Consider also that large packages will take longer to build, therefore they have a natural tendency to line up and build simultaneously, thus maximizing memory requirements all at the same time.

2 builders is the maximum I would suggest on 8 GB RAM (3 or 4 builders might work too but in my opinion will degrade the overall performance), if your CPU has many cores increase the number of jobs but keep down the number of builders.

To recap, configure the number of builders depending on available RAM, and configure the number of jobs depending on available CPU cores.


----------



## scottro (Dec 29, 2016)

Thanks, I'll keep that in mind.  The only times I've run into trouble though, were with libreoffice being one of the packages.  It's 9 cores, so I am guessing that that's why synth.ini had 6 as the number of jobs (the man page says default is 75% of the cores).


----------



## ASX (Dec 29, 2016)

scottro said:


> Thanks, I'll keep that in mind.  The only times I've run into trouble though, were with libreoffice being one of the packages.  It's 9 cores, so I am guessing that that's why synth.ini had 6 as the number of jobs (the man page says default is 75% of the cores).



I had to re-read the man page and do think that percentage refer to systems where RAM is very large, because it also state:


> Generally memory is the limiting resource when tmpfs is used


... and tmpfs should be used nearly always ...

The description of "Max Jobs per builder" seems to agree with my view:


> If memory is constrained, it's often a better performance tradeoff to reduce the number of builders and increase the number of jobs per builder.


----------



## fernandel (Dec 29, 2016)

As I understand that is okay that I live my system as is and don't thinking about resizing swap partition.
BTW LibreOffice build with ccache 28 minutes (last time).


----------



## ASX (Dec 29, 2016)

fernandel : yes


----------



## scottro (Dec 29, 2016)

Bah, I said 9 cores, I meant 8 of course.  (See what I did there--cores, course, nudge nudge).  So, at present, the only time I've had a problem is when libreoffice was building with something else.   I think I'm going to try resetting build number from 6 to 2 or 3, but of course, may not remember about this whole issue until it bites me again. 

Many thanks to all who have replied, and to marino@ who no only replied, but gave us synth and quickly replies to every query about it. 
I was thinking of marking this solved, with the solution being to limit builds in  /usr/local/etc/synth/synth.ini, but I'd like to wait till I try it again.  It seems the most likely solution, perhaps combined with adding temporary swap for an upgrade.


----------



## PacketMan (Jan 2, 2017)

ASX said:


> I think 6 builders are beyond the capability of an 8 GB RAM machine......
> ....
> Consider also that large packages will take longer to build, therefore they have a natural tendency to line up and build simultaneously, thus maximizing memory requirements all at the same time.
> 
> ...



Bullseye!

My builder server is 8GB ram, 4GB swap (gonna enlarge it I'm pretty sure based on this discussion). I too have decreased the defaults from 6 builders x 4 jobs, to 4x3. I've also tried 3x4, but I am still undecided.  One thing I am doing, is when I see a hog port show up on the display I do a `ctrl-q`. This causes Synth to not start any more new builds. After the hog port is built, I then restart synth, using the exact same command that was previously launched. But that requires watching the machine.  I'm still tinkering, but 4x3 seems to be a sweet spot.


----------



## scottro (Jan 2, 2017)

Today, getting a bit of lagging when it built firefox, thunderbird, and chromium at once. I now have builds limited to 3, but when it was those 3 it lagged (though didn't freeze) the machine. I left jobs per builder at its default of 4.  (I did add a large swap partition before starting, and it hasn't seemed to get close to overwhelming swap--I chose the RAM*4 that marino@ has suggested, 32 GB, so this combination seems to be working for me, though not perfectly.


----------



## ASX (Jan 2, 2017)

PacketMan said:


> I too have decreased the defaults from 6 builders x 4 jobs, to 4x3. I've also tried 3x4, but I am still undecided.


Assuming your system has 4 cores, try with 3x4 or even 3x3, with 8 GB RAM + 4  GB swap you may still need a bit of luck to not run out of that resources, where two buiders will be probably the "safe" setting.

Synth, by using tmpfs, make large use of the RAM, and that's good, the drawback is that when the RAM is limited there is no room for 'disk cache', and that is bad, and even worse when forced to make extensive use of swap, the extreme case being running out of swap completely.

Fact is that many (most) packages will require little RAM (that is on average 1.25 GB x builder, as desumed from Synth man page), but some will require more RAM, in those case the system will fallback to use disk I/O. (tmpfs filesystems are backed from swap).

Synth build up a clean environment for each packages to be build, (local base), and to do so must read the base files from disk and write them to tmpfs, I would expect that very often the base files are the same for each port and fetching them from the disk cache would be much faster than reading from disk, similar reasoning for build dependencies, that's why in my opinion could be better to keep down the number of builders and possibly allow for more disk cache.

At the end, the whole job might be faster by using less builders.


----------



## PacketMan (Jan 3, 2017)

ASX said:


> At the end, the whole job might be faster by using less builders.



Maybe.  I find you get through the few big ones, but then you got say 50 or 100+ small ones and you are only using a fraction of the resources available. For a machine that is in production that might be perfectly desirable. For me however on a machine that is dedicated for only building ports/packages it seems to be of a balancing act. I could go buy lots more ram for an older machine, or just live it with.  Marino had a good suggestion. Build your known big ones using less builders/jobs and then do the rest of the smaller ones using more builders/jobs. At the end of the day, something like 3x3, 4x3, 3x4 seems to be a sweet spot, and even if the machine has some idle resources, the outcome is still way better than what we had before with ports-mgmt/portmaster.

Just for testing and learning purposes only, I created a 30GB swap file on my builder. And yes its on the same spinny disk as the OS. Gonna try and pound the machine hard with a few big build jobs just to see what happens.  I seem to learn best when I get experimental. I'll buy a newer machine before I put any money into the older ones.


----------



## chrbr (Jan 3, 2017)

PacketMan said:


> Just for testing and learning purposes only, I created a 30GB swap file on my builder. And yes its on the same spinny disk as the OS. Gonna try and pound the machine hard with a few big build jobs just to see what happens. I seem to learn best when I get experimental. I'll buy a newer machine before I put any money into the older ones.


Replacing the spinning disks by SSDs is as day and night. The invest is little and the effect is huge. It should increase the speed also if you just use a very small SSD and use if for temporary data as ccache or swap.


----------



## PacketMan (Jan 3, 2017)

Yep know that. I was answering the question before it was asked.


----------



## rigoletto@ (Feb 7, 2017)

Hello!

Based on what was discussed here before, ports-mgmt/synth configured to run 4 builds and 8 jobs would be appropriated to (try to) avoid the use of swap on a system with 8 cores and 16GB ram?

How much swap would be recommended for that system/configuration?

Thanks!


----------



## marino (Feb 7, 2017)

you can't really avoid using swap when you hit the giant ports like chromium and libreoffice.

8 jobs per builder is lot with 4 builders.  You might be better served with 4-6 jobs per builder.

I've been throwing out guidelines of swap to be 4 times RAM, so you'd probably want around 40 to 64 G swap, especially with the 32 jobs you're proposing.


----------



## rigoletto@ (Feb 7, 2017)

marino@ Sorry, for some reason the jobs per builder option was in my mind as total jobs. What I was indeed proposing was 4 builds and 2 jobs per build.

Thank you!


----------



## PacketMan (Feb 7, 2017)

I have played with these numbers a lot of a machine that I use as a dedicated builder. It does nothing else other than synth.  My sweet spot seems to be 3 builders by 4 jobs.  I tired 4x3 too.  In the end there doesn't seem to be too much difference; all runs well until you hit the big ports like Marino mentioned. Only thing gonna handle that is lots of virtual memory (ram + swap).

My machine is 8 core with 8GB ram.  Might try and add 8GB more, but its an older machine so just might keep my money for a new server.


----------



## rigoletto@ (Feb 7, 2017)

So, the number of builders are what will determine the amount of memory needed and not of the jobs, am I correct?

Currently, using normal/single build system with 8 jobs, I do need about these amount of memory:


```
Chromium - 10GB
Firefox - 4GB; 8GB if using debug
Libreoffice - 6GB
```

If I do 4 builds and 2 jobs, and hypothetically have 48GB of memory, I would not need the swap in any case. Correct?

Thanks!


----------



## marino (Feb 7, 2017)

both do.  each builder has to provide a localbase and working area (both could be in the gigabytes).  Each job needs memory to compile.  The builders are the big ones though.

with 4 builders and 2 jobs and 48G RAM, you'd probably never touch swap.  You could probably add several more jobs per builder.
You're really only sizing for the monster ports.  95+% of ports probably never get close to consuming all the memory (assuming synth is only thing running)


----------



## PacketMan (Feb 8, 2017)

marino@ said:


> You're really only sizing for the monster ports.  95+% of ports probably never get close to consuming all the memory (assuming synth is only thing running)



When I build x11/gnome3 the swap on my machine hardly get used, until you hit the big ports.  Then the gears grind and smoke blows out the back. My machine absolutly could use a lot more ram, but with the 40G of swap it clunkers through it. I just `synth just-build x11/gnome3`, walk away and then later do the pkg upgrade on the GNOME3 machine. Their home use machines; not enterprise production.


----------

