# pkg-static eats alot of cpu



## Alain De Vos (Apr 12, 2022)

poudriere, make make-depends, pkg-static consumes alot of cpu. Playing youtube videos is interrupted i.e. an ULE scheduler issue ?
As even "nice -n 20" does not fix it.


----------



## grahamperrin@ (Apr 14, 2022)

Building which port? 

For multimedia/webcamd, here, after a build time of around eight-and-a-half minutes, packaging takes three seconds without excessive use of the CPU:


```
…
[00:00:56] [01] [00:00:30] Status   multimedia/webcamd | webcamd-5.17.1.0: build
[00:09:34] [01] [00:09:08] Status   multimedia/webcamd | webcamd-5.17.1.0: run-depends
[00:09:34] [01] [00:09:08] Status   multimedia/webcamd | webcamd-5.17.1.0: stage
[00:09:36] [01] [00:09:10] Status   multimedia/webcamd | webcamd-5.17.1.0: package
load: 1.43  cmd: sh 50275 [nanslp] 573.38r 0.60u 2.17s 0% 3672k
mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x1c2 kern_clock_nanosleep+0x1c1 sys_nanosleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8
[main-default] [2022-04-14_01h20m22s] [parallel_build:] Queued: 3  Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Fetched: 2  Tobuild: 1   Time: 00:09:35
 ID  TOTAL                ORIGIN   PKGNAME            PHASE PHASE    TMPFS     CPU% MEM%
[01] 00:09:13 multimedia/webcamd | webcamd-5.17.1.0 package 00:00:03 2.26 GiB 26.1% 0.5%
[00:09:39] Logs: /usr/local/poudriere/data/logs/bulk/main-default/2022-04-14_01h20m22s
[00:09:39] [01] [00:09:13] Status   multimedia/webcamd | webcamd-5.17.1.0: install
[00:09:40] [01] [00:09:14] Status   multimedia/webcamd | webcamd-5.17.1.0: deinstall
[00:09:40] [01] [00:09:14] Finished multimedia/webcamd | webcamd-5.17.1.0: Success
…
```


----------



## richardtoohey2 (Apr 14, 2022)

Yes, I've seen the same when using ports (rather than poudriere) directly, IIRC it _might_ be python - but there's definitely some times it really (pkg-static) "sticks" at 100% CPU for a while doing "something".

I've not investigated further, just saying it does happen sometimes.


----------



## Erichans (Apr 14, 2022)

Alain De Vos said:


> poudriere, make make-depends, pkg-static consumes alot of cpu. Playing youtube videos is interrupted i.e. an ULE scheduler issue ?
> As even "nice -n 20" does not fix it.


My suspicion is that nice does not play nice "hard enough" for all the things (jails & its processes) that poudriere does behind the scenes. 
Perhaps when you confine the poudriere (bulk) build as a whole to a separate jail, you'll have more direct influence and control to limit its resources.


----------



## grahamperrin@ (Apr 14, 2022)

Just rarely, I do notice the computer unusually busy (hogged) at the _package_ stage, however it's never hogged long enough for me to think of it as a problem. IIRC when this happens, the (preceding) _stage_ stage also hogs, and takes longer. Maybe only when I allow poudriere to use all CPUs, although I doubt this. 

Like, I don't think of staging and packaging as a problem if the preceding _build_ stage has taken hours, or days. When a build takes so long, I *really* want the final stages wrapped up without delay, at any cost.


----------



## Alain De Vos (Apr 14, 2022)

Erichans said:


> My suspicion is that nice does not play nice "hard enough" for all the things (jails & its processes) that poudriere does behind the scenes.
> Perhaps when you confine the poudriere (bulk) build as a whole to a separate jail, you'll have more direct influence and control to limit its resources.


Nice is not nice enough. I think freebsd needs a nicer nice


----------



## CraigW (Apr 14, 2022)

From:  https://hackmd.io/@dch/HkwIhv6x7

A few weeks ago, I discovered a gem in the above.  I've been running poudriere from a script like this:


```
nice -n 18 idprio 29 poudriere -N bulk -f /usr/local/etc/poudriere.d/myports.txt -J 3 -j 13amd64 -p default
```

This syntax only seems to work only in a bourne shell or script.
It runs poudriere at a nice and idle priority. 
 I have not found any other examples on the internet of the two being combined.

I can say that youtube in firefox and other interactive stuff seems to run a lot more smoothly.

Thanks to dch@ - I believe he/Dave is a FreeBSD developer.


----------



## grahamperrin@ (Apr 14, 2022)

dch ▲ and thanks; CraigW <https://people.freebsd.org/~dch/> and thanks.


----------



## covacat (Apr 15, 2022)

CraigW said:


> This syntax only seems to work only in a bourne shell or script.


csh/tcsh have built in nice with a simpler syntax, so use /usr/bin/nice


----------



## Alain De Vos (Apr 15, 2022)

CraigW said:


> From:  https://hackmd.io/@dch/HkwIhv6x7
> 
> A few weeks ago, I discovered a gem in the above.  I've been running poudriere from a script like this:
> 
> ...


Thanks, I was unaware of idprio.
But it's weird how the choice of shell could have an influence.
Currently trying on an 8-core with,

```
nice -n 15 idprio 23 /usr/local/bin/poudriere bulk -J 5 -j $JAIL -p $PORTS -f $PACKAGES
```
With this firefox-youtube seems to play acceptable.


----------



## CraigW (Apr 15, 2022)

This seems to work more universally like covacat said above 


```
/usr/bin/nice -n 18 /usr/sbin/idprio 29 poudriere -N bulk -f /usr/local/etc/poudriere.d/myports.txt -J 3 -j 13amd64 -p default
```


----------



## Alain De Vos (Apr 15, 2022)

Looks like a search path issue ?


----------



## Erichans (Apr 15, 2022)

Not quite; it has to do with how a shell interprets and executes commands.

When looking at the tcsh, various actions (such as alias and variable expansion) are performed before the complete line with its commands is executed. Because of the nature of builtin commands (they execute faster because they don't have to be loaded from disk) a possible command (like `nice`) is first looked for as a tcsh builtin command and ... that happens to be the case. As covacat already noted, the Bourne shell (sh(1)) does not have a builtin `nice`.

From tcsh(1) at subsection "Builtin and non-builtin command execution":


> When  a command to be executed is found not to be a builtin command the
> shell attempts to execute the command via execve(2).  Each word in  the
> variable path  names a directory in which the shell will look for the command.


This means that only after the shell decides a command is not a builtin (=internal) command (and also not an alias) that command is searched for externally, that is: in the filesystem. When searching in the filesystem, that is where the search path comes into play when the command is not preceded with an absolute or relative path.


```
nice -n 18 idprio 29 poudriere -N bulk -f /usr/local/etc/poudriere.d/myports.txt -J 3 -j 13amd64 -p default
```
When the above command is issued from a tcsh shell, the builtin `nice` is selected and executed. The tcsh(1) builtin commands `which` and `where`* are very useful to check in advance what is going to be executed**:

```
% which nice
nice: shell builtin command.
% where nice
nice is a shell builtin
/usr/bin/nice
```
To bypass the builtin nice, one can of course specify the path and the command explicitly with its absolute path (in that case it doesn't even have to be in the search path). However, because the matching for an internal command is based on a character by character basis you can also bypass the builtin nice command and let the search process go via the search path by "tweaking its spelling" a bit by preceding it with a backslash: `\nice`. The backslash is a way to specify a character literally; thereby losing its special meaning for the shell. In this particular case however, at first the string "\nice" will be taken by the tcsh as one word and it tries to match it with a builtin command: that doesn't succeed. Further on in the processing, the "\n" is converted to an ordinary character "n"; in the end the string "nice" is matched to the file /usr/bin/nice

```
% which \nice
/usr/bin/nice
% where \nice
/usr/bin/nice
```
In this case I have chosen to specify the character "n" literally. Preceding the command with a backslash seems the easiest when you type it. I could have chosen any and all characters from the word "nice". In the most extreme case that would be `\n\i\c\e`.

___
* from tcsh(1): 





> where _command_
> Reports all  known  instances  of  command, including aliases, builtins and executables in path.


If the output is more than one line then the ordered lines represent the instances where _command_ can be found; the first one is executed.

** other shells have comparable commands.


----------



## richardtoohey2 (Apr 16, 2022)

So pkg-static just done this from a portmaster -r curl, after rebuilding mysql57-server:


```
===>>> Creating a backup package for old version mysql57-server-5.7.37_1
Creating package for mysql57-server-5.7.37_1
load: 2.95  cmd: pkg-static 74876 [running] 49.17r 46.63u 0.20s 99% 105424k
```

Just matches what the OP said about using a lot of CPU.  This is normally when I see it - after the "Creating a backup packge" message.


----------

