# Issue with ULE scheduler



## CyberCr33p (May 26, 2022)

CPU: AMD Ryzen 9 5950X 16 Cores (32 Threads)

I use this command to compress a lot of images:

`find . -iname "*.jpg" -exec ~/graphicsmagick/bin/gm mogrify -strip -quality 75% {} > /dev/null 2>&1 \;`


The issue is that GraphicsMagick (the same with ImageMagick) after some time hangs, the process usage is 100% and truss shows:


```
sched_yield() = 0 (0x0)
sched_yield() = 0 (0x0)
sched_yield() = 0 (0x0)
```


Does anyone else has such issues with ULE scheduler ?


I found this https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208421 which may be related.


----------



## mer (May 26, 2022)

What version of FreeBSD, what is the filesystem, is this in a jail? 
Reason for asking is that bug is against FreeBSD 10.2 last update was back in 2018.
Have you tried other ways of accomplishing the task?  Maybe do the find command to get a list of files, then a script that reads the file and does the gm command on each line?
Basically to rule out an interaction between the find and the gm command.


----------



## CyberCr33p (May 26, 2022)

I use 13.1 and I don't run it under jail.


----------



## CyberCr33p (May 26, 2022)

The same issue exist if I use a shell script with a for loop to do the compressions:


```
#!/usr/local/bin/bash

for i in `cat list.txt`; do
  ~/graphicsmagick/bin/gm mogrify -strip -quality 75% "${i}"
done
```


----------



## mer (May 26, 2022)

I don't have any answers, just trying to get whatever data we can.
If you do a single file by hand, does the same issue happen?
Did you build the gm executable against 13.1, install a binary from somewhere?


----------



## covacat (May 26, 2022)

does it stall at the same image or random ?


----------



## CyberCr33p (May 26, 2022)

covacat said:


> does it stall at the same image or random ?


Random image



mer said:


> I don't have any answers, just trying to get whatever data we can.
> If you do a single file by hand, does the same issue happen?
> Did you build the gm executable against 13.1, install a binary from somewhere?


If I do a single file it doesn't hang but I am not sure for this as it hang in random file in the loop.

I build gm against 13.1.


----------



## mer (May 26, 2022)

What version did you build?  I'm asking because it's available in the ports/packages so a pkg install GraphicsMagick will install 1.3.37 it seems.  If you don't have admin access to install packages, building under your user is fine, just want to eliminate variables.
EDIT 1.3.37 is the latest in quarterly packages, latest packages may have 1.3.38


----------



## CyberCr33p (May 26, 2022)

mer said:


> What version did you build?  I'm asking because it's available in the ports/packages so a pkg install GraphicsMagick will install 1.3.37 it seems.  If you don't have admin access to install packages, building under your user is fine, just want to eliminate variables.


I use the latest version. The same issue happens with ImageMagick too.

The issue exist with "AMD Ryzen 9 5950X 16 Cores" and NVME disks but with "Intel  i7-6700 Quad-Core Skylake" and mechanical disks doesn't exist.


----------



## mer (May 26, 2022)

Is there a command line argument that limits the number of threads created/used?


----------



## 3301 (May 26, 2022)

If it's starting new process/thread before previous were stopped it can consume not only CPU but also RAM/swap - check if it's not the cause.
Edit: after closer inspection it looks like it's running sequentially, only one process/thread is created.


----------



## mer (May 26, 2022)

As a test in the scripted version, what happens if you add a "sleep 1" after the gm command so that the loop sleeps a little bit before the next iteration?
Yes I know it adds time if there are a lot of files, but it's a bit of a workaround and could help determine if it's a "process exit and cleanup" that's too slow and causing issues.


----------



## covacat (May 26, 2022)

try playing with OMP_NUM_THREADS env var or -limit threads gm arg


----------



## CyberCr33p (May 26, 2022)

Running `find . -iname "*.jpg" -exec jpegoptim -s -q {} \;` the issue doesn't exist. So far I was able to reproduce it only with GraphicsMagick / ImageMagick.


----------



## covacat (May 26, 2022)

CyberCr33p said:


> Running `find . -iname "*.jpg" -exec jpegoptim -s -q {} \;` the issue doesn't exist. So far I was able to reproduce it only with GraphicsMagick / ImageMagick.


it may have something to do with threads/openmp


----------



## CyberCr33p (May 26, 2022)

I will do some tests with OMP_NUM_THREADS when I have some free time and tell you the results.


----------



## Alain De Vos (May 29, 2022)

Does the problem remains when you put "/usr/bin/nice -n 15 /usr/sbin/idprio 23 " before your command ?


----------



## CyberCr33p (May 29, 2022)

I will test it and let you know.


----------

