# FreeBSD 12.4 - performance regression somewhere?



## rainer_d (Dec 20, 2022)

Hi,

I have a HPE DL380Gen10. After the update to 12.4, it shows a lot of load.


```
(server-prod </root>) 0 # top -b 20
last pid: 49342;  load averages: 423.85, 399.58, 348.51  up 0+10:33:23    10:38:39
546 processes: 430 running, 115 sleeping, 1 zombie
CPU: 89.0% user,  0.0% nice,  4.4% system,  0.2% interrupt,  6.4% idle
Mem: 3022M Active, 8197M Inact, 37G Wired, 76G Free
ARC: 25G Total, 12G MFU, 9363M MRU, 237M Anon, 238M Header, 3780M Other
     14G Compressed, 25G Uncompressed, 1.82:1 Ratio
Swap: 64G Total, 64G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
59235 root          1  80    0   212M    55M RUN     15   2:37  29.39% php-fpm
 9389 root          1  80    0   212M    55M CPU3     3   2:13  29.30% php-fpm
94937 root          1  81    0   212M    54M RUN     11   7:57  29.05% php-fpm
12494 root          1  80    0   212M    55M CPU15   15   1:22  28.86% php-fpm
14319 root          1  81    0   212M    55M RUN      8   1:20  28.17% php-fpm
52501 root          1  81    0   212M    55M RUN      4   0:14  27.39% php-fpm
82538 root          1  80    0   212M    55M CPU15   15   0:16  27.10% php-fpm
62323 root          1  80    0   212M    55M RUN      9   2:05  26.66% php-fpm
80293 root          1  81    0   212M    55M RUN     10   1:33  25.68% php-fpm
 8670 root          1  79    0   212M    55M CPU14   14   0:52  25.20% php-fpm
70489 root          1  81    0   212M    55M RUN      1   2:06  24.85% php-fpm
48369 root          1  81    0   212M    55M RUN      0   0:04  24.85% php-fpm
75881 root          1  81    0   212M    55M RUN     14   2:32  24.66% php-fpm
46526 root          1  79    0   212M    55M CPU7     7   2:32  24.17% php-fpm
73903 root          1  79    0   212M    54M CPU12   12   1:40  24.17% php-fpm
22940 root          1  81    0   212M    55M RUN      4   3:05  23.78% php-fpm
52520 root          1  81    0   212M    55M RUN      6   1:39  23.68% php-fpm
25714 root          1  79    0   212M    54M CPU12   12   5:55  23.49% php-fpm
12415 root          1  81    0   212M    54M RUN      0  10:14  23.29% php-fpm
39214 root          1  79    0   212M    55M RUN     12   1:39  23.10% php-fpm
```

It worked more or less well on 12.3

This server uses the SmartPQI-driver for the Adaptec 3154-8i HBA with firmware 3.53

In htop one can see that most of the load is in userland:







I'm reverting back to the 12.3 boot-environment. I will try 13.1 next.


----------



## mer (Dec 20, 2022)

rainer_d said:


> I'm reverting back to the 12.3 boot-environment. I will try 13.1 next.


If you try 13.x  keep in mind that ZFS changed to OpenZFS and don't do "zpool upgrade" when booted in 13.x or you may lose the ability to boot into 12.x.


----------



## rainer_d (Dec 20, 2022)

Thanks. I have rarely upgraded a root zpool for fear of not being able to boot anymore...


----------



## mer (Dec 20, 2022)

Interesting how many php-fpm processes are running.  Did you try truss or other utility to see where the processes are?  They could be sitting on select or some other "wait for input"


----------



## rainer_d (Dec 20, 2022)

These were sometimes almost 500 PHP-FPM processes. Currently (back on 12.3) there are maybe 60.

These normally run as the user of the individual fpm-pool. Only the master-process runs as root.

I will have to see if this repeats with 13.1. Thank god for boot-environments...


----------



## mer (Dec 20, 2022)

rainer_d said:


> Thank god for boot-environments...


Agree completely


----------



## cracauer@ (Dec 20, 2022)

I wonder whether this is another case of slow regex...


----------



## covacat (Dec 20, 2022)

cracauer@ said:


> I wonder whether this is another case of slow regex...


12.4 should not be affected by that


----------



## _martin (Dec 20, 2022)

mer said:


> They could be sitting on select or some other "wait for input"


Process would not be actively waiting for user input.

The very first thing that came to my mind is what covacat mentioned in other thread and cracauer@ said here - slow regex maybe? I went through the 12.4 relnotes and haven't seen mentioned it there though.

vmstat would help to see where are those load numbers coming from. truss-ing the process is also a good idea.


----------



## covacat (Dec 20, 2022)

load is in userland difficult to tell with tools like truss
maybe opcache is not working
binaries should be mostly interchangeable between 12.3 and 12.4 so you can play the elimination game but it's kind of complex anyway


----------



## _martin (Dec 20, 2022)

In the aforementioned thread I liked the idea of flame graphs, this is worth checking out. I personally have this in my todo list as I'm not familiar with those (shame on me as I have Brendan Gregg's book on my shelf).


----------



## covacat (Dec 20, 2022)

the 12.4 box has a faster cpu than 12.3 box and a newer revision of php 7.4
results are kind of expected exception being intl_calendar










						GitHub - rusoft/php-simple-benchmark-script: Очень простой скрипт тестирования быстродействия PHP | Very simple script for testing of PHP operations speed (rusoft repo mirror)
					

Очень простой скрипт тестирования быстродействия PHP | Very simple script for testing of PHP operations speed (rusoft repo mirror) - GitHub - rusoft/php-simple-benchmark-script: Очень простой скрип...




					github.com
				


*12.4*​*12.3*​*percent *​01_math​4.285​6.075​70.5​02_string_concat​1.927​2.596​74.2​03_1_string_number_concat​1.961​2.439​80.4​03_2_string_number_format​1.739​2.379​73.1​04_string_simple_functions​2.19​3.559​61.5​05_string_multibyte​1.281​1.733​73.9​06_string_manipulation​4.109​7.659​53.6​07_regex​2.246​3.526​63.7​08_1_hashing​2.931​3.527​83.1​08_2_crypt​10.062​11.019​91.3​09_json_encode​3.102​4.733​65.5​10_json_decode​3.987​6.958​57.3​11_serialize​2.081​2.953​70.5​12_unserialize​2.525​3.211​78.6​13_array_fill​5.172​6.774​76.4​14_array_range​2.441​3.351​72.8​14_array_unset​4.003​5.189​77.1​15_clean_loops​2.747​2.841​96.7​16_loop_ifelse​4.226​8.596​49.2​17_loop_ternary​5.631​9.523​59.1​18_1_loop_defined_access​1.713​2.104​81.4​18_2_loop_undefined_access​8.497​11.192​75.9​19_type_functions​1.833​2.866​64.0​20_type_casting​1.831​3.1​59.1​21_0_loop_exception_none​0.303​0.415​73.0​21_1_loop_exception_try​0.333​0.443​75.2​21_2_loop_exception_catch​4.826​7.234​66.7​22_loop_null_op​2.363​3.965​59.6​23_loop_spaceship_op​2.009​2.506​80.2​26_1_class_public_properties​0.262​0.447​58.6​26_2_class_getter_setter​0.842​1.082​77.8​26_3_class_magic_methods​1.568​2.236​70.1​27_simplexml​4.529​7.175​63.1​28_domxml​4.75​7.483​63.5​29_datetime​0.944​1.43​66.0​30_intl_number_format​1.276​2.014​63.4​31_intl_message_format​5.213​8.644​60.3​32_intl_calendar​1.846​1.285​143.7​33_phpinfo_generate​7.093​9.311​76.2​​120.677​173.573​69.5​


----------



## mer (Dec 20, 2022)

_martin said:


> Process would not be actively waiting for user input.


I was using "input" generically;  hence why I added sitting on select/poll.  They sit on file descriptors, which could represent a network socket or something else.


----------

