# Squid very slow on FreeBSD 13.1, CPU 100%



## OlivierW (Dec 11, 2022)

Hello,

I have run Squid on FreeBSD since FreeBSD 10.x (not sure of the exact minor version) and everything was working perfectly well. During all this years, I have regularly updated Squid and FreeBSD.

Currently I am using Squid 5.7 on FreeBSD 13.1-RELEASE-p5 but had the same problem with Squid 5.6.

Around July 2022, I have updated from FreeBSD 13.0 to 13.1.

And since this upgrade, when many requests over plain HTTP are made from my clients, Squid use 100% of my CPU. It's only using a single core, because I don't have SMP enabled on Squid.

And this CPU load really is noticeable because the web browsing experience is very degraded and slow.

A single computer can generate this load, a very common of HTTP requests causing this problems are:
- Windows 10 updates
- FreeBSD OS updates with "freebsd-update" 
- FreeBSD packages updates with "pkg"

My CPU isn't very powerful, it's an Intel Atom N2800, but I have used it for the past 12 years without any problem. First it was with Debian Linux, then FreeBSD since 10.x.

Are there any users with this problem?

What could I search for to find out what is the problem?

Thanks in advance!


----------



## cracauer@ (Dec 11, 2022)

Are you able to produce a flame graph while the problem occurs?



			CPU Flame Graphs
		


Looks intimidating, but it has ready-to-run scripts.


----------



## OlivierW (Dec 13, 2022)

Thanks cracauer@!

From this documentation, I guess I should run:

```
dtrace -x ustackframes=100 -n 'profile-99 /execname == "squid" && arg1/ {
    @[ustack()] = count(); } tick-60s { exit(0); }' -o out.stacks
```

Right?

At the moment I don't have any Windows 10 user and all my FreeBSD are up to date.
I'll try to reproduce the problem and generate a flame graph as soon as possible.


----------



## OlivierW (Dec 17, 2022)

Hello,
I've attached a Flame graph (svg), created when I had the problem a few hours ago.
It happened during a FreeBSD packages update and this time, Squid used "only" around 90% CPU usage.

Not sure what to do with this file. Can anyone help me?


----------



## cracauer@ (Dec 17, 2022)

You spend you CPU time in regular expression matching.

Regex matching is called by access control lists in IPCacheLookupForwarder, whatever that is. There seems to be something screwy going on with regex-driven filters of some kind, or load balancing? Not sure why that would happen a lot during retrieval of single URLs, but that's what it says.

Can you post your config file?


----------



## covacat (Dec 18, 2022)

complex regexes and also case insensitive ones are a lot slower in 13.x than in 12.x (which uses gnu implementation)
grep -i nxzcvnewerewweladsfjlds /usr/share/dict/web2
is 3x slower on a 13.1 box than on a 12.3 box
as you make the search string longer it goes even worse

the 13.x box cpu is 10% slower than 12.x (as of openssl speed md5)


----------



## OlivierW (Dec 18, 2022)

Thanks cracauer@!
It's strange, in the past I was using a lot of "url_regex" rules in Squid, but I removed them about 1 year ago, when the support for PCRE was removed (see the last 4.15 here: https://www.freshports.org/www/squid/ ).

Well, as soon as I have access to my squid machine, I'll share my configuration.


Thanks covacat for this very important information.
Do we have information regarding this problem? Some bug reports to follow?
Does that means regexes are also slower in other tools like PHP or nginx on FreeBSD 13.1?


----------



## covacat (Dec 18, 2022)

254763 – grep very slow with 13.0-RC4
					






					bugs.freebsd.org
				



i think software that uses pcre is not affected, just software using /usr/lib/libregex


----------



## cracauer@ (Dec 18, 2022)

Hmmm, I can't reproduce this using 14-current and textproc/gnugrep

If anything BSD grep takes a bit less CPU time.

Some of the test cases also use strings that are not regular expressions. They shouldn't be affected by slow regex engines.


----------



## cracauer@ (Dec 18, 2022)

This is confusing. The squid source code schlepps a GNU regex around.

./configure says:
checking if GNUregex needs to be compiled... no

However, it then happily builds and uses the GNU regex in its sources.

Meanwhile ldd shows it linked against the system libregex.


----------

