# FreeBSD 13.1 extremally slow



## cmiu147 (Oct 11, 2022)

Hi guys,

I have a home/nas server running on FreeBSD for over 1year...  Here is the config:

Mainboard: Intel S1200BTL (intel C202 chipset)
CPU: Intel Xeon E3-1200v2
RAM: 16GB
hard drives: 4x Seagate Barracuda ST2000DM008
filesystem: ZFS
Raid: ZFS raid10

Even when I open ssh to connect to this it takes couple of second until I get my login prompt.
Now I just run `iocage list` to list my iocages and the result came after 30seconds. On iocage I have a cage with apache24+mariadb+php and most of the time the response time is awful.

The server is connected to the network using an Intel 1Gbps card. 

Another problem is that the CPU is 99% in idle, is not even used... something is not working as expected, but i don't know how to investigate. Can you help me with some advice?


The hard drives are SMR drives. I haven't known about SMR drives when i build this machine. Could this be the answer for very slow performances?


Thank you!


----------



## SirDice (Oct 11, 2022)

cmiu147 said:


> Even when I open ssh to connect to this it takes couple of second until I get my login prompt.


Fix your DNS. Specifically reverse lookups.


----------



## gpw928 (Oct 11, 2022)

Also DNS related, where you have more than one name server.  If you your primary name server is not responding, clients will run very slowly as every DNS lookup will be delayed by waiting for a timeout on the primary server before trying the secondary server.  The usual fix is to use a caching name server on each client.


----------



## cmiu147 (Oct 11, 2022)

i don't need a DNS... i ssh directly to the ip address 
i'm in the same network...


----------



## SirDice (Oct 11, 2022)

cmiu147 said:


> i don't need a DNS... i ssh directly to the ip address


The sshd(8) does a reverse lookup when you login. That's your delay.


----------



## cmiu147 (Oct 11, 2022)

and the delay to iocage list...that takes 30s in the first place and then takes less than 5seconds ?

why my all apps are fantastic slow compared to a linux box using the same config?


----------



## SirDice (Oct 11, 2022)

They're not slow. You just have something incorrectly configured, and my guess is name resolving. Yes, even on a local network this is important. Doesn't matter how you do it, DNS or distributing hosts(5) files, but the fact remains this is important to configure correctly.


----------



## cmiu147 (Oct 11, 2022)

Well...definitely something is not working as it should...
I  checked the host file... everything fine there...

127.0.0.1 localhost localhost.localdomain
192.168.0.119 nfs nfs.mydomain.com
192.168.0.10 dev2 dev2.mydomain.com

where mydomain is the name of the real domain name.

what else should i look for? this was working fine couple of months ago... i haven't changed the config... 

how can i identify if is a problem with my SATA controller?


----------



## SirDice (Oct 11, 2022)

cmiu147 said:


> what else should i look for?


What's in /etc/resolv.conf and /etc/nsswitch.conf?


----------



## cmiu147 (Oct 11, 2022)

```
search mydomainname.com
#nameserver 192.168.0.1
nameserver 8.8.8.8
```


```
# $FreeBSD$
#
group: compat
group_compat: nis
hosts: files dns
netgroup: compat
networks: files
passwd: compat
passwd_compat: nis
shells: files
services: compat
services_compat: nis
protocols: files
rpc: files
```

also i deleted a 200mb db and took 9minutes.


```
Query OK, 112 rows affected (9 min 12.069 sec)
```

i guess the hardware is not used at his full potential... but i don't understand why...


----------



## rootbert (Oct 11, 2022)

what's the usage of your zpool? - at least 20% free space? Is your server swapping - did you limit zfs's ARC? Whats the chipset of your network card and your ifconfig configuration (offloading? tso etc)? Do your ssh connection initiate faster if you have "UseDNS no" in your /etc/ssh/sshd_config?


----------



## cmiu147 (Oct 11, 2022)

```
zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
backup  1.81T  1.28T   540G        -         -    10%    70%  1.00x    ONLINE  -
zroot   3.62T  2.85T   795G        -         -    22%    78%  1.00x    ONLINE  -
```


```
ZFS Subsystem Report                            Tue Oct 11 17:46:34 2022
------------------------------------------------------------------------

System Information:

        Kernel Version:                         1301000 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               5000
        ZFS Filesystem Version:                 5

FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC 5:46PM  up 81 days,  4:26, 2 users, load averages: 0.32, 0.28, 0.25

------------------------------------------------------------------------

System Memory:

        3.06%   477.29  MiB Active,     3.78%   590.14  MiB Inact
        89.34%  13.62   GiB Wired,      0.00%   0       Bytes Cache
        3.03%   472.43  MiB Free,       0.80%   124.27  MiB Gap

        Real Installed:                         16.00   GiB
        Real Available:                 98.02%  15.68   GiB
        Real Managed:                   97.24%  15.25   GiB

        Logical Total:                          16.00   GiB
        Logical Used:                   93.51%  14.96   GiB
        Logical Free:                   6.49%   1.04    GiB

Kernel Memory:                                  519.78  MiB
        Data:                           91.74%  476.86  MiB
        Text:                           8.26%   42.92   MiB

Kernel Memory Map:                              15.25   GiB
        Size:                           88.59%  13.51   GiB
        Free:                           11.41%  1.74    GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                27.43   m
        Mutex Misses:                           6.15    k
        Evict Skips:                            6.83    k

ARC Size:                               76.05%  11.17   GiB
        Target Size: (Adaptive)         76.12%  11.18   GiB
        Min Size (Hard Limit):          3.34%   501.87  MiB
        Max Size (High Water):          29:1    14.68   GiB
        Compressed Data Size:                   9.61    GiB
        Decompressed Data Size:                 28.06   GiB
        Compression Factor:                     2.92

ARC Size Breakdown:
        Recently Used Cache Size:       46.75%  5.23    GiB
        Frequently Used Cache Size:     53.25%  5.95    GiB

ARC Hash Breakdown:
        Elements Max:                           980.84  k
        Elements Current:               82.90%  813.12  k
        Collisions:                             30.13   m
        Chain Max:                              8
        Chains:                                 122.26  k

------------------------------------------------------------------------

ARC Efficiency:                                 14.07   b
        Cache Hit Ratio:                99.73%  14.04   b
        Cache Miss Ratio:               0.27%   37.34   m
        Actual Hit Ratio:               99.28%  13.97   b

        Data Demand Efficiency:         97.43%  488.58  m
        Data Prefetch Efficiency:       10.14%  11.23   m

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.35%   48.57   m
          Most Recently Used:           3.45%   484.13  m
          Most Frequently Used:         96.10%  13.49   b
          Most Recently Used Ghost:     0.05%   6.37    m
          Most Frequently Used Ghost:   0.06%   8.50    m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  3.39%   476.02  m
          Prefetch Data:                0.01%   1.14    m
          Demand Metadata:              80.68%  11.32   b
          Prefetch Metadata:            15.92%  2.23    b

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  33.64%  12.56   m
          Prefetch Data:                27.01%  10.09   m
          Demand Metadata:              22.49%  8.40    m
          Prefetch Metadata:            16.85%  6.29    m

------------------------------------------------------------------------

L2ARC is disabled

------------------------------------------------------------------------
```

Basically i haven't touched ARC parameters or zfs parameters... i only created the array, installed the OS and then configure the needed packages (samba, apache, etc).


----------



## Alain De Vos (Oct 11, 2022)

You write-down hostname.
Then:
"host hostname" should give an ipaddress.
And
"host ipaddress" should return the hostname.


----------



## rootbert (Oct 11, 2022)

whats the output of "zpool status", "top -b -o res", "iostat -c 5", "systat -vmstat"?


----------



## _martin (Oct 11, 2022)

When it comes to ssh annoyance make sure you have: 
	
	



```
grep -i usedns /etc/ssh/sshd_config
UseDNS no
```
Also note it does make a difference if fqdn is first in hosts or not. Fix those entries to `192.168.0.10  dev2.mydomain.com dev2` in /etc/hosts.

On Linux strace is your friend to quickly diagnose what are you waiting for. I'm sad to say this is no longer supported on FreeBSD. dtrace is a nice tool, more powerful too, but it has steep learning curve and sometimes it's harder to use for simple diagnostics.

To quickly check if DNS is culprit hash out all lines in resolv.conf so you don't have any DNS set. Make sure jails don't have that too. Then try that iocage command again. I'm assuming you can survive without DNS few minutes.


----------



## SirDice (Oct 11, 2022)

_martin said:


> On Linux strace is your friend to quickly diagnose what are you waiting for. I'm sad to say this is no longer supported on FreeBSD. dtrace is a nice tool, more powerful too, but it has steep learning curve and sometimes it's harder to use for simple diagnostics.


truss(1)


----------



## _martin (Oct 11, 2022)

SirDice said:


> truss(1)


Yeah, kinda, but not really. It's like retarded cousin of strace. It can do something but it's way way simpler than strace. Here it might help though, that's true.


----------



## SirDice (Oct 11, 2022)

_martin said:


> It can do something but it's way way simpler than strace.


It might be simpler but it comes with the base OS, so it's always available.


----------



## cmiu147 (Oct 11, 2022)

rootbert said:


> whats the output of "zpool status", "top -b -o res", "iostat -c 5", "systat -vmstat"?




```
zpool status

  pool: backup
 state: ONLINE
  scan: scrub repaired 0B in 04:40:21 with 0 errors on Wed Jun 15 14:31:04 2022
config:
    NAME        STATE     READ WRITE CKSUM
    backup      ONLINE       0     0     0
      ada4      ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 03:07:52 with 0 errors on Tue Jul 12 16:48:40 2022
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p3  ONLINE       0     0     0
        ada1p3  ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        ada2p3  ONLINE       0     0     0
        ada3p3  ONLINE       0     0     0

errors: No known data errors
```


```
top -b -o res
last pid:  2625;  load averages:  0.46,  0.33,  0.27; battery: -1%  up 81+06:25:52    19:45:58
53 processes:  1 running, 52 sleeping
CPU:  0.5% user,  0.0% nice,  0.3% system,  0.0% interrupt, 99.1% idle
Mem: 113M Active, 958M Inact, 19M Laundry, 14G Wired, 517M Free
ARC: 11G Total, 7453M MFU, 3146M MRU, 130K Anon, 187M Header, 658M Other
     9834M Compressed, 28G Uncompressed, 2.89:1 Ratio
Swap: 16G Total, 549M Used, 15G Free, 3% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 1645 mysql         8  20    0  1645M   452M select   2   0:31   0.00% mariadbd
97737 root          6  20    0   210M   126M select   2  20:59   3.42% smbd
93531 www           1  52    0   258M   122M accept   1   4:34   0.00% php-fpm
93533 www           1  52    0   274M   119M accept   1   4:33   0.00% php-fpm
93532 www           1  20    0   420M   110M accept   1   5:54   0.00% php-fpm
93545 www           1  20    0   237M   103M accept   1   4:36   0.00% php-fpm
97689 root          1  20    0   197M    37M select   2   0:02   0.00% smbd
  819 cristi        1  20    0    21M  9576K select   2   0:00   0.00% sshd
 2483 cristi        1  20    0    21M  9572K select   1   0:00   0.00% sshd
 2479 root          1  22    0    21M  9556K select   0   0:00   0.00% sshd
  815 root          1  20    0    21M  9548K select   1   0:00   0.00% sshd
18536 mysql        30  20    0  1934M  7360K select   3   1:56   0.00% mysqld
  824 root          1  20    0    16M  4752K ttyin    3   0:00   0.00% csh
 2488 root          1  20    0    16M  4588K pause    2   0:00   0.00% csh
 2625 root          1  20    0    14M  3356K CPU0     0   0:00   0.00% top
 2484 cristi        1  27    0    13M  3212K wait     0   0:00   0.00% sh
  820 cristi        1  22    0    13M  3116K wait     3   0:00   0.00% sh
 1429 mysql         1  52    0    13M  3040K wait     0   0:00   0.00% sh
```


```
iostat -c 5
       tty            ada0             ada1             ada2             cpu
 tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id
   2     8 26.0    6   0.2  26.0    6   0.2  25.0    7   0.2   1  0  0  0 99
   0   228 18.6   62   1.1   8.9   54   0.5   7.4   38   0.3   0  0  0  0 100
   0    79  4.0    4   0.0   4.0    4   0.0  12.0   10   0.1   0  0  0  0 100
   0    79  0.0    1   0.0   4.0    2   0.0   0.0    0   0.0   0  0  0  0 100
   0    81 10.4    5   0.1   9.3    6   0.1   7.4    7   0.1   0  0  0  0 100
```



```
3 users    Load  0.16  0.27  0.24                  Oct 11 19:47:35
   Mem usage:  97%Phy 88%Kmem                           VN PAGER   SWAP PAGER
Mem:      REAL           VIRTUAL                        in   out     in   out
       Tot   Share     Tot    Share     Free   count
Act  1060M  74664K   5858M     182M     517M   pages
All  1066M  80744K   6005M     298M                       ioflt  Interrupts
Proc:                                                  16 cow     160 total
  r   p   d    s   w   Csw  Trp  Sys  Int  Sof  Flt       zfod        uart0 4
              89   4   567   52  641   64    9   41       ozfod     2 ehci1 20
                                                         %ozfod     2 ehci0+ 22
 0.2%Sys   0.0%Intr  0.0%User  0.0%Nice 99.8%Idle         daefr    25 cpu0:timer
|    |    |    |    |    |    |    |    |    |    |    12 prcfr    21 cpu1:timer
                                                       25 totfr    25 cpu2:timer
                                           dtbuf          react    25 cpu3:timer
Namei     Name-cache   Dir-cache    345026 maxvn          pdwak    18 em0:irq0
   Calls    hits   %    hits   %    345024 numvn       46 pdpgs    42 ahci0 37
    1373    1367 100                303793 frevn          intrn
                                                      14G wire
Disks  ada0  ada1  ada2  ada3  ada4 pass0 pass1      114M act
KB/t  12.37 11.42 12.74 12.74  0.00  0.00  0.00      958M inact
tps       9    11    11    11     0     0     0       19M laund
MB/s   0.10  0.12  0.13  0.13  0.00  0.00  0.00      517M free
%busy    30    28    53    48     0     0     0         0 buf
```


----------



## W.hâ/t (Oct 11, 2022)

I had a similar issue due to a bad pf config


----------



## rootbert (Oct 11, 2022)

ok, I guess ARC is eating up all your memory and processes are being swapped. I recommend putting "vfs.zfs.arc_max=4294967296" into your /boot/loader.conf and reboot. This will limit your ARC to 4GB. Should you experience too much free memory afterwards you can increase that value (and reboot) to increase your ZFS cache.


----------



## cmiu147 (Oct 11, 2022)

W.hâ/t said:


> I had a similar issue due to a bad pf config


also with pf disabled the behaviour is the same. I will try what rootbert suggest and get back with a reply.

Thank you guys!


----------



## SirDice (Oct 11, 2022)

rootbert said:


> I guess ARC is eating up all your memory and processes are being swapped.


There's barely any swap being used. Swap usage in and of itself isn't bad. 

But I see a mariadbd and mysqld running. Those two plus ARC like to use a lot of memory. ARC and MySQL/MariaDB can and will fight over memory. So tune your databases to not use any more more memory than it actually needs and limit your ARC. And remember to keep some memory left over for the rest of your applications. With this much stuff on it I would probably opt for a memory upgrade too, 16GB isn't much.


----------



## vmb (Oct 11, 2022)

Sadly, SMR drives are effectively incompatible with ZFS. Yes, you can create a ZFS file system on them but as soon as the file system needs to write on a block that has already been written on, ZFS slows down due to the overhead in all the extra work it has to do because of the shingled recording. ZFS should only be used with Conventional Magnetic Recording (CMR) hard drives.


----------



## Lady Serena Kitty (Oct 11, 2022)

vmb said:


> Sadly, SMR drives are effectively incompatible with ZFS. Yes, you can create a ZFS file system on them but as soon as the file system needs to write on a block that has already been written on, ZFS slows down due to the overhead in all the extra work it has to do because of the shingled recording. ZFS should only be used with Conventional Magnetic Recording (CMR) hard drives.


Or some type of solid state storage device.  ZFS is insanely fast on my NVMe boot device.  I say use NVMe and if that's not pawssible, use SATA-SSDs.  Come to think of it, a zraid should be on SATA-SSD.


----------



## cmiu147 (Oct 11, 2022)

SirDice said:


> There's barely any swap being used. Swap usage in and of itself isn't bad.
> 
> But I see a mariadbd and mysqld running. Those two plus ARC like to use a lot of memory. ARC and MySQL/MariaDB can and will fight over memory. So tune your databases to not use any more more memory than it actually needs and limit your ARC. And remember to keep some memory left over for the rest of your applications. With this much stuff on it I would probably opt for a memory upgrade too, 16GB isn't much.


Using a nvme SSD with ZIL and L2ARC, would help?



vmb said:


> Sadly, SMR drives are effectively incompatible with ZFS. Yes, you can create a ZFS file system on them but as soon as the file system needs to write on a block that has already been written on, ZFS slows down due to the overhead in all the extra work it has to do because of the shingled recording. ZFS should only be used with Conventional Magnetic Recording (CMR) hard drives.


Haven't known that... So I should consider swap the drives in the future.


----------



## ralphbsz (Oct 11, 2022)

... empty post ...


----------



## ralphbsz (Oct 11, 2022)

If the root of the file system slowness is the SMR drives, that should be visible in iostat, vmstat, or zpool iostat. You should be seeing long latencies, and deep queues.



cmiu147 said:


> why my all apps are fantastic slow compared to a linux box using the same config?


What file system and RAID system do you use on Linux?


----------



## gpw928 (Oct 11, 2022)

cmiu147 said:


> Even when I open ssh to connect to this it takes couple of second until I get my login prompt.


You are showing a near idle system.  If it's like that when you try to login with ssh (and not flat-lined), something is mis-configured.  I think that the original instincts of SirDice are well founded.

I suggest you focus on why ssh is taking so long.  Run the ssh server daemon in the foreground in debug mode.  Run the ssh client also in debug mode.  Watch the traces.  See where the delays are happening:
	
	



```
# On the server
$ cd /tmp
$ script
$ sudo /usr/sbin/sshd -p 1234 -d -d -d
# watch until login is complete
^C
$ exit
 
# On the client
$ cd /tmp
$ script
$ ssh -Y -p 1234 -v -v -v YourLoginName@ServerName
# watch until login is complete
$ exit
$ exit
```


----------



## PMc (Oct 12, 2022)

I highly recommend `gstat -p` for disk issues.If there is an issue with a disk, it's immediateily visible turning red. If there is a problem with a disk in a raid, one can directly see that the counters of that disk are different to the others.


----------



## PMc (Oct 12, 2022)

cmiu147 said:


> Using a nvme SSD with ZIL and L2ARC, would help?


And a special device, yes, that may work. But: in order to do that, one needs information about the workload. The working set size, which filesystems should go into l2arc (and which shouldn't because they just overwrite the data that is needed). Etc.
And for zil we should look into `gstat -po` if we have flush events that delay our work.



cmiu147 said:


> Haven't known that... So I should consider swap the drives in the future.


Yes, Seagate doesn't tell openly.  They go to quite a length to fill their spec sheets with useless statements, while not telling the relevant facts. And then they resort on the position that a consumer desktop drive (like the "DM") is not suited for databases.
But, if you notice a cache of 256M on a consumer drive, something must be wrong (datacenter ultrastar have 128M).


----------



## SirDice (Oct 12, 2022)

cmiu147 said:


> Using a nvme SSD with ZIL and L2ARC, would help?


It's not going to alleviate the memory pressure. It would actually start using even more. ZIL and L2ARC could help with disk performance, but it really depends on the workload.


----------



## Alain De Vos (Oct 12, 2022)

To give more room to the databases you can limit the ZFS-ARC
sysctl.conf eg 2.5G

```
vfs.zfs.arc_max= 2500000000              #0
```

To reduce the fight of memory you can also limit database mariadb,
E.g. in my.cnf,

```
aria_pagecache_buffer_size=256M
innodb_buffer_pool_size=512M
```


----------



## Crivens (Oct 12, 2022)

_martin said:


> On Linux strace is your friend to quickly diagnose what are you waiting for. I'm sad to say this is no longer supported on FreeBSD. dtrace is a nice tool, more powerful too, but it has steep learning curve and sometimes it's harder to use for simple diagnostics.


What about ktrace?


----------



## _martin (Oct 12, 2022)

Crivens said:


> What about ktrace?


Personally I don't see benefit of it to truss. Certainly no versatility as strace. strace is able to do nice filtering with `-e` which is a feature I use the most. But other formatting features are missing from truss. I find it easy to use and for simple cases just one nice tool to have.

But in OPs problem I was hoping he'd react to SirDice suggestion about DNS and attach a truss to sshd daemon already with `truss -fp <pidofdaemon>` so we could see why there's lag in connection.


----------



## cmiu147 (Oct 12, 2022)

PMc said:


> I highly recommend `gstat -p` for disk issues.If there is an issue with a disk, it's immediateily visible turning red. If there is a problem with a disk in a raid, one can directly see that the counters of that disk are different to the others.


Ok, this really helped me... looks like my disks are doing something even in idle mode... The thing is that is fluctuating from 0.x straight to 100+ %. I stoped all the services and still get 100% spikes.


----------



## Alain De Vos (Oct 12, 2022)

How do you find the process-id of the one creating disk-activity ?


----------



## W.hâ/t (Oct 12, 2022)

Alain De Vos said:


> How do you find the process-id of the one creating disk-activity ?


In `top` key 'm' display processes iostat


----------



## Alain De Vos (Oct 12, 2022)

```
top -m io -o total
```


----------



## Peacekeeper2000 (Oct 12, 2022)

Let me throw my two cents in here: I had two troubles with DNS: using it with encryption has delayed everything as the timeout is long, when encryption is not working. Using DNS to provide Fingerprints for PKI is also slow, when not every  host is listed. And looks like your router works not as DNS server ? only Google (and noch cache) ?


----------



## gpw928 (Oct 13, 2022)

It's entirely possible that you are conflating a collection of maladies. 

The problem is that the possibilities being canvassed are growing at a pace.

So you need to narrow down the search be eliminating variates, and move forward one step at a time.

Yes, SMR disks are crap for ZFS.  Their achilles heel is mostly with deletions and re-writing.  They will never work well with a database that needs to be constantly updated.  Deleting database rows is going to be slow.  Resilvering a replacement disk will be a nightmare -- it will take an eternity (like two weeks instead of a day), and the system will run like a dog for the entire time.  Your only way forward with these SMR disks is to eventually replace them.

So to trouble-shoot, quiesce your applications, your databases, and your disks.  Then move forward, one step at a time.  You need to eliminate mis-configuration before considering hardware problems.

Several people have mentioned different ways in which delays in every DNS lookup can multiply to kill application performance.  This is experience speaking.  It needs to be near top of the list for your investigations.

Time DNS lookups (forward and reverse) on your server and your client. Repeat the tests several times.  Not all DNS queries are resolved in the same way.  This is why I suggested you test ssh connections above. 

Run the `iocage list`.  Is it still slow?


----------



## cmiu147 (Oct 13, 2022)

gpw928 said:


> It's entirely possible that you are conflating a collection of maladies.
> 
> The problem is that the possibilities being canvassed are growing at a pace.
> 
> ...


If I run iocage list after a period of time... let's say couple of hours, take around 10-20s until list the cages. If I run it again immediately, the cages are listed instantly. I sort out the dns problem... looks like now ssh password promo is showing in 1-2seconds.This happens with a wordpress website for example... if I open it for the first time take 10-20seconds until I get the response and then works almost normally... pretty slow compared to the same apache/php config in a similar server with centos and SAS Hitachi drives.

I suspect 2 things here... 

1. Controller
2. SMR drives


----------



## Crivens (Oct 13, 2022)

SMR Drives are slow in writes. What may help is to disable the atime tracking. That creates a write with every open() of a file, even for reading.


----------



## cmiu147 (Oct 13, 2022)

One more thing... now the drives are under ZFS raid10.  will I have a performance gain if I use a hardware raid controller, set to raid5 or raid10?


----------



## rootbert (Oct 13, 2022)

if at all probably not by much. In addition to that, the error-correcting-codes used in hardware raid controllers are not as advanced as those of zfs, furthermore controller might pass the wrong information from the device up to zfs (though this probability is really low). Also note: most hardware controllers do save metadata, so in case your controller does not work any more, data restoration is a more complicated process. I'd rather spend the money on two SSDs, have a fast zpool and add two small partitions of each as ZIL (mirror) + L2ARC to my slow hdd based zpool.


----------



## PMc (Oct 13, 2022)

cmiu147 said:


> If I run iocage list after a period of time... let's say couple of hours, take around 10-20s until list the cages. If I run it again immediately, the cages are listed instantly.


Do the drives go into a powersave mode? Desktop drives usually do, and then this would be perfectly normal.


----------



## gpw928 (Oct 13, 2022)

cmiu147 said:


> If I run iocage list after a period of time... let's say couple of hours, take around 10-20s until list the cages. If I run it again immediately, the cages are listed instantly.


That's new and important information.  I suspect that identifying the cause will solve the problem.

The powersave mode suggested by PMc is a strong contender.  Look at what the disks are doing during that very long delay.

Edit, the Seagate ST2000DM008 has a "Standby Mode/Sleep Mode".  This post on the use of camcontrol(8)  may be useful.


----------



## cmiu147 (Oct 13, 2022)

t


gpw928 said:


> That's new and important information.  I suspect that identifying the cause will solve the problem.
> 
> The powersave mode suggested by PMc is a strong contender.  Look at what the disks are doing during that very long delay.


Hi,
They are spiking like crazy... and the speed transfer is... I don't want to talk about it... any little thing, the disk goes to 100%... is like I was having an old ATA33 drive ... not a modern sata drive... I understand that SMR is crap... but even that bad?


----------



## gpw928 (Oct 13, 2022)

The fact that the delay only happens once is really important.  If the delay problem was because of write amplification due to SMR, it should be consistent.  It's not.  I was editing my post above while you were responding.  Suggest you follow the "Standby Mode/Sleep Mode" trail.


----------



## PMc (Oct 13, 2022)

cmiu147 said:


> They are spiking like crazy... and the speed transfer is... I don't want to talk about it... any little thing, the disk goes to 100%... is like I was having an old ATA33 drive ... not a modern sata drive... I understand that SMR is crap... but even that bad?


Disk going to 100% without heavy traffic usually means it is internally busy. SATA does auto-startup, so the first command will take a while:

```
# camcontrol standby ada2
# time dd if=/dev/ada2 count=1 of=/dev/null
1+0 records in
1+0 records out
512 bytes transferred in 6.262433 secs (82 bytes/sec)
0.000u 0.003s 0:06.26 0.0%      0+0k 1+0io 0pf+0w

 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      1      1      0   2442      0      0    0.0  243.1| ada2
```

First, find out if these disks support epc (I don't think so, but let's check):

```
# camcontrol epc ada2 -c list
(pass9:ahcich2:0:0:0): READ_LOG_DMA_EXT. ACB: 47 00 08 00 00 40 00 00 00 00 02 00
(pass9:ahcich2:0:0:0): CAM status: ATA Status Error
[etc.etc.]
```
 -> no epc support

```
# camcontrol epc ada7 -c list
ATA Power Conditions Log:
  Idle power conditions page:
    Idle A condition:
[etc.etc.]
```
-> disk with epc support

Then check apm:

```
# camcontrol identify ada2 | egrep "^(device m|Feature|advan)"
device model          ST3000DM008-2DM166
Feature                      Support  Enabled   Value           Vendor
advanced power management      yes      yes     192/0xC0
```

This seems to switch it off:

```
# camcontrol apm ada2
# camcontrol identify ada2 | egrep "^(device m|Feature|advan)"
device model          ST3000DM008-2DM166
Feature                      Support  Enabled   Value           Vendor
advanced power management      yes      no      0/0x00
```


----------



## rootbert (Oct 13, 2022)

SMR is really that bad, yes. While probably not applicable in this case, head parking can also be a problem I have encountered even on enterprise disks - sysutils/parkverbot to the rescue.


----------



## cmiu147 (Oct 14, 2022)

So.. this morning I did a zpool scrub root .... took around3 hours... reported 0 errors.

```
scan: scrub repaired 0B in 02:57:50
```

The strange thing is that after this, the disk are not spiking from 0 to 100%.They have normal usage... between 10-80%. And the system is way more faster... After a MariaDB import which finished way more faster, the drives started to spike again, but not as before... I got another BSD machine with raid0, where I was playing more than 1y ago... I powered on this morning and was fast like a rocket... ZFS raid0 2xToshiba CMR drives.
now... the questions is as follow, regarding the drives:

1. I found a nice offer on a local store with some old SEAGATE Pipeline 2tb 5900rpm drives at 50$ / piece
2. Toshiba DT01ACA200 2tb (exactly as the one from the test machine, the only difference is that they are 2tb) at 70$ / piece
3. Put some extra money and buy some SAS drives.
4. Refurbished SAS drives at around 100$ / piece (2tb 3.5inch).

Thanks!

​


----------



## Erichans (Oct 14, 2022)

My 2 cents. 
As to spinning rust platters: choose drives that are certified/meant for NAS application and or server application. I have no up to date advice as to current SATA drives most favourable for NAS/server applications. I used to use WD red (non-pro) (CMR) drives but they seem less popular today, or so I heard. 

Unless you have qualified reasons to use SAS drives and are willing to pay for those, choose SATA drives. Such reasons might be higher reliability, higher base throughput (higher rpm), use for dual data access and are capable to host SAS drives through a SAS host adapter. With transitioning from SMR to CMR (background: WD Red SMR vs CMR Tested Avoid Red SMR) you'll have taken the biggest drive hardware step forward. I'd stay away from refurbished drives for professional applications.

As to adding additional hardware in relation to your applications: you might benefit from adding a SLOG and L2ARC. However, under normal circumstances these are not the first goto solutions to get a (too) slow system up to speed; see e.g. What Makes a Good Time to Use OpenZFS Slog and When Should You Avoid It and OpenZFS: All about the cache vdev or L2ARC. After installing CMR drives, IMO your first attention should go to get an appropriate amount of RAM for your set up; after that manage/tune the ARC size in relation to the needs of your applications (especially your db's) if required.


----------



## gpw928 (Oct 15, 2022)

Some random thoughts:

You really should turn off "advanced power management" in your existing drives!
It's difficult to make disk recommendations without knowing your workload or risk profile.
Disk drives marketed for "NAS" generally don't park their heads uninvited, and don't use SMR recording.  But you need to check!
WD Red CMR drives are at the low end of NAS performance.  But, my personal  experience is that their value for money equation is OK (50% of my 10 year old CMR Reds are still running).  The WD Red SMR drives are easily mistaken for CMR drives.  
Any CMR disk doing random writing is going to run like a rocket when compared to an SMR drive.
The Toshiba DT01ACA200 is a consumer grade disk, and it's not mentioned in the list of SMR drives published by Toshiba in 2020. They are available on Newegg for $50.
These days 4 TB drives are often not significantly more expensive than 2 TB drives.


----------



## cmiu147 (Oct 15, 2022)

I use this server for the following:
1. Backup my Mac files (time machine setup via samba)
2. Samba for sharing and storing files... Like photos from my holidays
3. Backup of various windows machines
4. Nightly backup of my office computer.
5. Iocage for Apache/mysql/PHP development
6. Iocage for a Django app
7. In the future I want to make a new pool for NFS and iscsi.
8.This server sync all the files to a standalone drive nightly as a second backup excepting the raid.
9. The holidays photos are also synced to a second location from my home which is an old time capsule.


----------



## gpw928 (Oct 15, 2022)

I infer that it's a home system, not under sustained load, and that sensible pricing is desirable.

So choose CMR disks without "advanced power management" that fit in your budget.

The Toshiba drive you mention looks to have good performance but the failure rate reported on Newegg is a worry.

I really don't buy enough disks to make a strong recommendation on anything.  You can examine most of the usual options at Newegg.


----------



## cmiu147 (Oct 17, 2022)

I got several dumb questions... and for not opening another thread, I will ask here.

1. I want to migrate the zpool to new disks configured as raid-z1 (raid5). this is doable? The current setup is zfs-raid10.
2. Over time, if I add new disks to the pool, can I expand the pool?
3. What would you choose between some 5y old seagate savvio SAS 10k rpm with a total of 1gb read/write data (was in a server as hot-spare drives), some desktop 7200rpm CMR or some new seagate constellation es3 7200rpm  ( this is the most expensive option)?
The cheapest option is to use the 5y old savvio. I was thinking to an array with 8 disks, configured in raid6.
4. My SAS controller is a ibm 5015... so it cannot be flashed to ITmode. Can I put the controller in JBOD and create zfs raid-z2 or is not recommended?


----------



## gpw928 (Oct 17, 2022)

cmiu147 said:


> I got several dumb questions... and for not opening another thread, I will ask here.



Yes, you use zfs-send(8).  Beware, RAID-Z is slower at writing than mirrors.
Not traditionally, but that's changing.
Used disks are a crap shoot.  If they were extremely low cost, I might run them up and have a close look with smartctl(8).  Consider the cost of replacement 10K SAS disks when they fail (and your cables will be SAS, so you may not be able to switch to SATA easily).  Make sure you understand what sort of data cables you need (LSI controllers typically have custom 4-into-1 cables).  Also look at higher capacity new drives, as their value equation may be better.
I have no specific knowledge of the IBM M5015, but Google suggests it has no IT Mode, and no JBOD option, so not what you want for ZFS.
Building a NAS server can provide a sense of achievement and fun, but you are planning to add a lot of noisy, power hungry, hot disks into your case.  Come summer, you will probably be asking how to improve the cooling, or reduce the noise...  Then there will be backups...


----------



## cmiu147 (Oct 18, 2022)

Hi,
Sorry my fault ... I already own the SAS savvio disks, the controller and also the cables.
The server is on my basement where is a constant temperature. I don't care about the noise of the disks... Is not so much difference in terms of dB between 3.5 SATA 7200rpm and 2.5SAS 10k rpm.

How this bsd machine will work using a hardware raid array instead of zfs raid array?


In the past I was using as a lamp server, a machine with centos8 + hardware raid10 (desktop drives) + lvm + dm-cache on nvme. Was pretty fast compared to my iocage on fbsd13. The fbsd machine is running zfs raid10 on smr drives with iocage for FEMP.

For what I'm using this machine I can use any OS basically... Is like you are taking photos in raw format and then process it... The results will be similar.

Is just a matter of flavour... That's why I want to use BSD.


----------



## gpw928 (Oct 18, 2022)

I don't think you can use the IBM M5015 with ZFS.

You could use it with any O/S in RAID mode -- it just presents virtual disks -- so Linux of FreeBSD should be fine with native file systems other than ZFS.

The Art of Server appears to genuine refurbished LSI controllers flashed to IT mode, if you want to go with ZFS.

Figuring out the best use for the NVMe SSD would take serious some study with ZFS.  Separate ZIL, L2ARC, and "special vdev" are all candidates, but may or may not help performance or be safe, depending on context.

Personally, I'd settle in with my first FreeBSD system.  Get comfortable, and then move on.


----------



## cmiu147 (Oct 18, 2022)

From what I remember m5015 support JBOD according to mfiutil. I will recheck today.


----------



## cmiu147 (Oct 19, 2022)

Yes, m5015 has JBOD mode using mfiutil. Using the LSI interface, JBOD doesn't appear.


----------



## cmiu147 (Oct 19, 2022)

I put a nvme as a cache... The SMR drives are much better now... Infinite better... Next step to move from onboard controller to a JBOD SAS controller like IBM m5015...


----------



## alexseitsinger (Oct 19, 2022)

To this day, I am still flabbergasted by how many people (suckers) actually spend their money on NEW hard drives! I don't think I own a hard drive that was manufactured after 2000. Even though I don't really know what the fuck goes on inside those metal blocks (magic?), and even though I don't normally believe in magic, I DO KNOW that I can trust these old things basically forever.


----------



## Alain De Vos (Oct 19, 2022)

If i would have money i would have been on NVME.
- Number of concurrent traits by the CPU.
- How much Gb Memory.
They remain important settings.
Currenlty i use SSD because of the speed compared to spinning drives.
I use spinning drives for larger data.


----------



## alexseitsinger (Oct 19, 2022)

Alain De Vos said:


> If i would have money i would have been on NVME.
> - Number of concurrent traits by the CPU.
> - How much Gb Memory.
> They remain important settings.
> ...


The re-writ-ability of the spinning magnetic platters makes them ideal for lots of (re)writing. However, Mr. Burn-in has not yet transformed any of my flash drives into some type of useless plastic and sh*t...


----------



## alexseitsinger (Oct 19, 2022)

alexseitsinger said:


> The re-writ-ability of the spinning magnetic platters makes them ideal for lots of (re)writing. However, Mr. Burn-in has not yet transformed any of my flash drives into some type of useless plastic and sh*t...



Unfortunately, I have to self-correct myself--I now recall one flash drive being unusually slow which was likely due to burn-in. Please respect me and pretend you never read my previous (embarassing) message, friends.


----------



## PMc (Oct 19, 2022)

alexseitsinger said:


> To this day, I am still flabbergasted by how many people (suckers) actually spend their money on NEW hard drives!


Well, it depends. Last month I bought a new disk, then found a writing on the label: `DOM:28MAR2017`.
But then, that's different to the new Ultrastar that come with an erased date on the label and zeroed SMART data.


----------

