# Poudriere Bulk Build Slower on One Server



## tuaris (Apr 17, 2021)

I have two servers with nearly identical CPU's (Dual Xeon E5530 @2.40GHz and Dual Xeon E5540 @2.53GHz).   The E5540 is slightly "beefier" with 64 GB of RAM.

I would expect Poudriere to preform the same or better on the E5540, but I am observing the opposite.  The difference is significant.   I currently have a build for the entire ports tree running.  The E5540 has built *4011* packages in *43* hours and the E5530 has so far built *19,591* packages in *57* hours.

Here's some additional details on the hardware and configuration:

*E5540*

HP DL180 G6 (The FreeBSD version isn't a variable since the same performance difference was observed with 12.2)
64 GB of RAM
/var/run/dmesg.boot
HP Smart Array RAID controller with a RAID5 configuration resulting in a single logical volume.


```
# gpart show
=>        34  3906918765  da0  GPT  (1.8T)
          34        1024    1  freebsd-boot  (512K)
        1058   419430400    2  freebsd-ufs  (200G)
   419431458  3355443200    3  freebsd-zfs  (1.6T)
  3774874658   132044140    4  freebsd-swap  (63G)
  3906918798           1       - free -  (512B)

# zpool status
  pool: poudriere
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 02:27:01 with 0 errors on Sun Feb  7 01:16:21 2021
config:

        NAME        STATE     READ WRITE CKSUM
        poudriere   ONLINE       0     0     0
          da0p3     ONLINE       0     0     0
```

*E5530*

IBM HS22 Blade
48 GB of RAM
/var/run/dmesg.boot
4 SAS drives in a JBOD configuration.  RAID is done with ZFS.


```
# gpart show
=>      40  62499920  da0  GPT  (30G)
        40    409600    1  efi  (200M)
    409640  58310656    2  freebsd-ufs  (28G)
  58720296   3125248    3  freebsd-swap  (1.5G)
  61845544    654416       - free -  (320M)

# zpool status
  pool: storage
state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
```

Poudriere is configured (mostly) the same on both systems (E5540, E5530). One difference is that the E5530 doesn't have ccache.  The E5540 saw a slight improvement when ccache was enabled.
I can't imagine that ZFS has that much of a performance advantage?  There must be something else that's slowing down Poudriere on the HP server.


----------



## Jose (Apr 18, 2021)

tuaris said:


> I can't imagine that ZFS has that much of a performance advantage?


Yes, it has:


> The NO_ZFS=yes is the culprit. This causes a huge amount of I/O when setting up (and tearing down) the jails. This results in significant slowdowns even on the dual 6 core Xeons I have at work.











						poudriere and jobs number
					

I'm trying to build packages for i386 machine on amd64 machine. As advised in 59131, I've installed mgmt/poudriere (-devel specifically).  One question. For some reason poudriere builds in one thread each of PARALLEL_JOBS (I set PARALLEL_JOBS=3 in poudriere.conf because without setting it...




					forums.FreeBSD.org


----------



## SirDice (Apr 19, 2021)

Yep. Instead of simply cloning a clean jail to set up a new build (which barely uses any time and I/O access) it has to unpack a clean jail using cpio(1). And it has to do this for each port it builds. All that I/O and time adds up. Especially if you push your I/O to the limit by having a bunch of new jails created at the same time.


----------

