# The llvm's have a newer version, i need to recompile 2500 ports.



## Alain De Vos (Dec 7, 2022)

Can this be avoided ? This takes a week on my PC.


----------



## laurentis (Dec 7, 2022)

Aah the new version of llvm!
Yesterday I tried to build devel/llvm15 but that consumed all my memory and swap, drank all my booze, and ordered pizza with toppings I didn't like.
I had to recompile with BE_NATIVE and option FLANG off and then it behaved. 

As for your 2500 dependencies, you could cheat a bit and try to build direct dependencies only.


----------



## Alain De Vos (Dec 7, 2022)

I use poudriere. No idea what direct dependencies mean.


----------



## Datapanic (Dec 7, 2022)

Have you tried ports-mgmt/poudriere-devel and the PACKAGE_FETCH_* options in poudriere.conf?  I think it might be used to fetch packages like llvm* and use that instead of building llvm* package(s), but I might be wrong.  I have been able to fetch the packages I give to PACKAGE_FETCH_WHITELIST but it still builds those packages.


----------



## Alain De Vos (Dec 7, 2022)

Still even if i don't build llvm directly i must recompile thousands of it's dependencies.
Even if llvm goes a "minor-upgrade".


----------



## sko (Dec 8, 2022)

Datapanic said:


> Have you tried ports-mgmt/poudriere-devel and the PACKAGE_FETCH_* options in poudriere.conf?  I think it might be used to fetch packages like llvm* and use that instead of building llvm* package(s), but I might be wrong.  I have been able to fetch the packages I give to PACKAGE_FETCH_WHITELIST but it still builds those packages.



I had the same experience when trying to avoid building rust, which is a ressource-hogging piece of junk (fails 4 out of 5 times on a host with 32GB RAM and kills half of everything running on it during the process...). It seems PACKAGE_FETCH_* is completely ignored or just broken...


----------



## astyle (Dec 8, 2022)

This is why I'm looking at ways to use Poudriere - figuring out how to build just a subset of stuff I want to upgrade (compiled with my own options, of course), while leaving everything else untouched. In my case, that subset is KDE. But the process, once ironed out, should be applicable for just about any subset of packages. A few other subsets could include OpenCV/multimedia stuff... 

That kind of use does have limitations - eventually dependencies become stale, and need to be updated.

BTW, as I recall, OP seems to like make.conf... in there, OP can specify that default LLVM should be v. 13 or 14...


----------



## zirias@ (Dec 9, 2022)

sko said:


> I had the same experience when trying to avoid building rust, which is a ressource-hogging piece of junk (fails 4 out of 5 times on a host with 32GB RAM and kills half of everything running on it during the process...). It seems PACKAGE_FETCH_* is completely ignored or just broken...


It is not. But no matter whether packages are fetched or were built locally, strict rules apply: If a package has different options, a different version, different linked library versions _or even_ different versions of _dependencies_, it is deleted and rebuilt. This is done recursively, so any transitively dependent packages are built as well.

Your ports tree will almost always be a bit newer than the state used to build the latest official packages. Especially with packages depending on lots of other packages or on very "active" packages, you'll end up most of the time building the package because the official package isn't a perfect match any more.

As a workaround, you can checkout the exact revision of the ports tree from which the official packages were built in the last run. Then, only packages affected by your options (and their dependencies) will be built locally.

edit: I didn't find a really sane way to find the commit hash of the latest package build yet, but here's a "hacky" way to find it for 13.1-amd64 "latest" (requires IPv6):

```
fetch -qo- http://beefy16.nyi.freebsd.org/data/131amd64-default/latest-done/.data.json \
    | head -c80 | sed 's/.*buildname":"\([^"]*\)",.*/\1/'
```


----------



## sko (Dec 9, 2022)

I haven't touched the options for rust or its dependencies; I even tried with a new, default set of options for testing and tried building firefox-esr, for which rust is built as a dependency. Despite rust being listed in "PACKAGE_FETCH_WHITELIST", and the correct _BRANCH and _URL being set; rust is never pulled from packages. Same for "gcc*", "llvm" , "py39-*" or "gtk*" listed in _WHITELIST - Poudriere *always* builds 100% of all dependencies.


----------



## zirias@ (Dec 9, 2022)

That's not true and I literally gave the complete explanation above: You're not on exactly the same state of the ports tree as was used for the official packages ....


----------



## Datapanic (Dec 9, 2022)

Zirias explanation is what I just saw here.  I have llvm* in the PACKAGE_FETCH_WHITELIST and they did get fetched, but in the Sanity check a missing dependency was found (libedit-3.1.20021030,1) so the fetched packages were deleted:


```
[00:01:04] Package fetch: Will fetch 3 packages from remote or local pkg cache
Updating database digests format: 100%
The following packages will be fetched:

New packages to be FETCHED:
        llvm11: 11.0.1_7 (107 MiB: 28.11% of the 381 MiB to download)
        llvm12: 12.0.1_10 (114 MiB: 30.00% of the 381 MiB to download)
        llvm13: 13.0.1_3 (159 MiB: 41.89% of the 381 MiB to download)

Number of packages to be fetched: 3

The process will require 381 MiB more space.
381 MiB to be downloaded.
[freebsd_12-4x64-HEAD-12] Fetching llvm12-12.0.1_10.pkg: 100%  114 MiB   2.0MB/s    00:59
[freebsd_12-4x64-HEAD-12] Fetching llvm11-11.0.1_7.pkg: 100%  107 MiB   1.9MB/s    00:58
[freebsd_12-4x64-HEAD-12] Fetching llvm13-13.0.1_3.pkg: 100%  159 MiB   2.4MB/s    01:11
[00:04:17] Package fetch: Using cached copy of llvm11-11.0.1_7
[00:04:17] Package fetch: Using cached copy of llvm12-12.0.1_10
[00:04:17] Package fetch: Using cached copy of llvm13-13.0.1_3
[00:04:17] Sanity checking the repository
[00:04:17] Checking packages for incremental rebuild needs
[00:04:17] Deleting llvm11-11.0.1_7.pkg: missing dependency: libedit-3.1.20221030,1
[00:04:17] Deleting llvm12-12.0.1_10.pkg: missing dependency: libedit-3.1.20221030,1
[00:04:17] Deleting llvm13-13.0.1_3.pkg: missing dependency: libedit-3.1.20221030,1
[00:04:17] Deleting stale symlinks... done
[00:04:17] Deleting empty directories... done
[00:04:17] Package fetch: Generating logs for fetched packages
[00:04:17] Unqueueing existing packages
[00:04:17] Unqueueing orphaned build dependencies
[00:04:19] Sanity checking build queue
```

My poudriere build script always updates the jails and the ports tree before the bulk build.  With Zirias's explanation, it may only be once in a blue moon when these packages are fetched and used...


----------



## VladiBG (Dec 9, 2022)

If you run low on memory during the build of llvm13 you can try to uncheck "Flang FORTRAN compiler" in the config option of the devel/llvm13


----------



## zirias@ (Dec 9, 2022)

Datapanic said:


> My poudriere build script always updates the jails and the ports tree before the bulk build.


You _could_ add an additional `git checkout <hash>` to these scripts with a hash obtained from one of the official builders...

What's your architecture, FreeBSD version and ports branch?


----------



## Alain De Vos (Dec 9, 2022)

I have

```
DEFAULT_VERSIONS+= ssl=openssl
```
With some 1500 dependencies...


----------



## Datapanic (Dec 9, 2022)

zirias@ said:


> You _could_ add an additional `git checkout <hash>` to these scripts with a hash obtained from one of the official builders...
> 
> What's your architecture, FreeBSD version and ports branch?


I have a VMWare vm building all the time with 4 jails - FreeBSD_12-4-RELEASE amd64 64GB .  It always uses the latest ports branch.  It runs forever in a cycle of Upgrade Jails -> Upgrade Ports -> PortShaker -> Bulk Builds -> Clean Logs -> Clean Distfiles -> Clean Old Pkgs.  Flags are checked and any of these steps can be skipped or added.  It sleeps at the end for X amount of time and can be set to sleep less if there is a failed build.  It also uses the poudriere hooks script bulk.sh to handle what to do on different statuses.  It's a pretty cool script that is configured under rc.conf.

I don't want to go off-topic of Alain's thread.  Maybe the thread can be split off to a more specific thread about poudriere administation


----------



## zirias@ (Dec 9, 2022)

Datapanic, well, try adding something like this to your scripts before starting the next bulk build:

```
git checkout $(fetch -qo- \
    http://beefy6.nyi.freebsd.org/data/123amd64-default/latest-done/.data.json \
    | head -c80 | sed 's/.*buildname":"\([^"]*\)",.*/\1/')
```
(The official builders still build packages for 12 on 12.3, you'll have to change this URL once 12.3 is EOL)

Then, you'll probably also need `git checkout main` before updating your ports tree again.

This assumes you're already getting your ports tree directly from git....

edit: as for hijacking Alain's thread, I _think_ package fetching might help him solve his problem as well, so it's IMHO not too far off-topic, but of course up to the mods to decide


----------



## Alain De Vos (Dec 9, 2022)

In my current poudriere make.conf i have:

```
DEFAULT_VERSIONS+= ssl=openssl
```
That's 1500 package dependencies of 1/third of all my packages installed.

In a good strategy you take almost all from binary- packages and a few/none source-ports.
Or you take almost all from source-ports and a few/none  binary-packages.

Mixing 50-50 does not seem whise as .so or .a hell.


----------



## astyle (Dec 9, 2022)

Alain De Vos said:


> Mixing 50-50 does not seem whise as .so or .a hell.


Duh


----------



## Alain De Vos (Dec 9, 2022)

Or you go mainly for the ports or you go mainly for the packages. But too much mixing is not a good idea ?


----------



## Datapanic (Dec 10, 2022)

zirias@ said:


> Datapanic, well, try adding something like this to your scripts before starting the next bulk build:
> 
> ```
> git checkout $(fetch -qo- \
> ...



I'm running into issues with fetch and ipv6 to the beefy servers.  Getting "No address record" without -6 in fetch's switches and "Non-recoverable resolver failure" when -6 is used.  ISP is T-Mobile Home Internet but I am getting the same results with my work builder.  I wish the beefies had IPv4 addresses...


----------



## astyle (Dec 10, 2022)

Datapanic said:


> I'm running into issues with fetch and ipv6 to the beefy servers.  Getting "No address record" without -6 in fetch's switches and "Non-recoverable resolver failure" when -6 is used.  ISP is T-Mobile Home Internet but I am getting the same results with my work builder.  I wish the beefies had IPv4 addresses...


beefy6 doesn't have an IPv4 address... if it did, any IPv4-only machine would be able to ping it. Yeah, there's lots of details I'm omitting here, but long story short, beefy6 is IPv6-only.


----------



## zirias@ (Dec 10, 2022)

Datapanic said:


> I'm running into issues with fetch and ipv6 to the beefy servers.  Getting "No address record" without -6 in fetch's switches and "Non-recoverable resolver failure" when -6 is used.  ISP is T-Mobile Home Internet but I am getting the same results with my work builder.  I wish the beefies had IPv4 addresses...


Well, the problem certainly isn't IPv6 here, _maybe_ a broken IPv6 connectivity (in which case you could try for example tunnelbroker.net for a free IPv6-tunnel (on v4)). I've never had any issues with the official builders so far

```
$ host beefy6.nyi.freebsd.org
beefy6.nyi.freebsd.org has IPv6 address 2610:1c1:1:6080::16:15
beefy6.nyi.freebsd.org mail is handled by 0 .
 
$ ping beefy6.nyi.freebsd.org
PING6(56=40+8+8 bytes) [MY_HOST] --> 2610:1c1:1:6080::16:15
16 bytes from 2610:1c1:1:6080::16:15, icmp_seq=0 hlim=47 time=116.228 ms
16 bytes from 2610:1c1:1:6080::16:15, icmp_seq=1 hlim=47 time=97.309 ms
^C
```

That said someone on here recently posted that the builders are (partially?) also reachable behind some v4 reverse proxy? Unfortunately, I can't find the corresponding thread.


----------



## zirias@ (Dec 10, 2022)

Datapanic I found a slightly more "elegant" way to obtain the info directly from the pkg-status overview (which is available on IPv4) and wrote a little example script:

```
#!/bin/sh

RELEASE=${1:-13.1}
ARCH=${2:-amd64}
BRANCH=${3:-latest}

if [ "$BRANCH" = "quarterly" ]; then PORTS=quarterly; else PORTS=default; fi
if [ "$RELEASE" = "CURRENT" ]; then
    JAIL=main-$ARCH
    HASHPAT='p[0-9a-f]*_s[0-9a-f]*'
    HASHCAP='p\([^_]*\)_s.*'
else
    JAIL="$(echo $RELEASE | sed -e 's:\.::')$ARCH"
    HASHPAT='[0-9a-f]*'
    HASHCAP='\([^"]*\)'
fi

fetch -qo- \
    "https://pkg-status.freebsd.org/api/1/builds?type=package&latest=True" \
    | sed -e 's@{"_id":"@\n@g' \
    | grep ":${PORTS}:" \
    | grep '"jailname":"'$JAIL'"' \
    | grep '"remaining":0' \
    | grep '"buildname":"'$HASHPAT'"' \
    | sed -e 's@.*"buildname":"'$HASHCAP'".*@\1@'
```

Example usage:

```
$ ./getporthash.sh
908640ce3450
$ ./getporthash.sh 12.3 i386
bdb5353bacce
```

Edit: Of course this is *still* hacky, you don't parse json with grep/sed, and if pkg-status ever changes some formatting, this script will break for sure  but hey, for now it works!

Edit2: little improvements, script now also supports -CURRENT and matches "0 remaining packages" instead of a specific status.

Edit3: I am a bit unsure how reliable the information obtained directly from pkg-status is ... it definitely looks fishy for -CURRENT, most builds listed are very old. On a quick glance, it looks somewhat sane for -RELEASE versions.


----------

