# Rough seas in the ports tree?



## semi-ambivalent (Jun 25, 2020)

I have been doing a complete re-install of a 12.1 box. I'm having a lot of problems getting ports to compile; libreoffice with FreeBSD patches that fail to apply, QT5-core failing to compile, smaller things that are apparently needed by several applications. I've even installed some pkgs just to get some ports to finish. A mess I attribute to being after the release of 11.4 and lots of new changes being committed. (Along with things like thunderbird sharing code with firefox, but I won't mention that.)

What I want to do is roll all of this back to a point where the ports pool environment appeared nice and calm. Can I do a while-in-use point-in-time with, say, portmaster, that doesn't include "rm -rf /usr/local/*"? Or using any other ports manager? origin ports count is 134, and I can prune a bit first. I'd like to do this live.

thx


----------



## Mjölnir (Jun 25, 2020)

Did you build the ports with ports-mgmt/poudriere or ports-mgmt/synth?


----------



## semi-ambivalent (Jun 25, 2020)

mjollnir said:


> Did you build the ports with ports-mgmt/poudriere or ports-mgmt/synth?



portmaster

Happy to try either of these, do they have differing strengths?


----------



## SirDice (Jun 25, 2020)

semi-ambivalent said:


> What I want to do is roll all of this back to a point where the ports pool environment appeared nice and calm


So, that's never. The ports tree is continuously updated and changed, there's never a dull moment. You can checkout a quarterly branch though, that's more or less stable for 3 month periods (only receives security/stability updates during that period).

`svnlite co https://svn.freebsd.org/ports/branches/2020Q2 /usr/ports`

You may want to hold off for a few days though, first days of July a new quarterly branch will get branched off (2020Q3).


----------



## semi-ambivalent (Jun 25, 2020)

SirDice said:


> So, that's never. The ports tree is continuously updated and changed, there's never a dull moment.
> 
> `svnlite co https://svn.freebsd.org/ports/branches/2020Q2 /usr/ports`
> 
> You may want to hold off for a few days though, first days of July a new quarterly branch will get branched off (2020Q3).



True, but there are times of less-dullness. I suppose it doesn't matter if a particular port is or is not a train wreak if I never need it. I'll keep the quarterlies in mind since I no longer need the latest.

thx


----------



## Mjölnir (Jun 25, 2020)

I'm for shure not the only one who suggests not to use ports-mgmt/portmaster (which builds ports in the main system) but _poudriere_ or _synth_ to build your ports.  Installing _devel/ccache_ will save you a great amount of time.


----------



## Mjölnir (Jun 25, 2020)

SirDice said:


> So, that's never. The ports tree is continuously updated and changed, there's never a dull moment. You can checkout a quarterly branch though, that's more or less stable for 3 month periods (only receives security/stability updates during that period).
> 
> `svnlite co https://svn.freebsd.org/ports/branches/2020Q2 /usr/ports`
> 
> You may want to hold off for a few days though, first days of July a new quarterly branch will get branched off (2020Q3).


Which one is the ports tree installed during installation -- the quarterly one or the _head_ branch?


----------



## T-Daemon (Jun 25, 2020)

mjollnir said:


> Which one is the ports tree installed during installation -- the quarterly one or the _head_ branch?


If you refer to the ports.txz package offered during installation, then (outdated now) head branch, assuming the release(7) documentations is up to date and the Builders Release Engineering Team is following the described procedure:

http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/BUILDDATE :

20191101

release(7):

```
For some users, it may be desirable to provide an absolutely clean build
     environment, with no local modifications to the source tree or to
     make.conf(5), and with clean checkouts of specific versions of the doc,
     src, and ports trees.  For this purpose, a script
     (src/release/release.sh) is provided to automate these checkouts and then
     execute “make release” in a clean chroot(8).
```
/src/release/release.sh:

```
# The default svn checkout server, and svn branches for src/, doc/,
        # and ports/.
        SVNROOT="svn://svn.FreeBSD.org/"
        SRCBRANCH="base/head@rHEAD"
        DOCBRANCH="doc/head@rHEAD"
        PORTBRANCH="ports/head@rHEAD"
```


----------



## SirDice (Jun 25, 2020)

mjollnir said:


> Which one is the ports tree installed during installation -- the quarterly one or the _head_ branch?


The one that's installed from a ports.txz on the install media? That's a snapshot of head at the time the -RELEASE was made. A tool like portsnap(8) also grabs head,  aka latest. For pkg(8) however the default is set to quarterly on  -RELEASE versions  and  latest on -STABLE.


----------



## semi-ambivalent (Jun 25, 2020)

mjollnir said:


> I'm for shure not the only one who suggests not to use ports-mgmt/portmaster (which builds ports in the main system) but _poudriere_ or _synth_ to build your ports.  Installing _devel/ccache_ will save you a great amount of time.


I have used portmaster for a very long time now and it has always worked well -- except in these periods where updates are touching complex software. No doubt there are newer solutions but even with those there will be things like this which will cause headaches:


```
bixby [105] [11:11am] [/usr/ports/devel/qt5-core]# make
===>  License LGPL21 accepted by the user
===>   qt5-core-5.14.2_3 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by qt5-core-5.14.2_3 for building
===>  Extracting for qt5-core-5.14.2_3
=> SHA256 Checksum OK for KDE/Qt/5.14.2/qtbase-everywhere-src-5.14.2.tar.xz.
===>  Patching for qt5-core-5.14.2_3
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-configure
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-mkspecs_features_create__cmake.prf
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-mkspecs_features_qt__module.prf
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-mkspecs_common_bsd_bsd.conf
===>  Applying extra patch /usr/ports/devel/qt5/files/extrapatch-mkspecs_freebsd-clang_qmake.conf
===>  Applying FreeBSD patches for qt5-core-5.14.2_3 from /usr/ports/devel/qt5-core/files
Ignoring previously applied (or reversed) patch.
1 out of 1 hunks ignored--saving rejects to src/corelib/global/qfloat16.cpp.rej
===>  FAILED Applying FreeBSD patch-src_corelib_global_qfloat16.cpp
===> Cleanly applied FreeBSD patch(es)  patch-git_276fa83 patch-src_corelib_corelib.pro
===> FAILED to apply cleanly FreeBSD patch(es)  patch-src_corelib_global_qfloat16.cpp
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/qt5-core
```

I will use ccache in the future.

thx all


----------



## drhowarddrfine (Jun 25, 2020)

I still use portmaster. Haven't had any issues in a long time.


----------



## a6h (Jun 25, 2020)

I use poudriere and switch to a stable branch, when I have a problem with unstable port tree, This is an example for branches/2020Q1. I show these steps in 2020Q1 branch, because I've tested it, and it was stable. Change it to more recent branch. And I use some name conventions, refer to the end of the post (Note section).

Install ports-mgmt/poudriere and devel/ccache
`pkg install ports-mgmt/poudriere devel/ccache`

Make few directory, copy settings and change persmissions:
`mkdir -p /usr/ports/distfiles`
`mkdir -p /var/cache/ccache`
`cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf`
`chmod 640 /usr/local/etc/poudriere.conf`

Minimal edit to /usr/local/etc/poudriere.conf

```
FREEBSD_HOST=https://download.FreeBSD.org
CCACHE_DIR=/var/cache/ccache
CHECK_CHANGED_DEPS=yes
CHECK_CHANGED_OPTIONS=verbose
```

List all your packages here: jf121x64-p2020Q1-set0
e.g. I want to build security/libressl and security/ca_root_nss with poudriere. jf121x64-p2020Q1-set0 is going to look like this:

/usr/local/etc/poudriere.d/jf121x64-p2020Q1-set0

```
security/libressl
security/ca_root_nss
```

Create a jail
`poudriere jail -c -j jf121x64 -v 12.1-RELEASE -a amd64`

Create a port tree
`poudriere ports -c -p 2020Q1 -B branches/2020Q1 -m svn`

Edit FreeBSD.conf to refelect poudriere packages:
/usr/local/etc/pkg/repos/FreeBSD.conf

```
Poudriere:{url:"file:///usr/local/poudriere/data/packages/jf121x64-p2020Q1-set0"}
FreeBSD:{enabled:no}
```

View, keep or change ports options:
`poudriere options -j jf121x64 -p 2020Q1 -z set0 -f /usr/local/etc/poudriere.d/jf121x64-p2020Q1-set0`

Build the ports:
`poudriere bulk -j jf121x64 -p 2020Q1 -z set0 -f /usr/local/etc/poudriere.d/jf121x64-p2020Q1-set0`

Update and install packages from poudriere build:
`pkg update -f`
`pkg install security/libressl security/ca_root_nss`

Notes:

jf121x64: it's just a name for jail (Jail FreeBSD 12.1 amd64)
12.1-RELEASE: FreeBSD version
amd64: arch
2020Q1: name for port tree
branches/2020Q1: is the name of branch


----------



## Mjölnir (Jun 26, 2020)

drhowarddrfine said:


> I still use portmaster. Haven't had any issues in a long time.


This might be because you most likely only build ports with a fairly straight/easy set of dependencies.  _Portmaster_ is _broken by design_ concerning handling of dependencies.  In other words: it does not deserve it's name, because this is an essential preliminary to build ports correctly.  If you ever try to build a full KDE workspace/distribution with it, you'll know this is true.  _Portmaster_ tries to calculate _by-feet_ what should be done by integrating make(1), and give it the complete set of dependencies all at once (i.e. all Makefiles of all ports involved).  See the well known Recursive Make Considered Harmful -- a _must-read_ for every developer and people involved in maintaining/building software.  Obviously, there might be dubious rules in some under-maintained Makefiles, which can be another source of failure; that's human and can always happen.  IMHO such thing like _portmaster_ should not be allowed to be available in the FreeBSD ports tree.  When following the recipe given in the mentioned paper, you'll not have such problems (except caused by incomplete rules), and as a nice side-effect, parallel _make_ works correctly because make(1) _stratifies_ the dependency graph.  Spread the word: do not use so-called _portmaster_ until it's fixed.

Besides that, the issues of the OP seem to come from an unclean build environment, i.e. there is some /wrkdir left over from a previous build (_"Ignoring previously applied (or reversed) patch"_).  _Poudriere_ or _synth_ together with _ccache_ will avoid such, because they always build in a fresh jail.


----------



## richardtoohey2 (Jun 26, 2020)

drhowarddrfine said:


> I still use portmaster. Haven't had any issues in a long time.


Same here - works for me looking after Apache, MySQL and PHP and the things they pull in.  Occasional glitch but mostly fine.

There are definitely better and cleaner and more liked tools, and if you are starting with FreeBSD - definitely look at those first.


----------



## drhowarddrfine (Jun 26, 2020)

I've always hesitated to use synth because, when it first came out, many on this forum stated it was more for people who were not technically inclined so I never bothered with it. Poudriere I never bothered with because portmaster worked fine for me and my impression was always that it was more useful when you had a port that needed distribution among several servers, it needed ZFS, and there was more configuration involved. I also question how much space this uses as some of the servers I used were limited. So I never bothered since I don't run my company anymore.

I won't use KDE or Gnome for a desktop and I only install developer-ish software like servers and Gimp and browsers.


----------



## zirias@ (Jun 26, 2020)

mjollnir said:


> _Portmaster_ is _broken by design_ concerning handling of dependencies.


Although I'd never use portmaster as poudriere is _way_ superior, this statement isn't true. The ports tree and make framework in /usr/ports/Mk already handles dependencies correctly. Portmaster doesn't add much here, while poudriere takes a completely different approach by evaluating the dependencies itself, building each port in a clean jail and installing exactly the required dependencies from packages built earlier. The key difference is the "clean environment" build.



mjollnir said:


> _Portmaster_ tries to calculate _by-feet_ what should be done by integrating make(1), and give it the complete set of dependencies all at once (i.e. all Makefiles of all ports involved). See the well known Recursive Make Considered Harmful -- a _must-read_ for every developer and people involved in maintaining/building software. Obviously, there might be dubious rules in some under-maintained Makefiles, which can be another source of failure; that's human and can always happen. IMHO such thing like _portmaster_ should not be allowed to be available in the FreeBSD ports tree. When following the recipe given in the mentioned paper, you'll not have such problems (except caused by incomplete rules), and as a nice side-effect, parallel _make_ works correctly because make(1) _stratifies_ the dependency graph.


And this statement is, in fact, FUD. A port build is "recursive", yes, because that's the only choice you have when you want to build third-party software using all kinds of build-systems from your running make job. But it includes all the steps from downloading the distribution file to packaging the results of the build, so this is _nothing_ like the traditional "recursive make"* and doesn't affect any dependencies. The ports tree itself is designed recursive as well, but that doesn't matter, as the Makefile of any port you build will include all the framework and become the "master" for this build.

The problems you typically run into with a tool like portmaster are all related to unclean build environments -- you could have dependencies installed in non-matching versions, you could run into "automagic" dependencies, etc.

* (As a side note, it's very well possible to get recursive make to operate correctly as well, although I peref to avoid it if possible.)


----------



## Mjölnir (Jun 26, 2020)

Zirias said:


> Although I'd never use portmaster as poudriere is _way_ superior, this statement isn't true. The ports tree and make framework in /usr/ports/Mk already handles dependencies correctly.


So then `make -j 4 fetch-recursive` in a big meta-ports directory to fetch all tar balls needed to build this port with 4 parallel downloads, will succeed and not try to download anything twice? (A message like "xyz-1.2.3.tar.gz already in distfiles; skipping" indicates a broken dependency graph: it is not stratified correctly)


> Portmaster doesn't add much here


Try `portmaster -F x11/kde5` and be _very_ patient...


> And this statement is, in fact, FUD. A port build is "recursive", yes, because that's the only choice you have when you want to build third-party software using all kinds of build-systems from your running make job.


I don't know what _FUD_ means.  Unfortunately, in practice: yes.  The sound solution would be to require the maintainers to rewrite all external _make_ dialects into BSD Makefiles, which is no practical alternative.


> The problems you typically run into with a tool like portmaster are all related to unclean build environments -- you could have dependencies installed in non-matching versions, you could run into "automagic" dependencies, etc.
> ossible.)


See the above example.


----------



## SirDice (Jun 26, 2020)

mjollnir said:


> So then  make -j 4 fetch-recursive in a big meta-ports directory to fetch all tar balls to build this port with 4 parallel downloads, will succeed and not try to download anything twice?


It won't if you remove that `-j 4`. You're trying to run the same thing in 4 parallel  jobs, that's bound to cause race conditions.

Job 1:  checks if file is there. File isn't  there and starts  the download
Job  2: checks if file is there. File isn't there (job 1 hasn't finished downloading yet), and starts the  download.


----------



## Mjölnir (Jun 26, 2020)

SirDice said:


> It won't if you remove that `-j 4`. You're trying to run the same thing in 4 parallel  jobs, that's bound to cause race conditions.
> 
> Job 1:  checks if file is there. File isn't  there and starts  the download
> Job  2: checks if file is there. File isn't there (job 1 hasn't finished downloading yet), and starts the  download.


If the dependency graph was stratified correctly, _make_ would not start a 2nd job to fetch the very same target.
This is the same case like a parallel _make_ will not start a 2nd compile job on the very same target, iff the rules given to it via the Makefiles are correct - which is not always the case due to human mistakes.  That's normal.  Maybe you're refering to these errors as _race conditions._  My understanding of this term does not match this kind of failure.


----------



## zirias@ (Jun 26, 2020)

mjollnir said:


> So then  make -j 4 fetch-recursive in a big meta-ports directory to fetch all tar balls needed to build this port with 4 parallel downloads, will succeed and not try to download anything twice? (A message like "xyz-1.2.3.tar.gz already in distfiles; skipping" indicates a broken dependency graph: it is not stratified correctly)


In the ports tree, make is used as a tool to control jobs, and the dependencies for that work correctly (by using some "stamp files", but they are only created after finishing a job). Make avoids races when the targets are files, they are either there (so make can check the timestamp) or they aren't. This isn't possible in a usecase like the ports tree, distfiles have to be downloaded, checksummed, there are some scripts involved. You can't make a distfile itself a make target, this would kill all the flexibility of the ports framework.

So, yes, the ports tree isn't designed for parallel make, and never was. But it can of course build the actual ports using parallel make if their build system supports it. All of this has nothing to do with portmaster.


----------



## Mjölnir (Jun 26, 2020)

Zirias said:


> [...] So, yes, the ports tree isn't designed for parallel make, and never was. But it can of course build the actual ports using parallel make if their build system supports it. All of this has nothing to do with portmaster.


I am aware that the ports collection and it's handling has evolved over time, and it might add too much complexity to the Makefiles if the inter-port dependencies were reflected in strictly correct rules.  I do not agree that this does not have to do with _portmaster_: contrast fetching the distfiles for a huge project like kde5 done by

`portmaster -F x11/kde5` will ponder the dependencies for _hours_ and download nothing after 1 hour. A newbie is tempted to do this, once s/he took the (IMHO wrong) decision to use _portmaster_ to handle the ports
vs. letting _make_ handle the dependencies: `make -f /usr/ports/x11/kde/Makefile fetch-recursive` will start fetching the distfiles after a few _seconds_ to construct the correct dependency graph (and it will not attempt to fetch the same file twice if the deps rules are correct, even in parallel mode)


----------



## drhowarddrfine (Jun 26, 2020)

mjollnir said:


> I don't know what _FUD_ means.


Fear, Uncertainty, Doubt


----------



## Sevendogsbsd (Jun 26, 2020)

Microsoft pushed FUD campaigns heavily 10+ years ago in a futile attempt to thwart the proliferation of Linux and other open source projects.


----------



## Jose (Jun 26, 2020)

Origins go back to the '80s, and it was originally coined to describe IBM's behavior


> Defined by Gene Amdahl after he left IBM to found his own company: “FUD is the fear, uncertainty, and doubt that IBM sales people instill in the minds of potential customers who might be considering [Amdahl] products.” The idea, of course, was to persuade them to go with safe IBM gear rather than with competitors' equipment. This implicit coercion was traditionally accomplished by promising that Good Things would happen to people who stuck with IBM, but Dark Shadows loomed over the future of competitors' equipment or software. See IBM. After 1990 the term FUD was associated increasingly frequently with Microsoft, and has become generalized to refer to any kind of disinformation used as a competitive weapon.





			FUD


----------



## Sevendogsbsd (Jun 26, 2020)

Ah ok, thanks for the history lesson  I normally blame Microsoft for everything so they get a reprieve on this one...IBM was probably as evil as Microsoft is/was though (thinking of Compaq rise to power).


----------

