# Do not use ports-mgmt/portmaster and other tools who build in the main system



## Mjölnir (Jun 26, 2020)

Please consult _Howto build ports with ports-mgmt/poudriere_ for a set of historically evolved issues concerning the ports tree.
Please consult the well-known _Recursive Make Considered Harmful_ for reasons why such tools do not handle the build process theoretically sound -- they do not use make(1) correctly to compute the graph of dependencies.
These tools have other issues, mainly that building in the main system clutters your system with build-deps/build-tools you do not need otherwise.
Instead, use ports-mgmt/poudriere or ports-mgmt/synth to build your ports.
These always build in a fresh, clean jail.  Installing devel/ccache is optional but common practice, and integration into these tools will save a great amount of build time.  The vast majority of ports build fine with _ccache_. 
This adds a small amount of planing & taking a few decisions, and manual setup & configuration _once_ (~1 hour). The benefit is much less hazzle when building ports until s/th basically changes in these two more advanced build tools.
Period.


----------



## richardtoohey2 (Jun 26, 2020)

I thought synth was about as popular (with the FreeBSD developers) as portmaster?  That is to say, not popular at all.
portmaster for all its faults, works for me, but going forward I'm going to try binary packages with pkg.

My understanding (I could very definitely be wrong!) of the current "best" way of dealing with third-party code is:

1) If you don't *need* to use ports, then use pkg and binary packages.
2) If you do need to use ports - use poudriere to build the ports (and then, I think, pkg to distribute the binaries you've built).

I've tried poudriere and just found the extra faff of jails etc. too much for my attention span.  A few make installs and portsnap & portmaster (to keep things up-to-date) - seems to be easier and work most of the time.  BUT it's not liked by the developers who have kindly looked after it.


----------



## VladiBG (Jun 26, 2020)

I don't agree with above statement.
I'm using FreeBSD from RELEASE 6.0 and back then there was no _portmaster _script to help you upgrade your ports. The ports was installed manually and then you can upgrade them manually or using _portupgrade _which i learn and still use today. The _portupgrade _uses database and depend of ruby so after a while _portmaster _came as script only replacement of _portupgrade _with no db dependencies. During build process all ports install the build-deps which then you can remove with `pkg autoremove` but it doesn't hurt to leave them as you need them again on the next upgrade.
When you need to build packages for different architecture or maintaining a farm of servers which have same program with different build options using only above tools it was a nightmare. To fill this gap _poudriere _was born which made this build process easy but it didn't replace the other tools like _portupgrade _and _portmaster_. _Synth _came after that as custom repository.

So in short retrospect it was:

2002 FreeBSD 4.5 - ports-mgmt/portupgrade
2007 FreeBSD 6.2 - ports-mgmt/portmaster
2012 FreeBSD 8.3 - ports-mgmt/poudriere
2016 FreeBSD 11 - ports-mgmt/synth

Every tool have different purpose and depending of your goal you can use which suit you best.

If you need to build a program with different build option then you have to stop using pkg and build and upgrade all your programs from the ports. It's a very time consuming especially if you are using GUI as some ports may took days to build and consume huge amount of disk space. Exception of this is only when you need to upgrade the pkg using `pkg-static install pkg`


----------



## Mjölnir (Jun 26, 2020)

VladiBG said:


> I don't agree with above statement.
> I'm using FreeBSD from RELEASE 6.0 and back then there was no _portmaster _script to help you upgrade your ports. [...]


You don't have to, because this is not a question of agreement, but right and wrong  Did you actually read the two sources/links mentioned above?  The dependies of the ports are declared in their Makefiles and tools like so-called portmaster(8) try to process these depencies _by-feet_ (themself) instead of letting the program do it that is dedicated to this - make(1), which is mature since decades and does this well.  Of course: there is no _make_ library you can use to integrate this functionality in external programs or scripts; there is no switch to let _make_ write out the computed dependency graph in a parsable format.  Did you ever try a command like this

`portmaster  -F x11/kde5` (recursively fetch all distfiles needed to build kde5)
The result is that after more than one hour of uselessly processing the dependency graph, this self-called _master_ still fetches _nothing_.
In contrast, `cd /usr/ports/x11/kde5; make fetch-recursive` starts to fetch the files after a few seconds.


----------



## SKull (Jun 26, 2020)

I'm afraid I can't answer the poll. The question makes no sense.

I use pkg for pretty much everything.
Only for stuff where I want none default options I use ports.
And I use them with a simple:

```
make config && make install clean
```


----------



## VladiBG (Jun 26, 2020)

mjollnir said:


> tools like so-called portmaster(8) try to process these depencies _by-feet_ (themself)


Can you explain it in details?

btw. did you read the manual of portmaster and did you check the source code of it? (rhetorical question)
Portmaster is a just a big shell script which help you to use the standard tools to track the dependence and upgrade the ports.


----------



## zirias@ (Jun 26, 2020)

I agree with the recommendation to prefer poudriere. The builds in clean jails avoid a lot of common problems. You need ZFS for efficient operation (so jails can be quickly created from snapshots)
I'm not sure about synth. It sounds like a good alternative, but AFAIK, it's discontinued. There's dsynth, maybe this is coming to FreeBSD as well?
The problems have _nothing_ to do with "recursive make", that's a red herring here. Recursive make can have its problems, I agree to better avoid it, but something like the ports tree can't be implemented non-recursive in a sane way, and it still works fine. The whole thing is *only* about the clean build environments tools like poudriere and (d)synth offer, which are crucial to avoid some typical problems.


----------



## Sevendogsbsd (Jun 26, 2020)

ports-mgmt/synth hasn't had an update in a year but out isn't flagged as "no maintainer" or anything. I used it a few months ago and it worked fine. For building on the machine you are using the ports for, I prefer it to ports-mgmt/poudriere because it is far simpler. Poudriere isn't that complicated but the set up takes a bit of time.


----------



## mickey (Jun 26, 2020)

Building and installing software on a live system is exactly the way the ports tree was designed to be used in the first place.
A clean room environment should not even be necessary. I also wouldn't say it avoids problems, rather that it circumvents problems thereby masking the existence of such problems which in turn makes it more unlikely that such problems ever get addressed.
I have been using the ports tree (and portmaster/portupgrade) almost exclusively from the very beginnings and never had any major headaches I wasn't able to somehow solve by myself. When stuff breaks it is often due to software X version B using a header file or library that software X version A installed on the system, instead of using the correct version that comes bundled with the source. Of course a clean room environment would circumvent such problems, but that is hardly a solution to the problem in itself.


----------



## Mjölnir (Jun 26, 2020)

VladiBG said:


> Can you explain it in details?


just do `portmaster -F x11/kde5` and see how it works... it does not use _make_ but goes _by-feet_. Do not hesitate: it will not download anything within a reasonable amount of time, so you can cancel it before it starts to fetch the 1st tar ball.


> btw. did you read the manual of portmaster and did you check the source code of it? (rhetorical question)
> Portmaster is a just a big shell script which help you to use the standard tools to track the dependence and build/install the ports.


Yes I did both when I ran into these kind of bugs.  After several hours of trying to understand the source and where I can fix this, I filed in a bug report.  A shell script should not have more than about 1k lines of source code.  _portmaster_ is beeing rewritten in LUA due to the fact that shell scipts are not suitable for such complex tasks.  And no, it is not helpful if a tool that calls itself _master_ violates basic principles of software engineering resulting in the numerous bugs you can find on bugzilla.


----------



## VladiBG (Jun 26, 2020)

Ok, i will try it


----------



## Jose (Jun 26, 2020)

This is a matter of some debate on the -ports mailing list. Personally, I'm for separating the build system from the deployment system. I did the compile-and-install on the same machine thing for more than a decade. It works great for snowflake systems, but breaks down badly when you try to automate system setup.

Portmaster, vanilla ports, and derived systems like Gentoo portage don't lend themselves to reproducible system builds. You're going to have to separate building packages from the system they're installed into if you want to build system images automatically. Building the build system is one of the things I hope to automate someday when I slay my procrastination demons.

This is just my opinion. There are plenty on folks on -ports that still use Portmaster and/or expect to be able to install software by doing a `make install` in the ports tree. This debate is far from being settled. I'm definitely rooting for the Poudriere team, though.


----------



## VladiBG (Jun 26, 2020)

I see what you mean. When you try to fetch the entire metaport of KDE5 without using the port-index portmaster goes trough all Makefiles to build the depend-list  and then start to download the dist under /usr/ports/distfiles



mjollnir said:


> it does not use _make_ but goes _by-feet_.



It actually use _make_ see bellow


```
make_dep_list () {
    local dep_type var_opt

    for dep_type in $*; do
        case $dep_type in
        all-depends-list|build-depends-list)
            var_opt="$var_opt -V PKG_DEPENDS -V EXTRACT_DEPENDS \
                -V PATCH_DEPENDS -V FETCH_DEPENDS -V BUILD_DEPENDS -V LIB_DEPENDS"
            [ "$dep_type" = all-depends-list ] && var_opt="$var_opt -V RUN_DEPENDS" ;;
        run-depends-list)
            var_opt="$var_opt -V RUN_DEPENDS -V LIB_DEPENDS" ;;
        test-depends-list)
            var_opt="$var_opt -V TEST_DEPENDS" ;;
        *)
            fail "make_dep_list: Unsupported option '$dep_type'"
        esac
    done
    [ -n "$var_opt" ] && make $var_opt | tr ' ' '\n' | cut -d: -f2 | sort -u
}
```
 

portmaster and portupgrade are tools for upgrading already installed ports on the system and they will fail to fetch meta-port which by itself doesn't install files. To fetch such ports you need to use  ports(7)


----------



## zirias@ (Jun 26, 2020)

mickey said:


> Building and installing software on a live system is exactly the way the ports tree was designed to be used in the first place.
> A clean room environment should not even be necessary. I also wouldn't say it avoids problems, rather that it circumvents problems thereby masking the existence of such problems which in turn makes it more unlikely that such problems ever get addressed.
> I have been using the ports tree (and portmaster/portupgrade) almost exclusively from the very beginnings and never had any major headaches I wasn't able to somehow solve by myself. When stuff breaks it is often due to software X version B using a header file or library that software X version A installed on the system, instead of using the correct version that comes bundled with the source. Of course a clean room environment would circumvent such problems, but that is hardly a solution to the problem in itself.



Strongly disagree with most of that. But let's start with the part I agree with: You have a point about "masking" problems in the sense that: If ports are never tested in an "unclean" environment at all, it's very likely that they will miss any/most necessary *_CONFLICTS, cause these won't show in a clean environment. But OTOH, why even bother, if clean build environments are possible nowadays on virtually any hardware?

Now for my disagreement: The ports tree was designed for, well, *building ports*  Back when it was first created, building on the "live" system was basically the only option. That doesn't mean it isn't suited for usage in a clean build environment. The ports tree always worked as designed, in any environment -- and in an "unclean" environment, you were from time to time required to solve problems that arose because of that environment. If you know what you do, this is very well possible (and, will always be), but even then, it's just annoying to run into these kind of problems (and solving them) all over again. And I'm quite positive it will *never* be possible to avoid all these problems upfront, cause as a ports maintainer, there's no way to foresee any environmental condition the port could ever be built in.

So, agree with another statement of yours, a clean environment shouldn't be strictly necessary. I don't think it will ever be, the ports tree will always support building a port by just issuing a `make` in its directory. *But:* If you're building all your software yourself, you can avoid a lot of unnecessary annoyance by just using a tool like e.g. poudriere.


----------



## ljboiler (Jun 26, 2020)

VladiBG said:


> portmaster and portupgrade are tools for upgrading already installed ports on the system


portmaster can be used to install something that's not already on the system - I've used it often.



VladiBG said:


> portmaster goes trough all Makefiles to build the depend-list via RUN_DEPENDS


And don't forget there are the BUILD_DEPENDS along with the RUN_DEPENDS when building _any_ of the ports.  That list can get rather long.


----------



## memreflect (Jun 27, 2020)

Zirias said:


> The ports tree always worked as designed, in any environment -- and in an "unclean" environment, you were from time to time required to solve problems that arose because of that environment.


Poudriere's clean environment is not necessarily compatible with the behavior of the ports tree in its current form.  For example, `USES=ncurses` automatically selects whether to build using devel/ncurses or the base system's ncurses when building a port dependent upon ncurses.  The selection is based upon whether devel/ncurses is already installed on the system, which means any jail originally created by Poudriere will never use the ncurses port without a dependency on the ncurses port.  One workaround involves modifying the jail directly (or cloning it and creating a separate jail).  Another workaround would be to modify the ports tree used by introducing a make variable NCURSES_DEFAULT that can be set using DEFAULT_VERSIONS in an applicable make.conf file, similar to using `DEFAULT_VERSIONS+= ssl=libressl` to set the SSL library desired.  The latter option could be used with Poudriere's "sets" with no need for a completely separate jail, and it would also work for people building on a live system using make, portupgrade, portmaster, etc.



Zirias said:


> If you're building all your software yourself, you can avoid a lot of unnecessary annoyance by just using a tool like e.g. poudriere.


Despite the long build times for ports of software written in languages like Go, Rust, and Haskell that have package managers already and would benefit from a shared cache between builds of different ports using the same packages, Poudriere is definitely less hassle, not to mention its manpages are very readable compared to portmaster.  And if you need to build packages with different make.conf configurations—as one might need to do with different SSL versions, different Python 3 versions, etc.—Poudriere's flexibility in that regard is invaluable.


----------



## zirias@ (Jun 27, 2020)

memreflect said:


> Poudriere's clean environment is not necessarily compatible with the behavior of the ports tree in its current form. For example,  USES=ncurses automatically selects whether to build using devel/ncurses or the base system's ncurses when building a port dependent upon ncurses. The selection is based upon whether devel/ncurses is already installed on the system


That's a so-called "automagic dependency", which should be avoided in any case. They happen by accident when building on a live system, but they definitely shouldn't be expected behavior.


memreflect said:


> Another workaround would be to modify the ports tree used by introducing a make variable NCURSES_DEFAULT that can be set using DEFAULT_VERSIONS in an applicable make.conf file, similar to using  DEFAULT_VERSIONS+= ssl=libressl to set the SSL library desired.


I wouldn't call that a "workaround" but the only correct solution. Do you know any more examples? I would just assume nobody got around fixing this one yet...


----------



## Jose (Jun 28, 2020)

memreflect said:


> Despite the long build times for ports of software written in languages like Go, Rust, and Haskell that have package managers already...


This is not the solution, it's part of the problem. Setting aside for the moment the antisocial "I'm gonna download all my own stuff and keep it in a secret place" approach these language-specific package managers take, what happens when they depend on a C library that some other language also needs? What happens when there's a version of that library already in the base system?

The approach of downloading all dependencies at install time also guarantees that two installs of the same packages are likely to be different if they were installed a few hours apart. Makes troubleshooting nice and simple.

Package management should be provided by the operating system. I'm not interested in your shiny new buzzword-compliant language if it won't play nice with my system's package manager.


----------



## memreflect (Jun 28, 2020)

Jose said:


> This is not the solution, it's part of the problem.


I respectfully disagree.  If you can't build software written in a language like C due to a missing dependency, you only have two options:

Mail the sysadmin requesting the dependency be installed and hope they fulfill your request.  On a personal machine, you can forgo this and just install the dependency yourself via pkg/ports.  Of course, if your request is denied, or there is no port for the dependency, you only have one other option:
Build and install the dependency yourself.  Hopefully it doesn't have its own dependencies that need built, and hopefully it builds without any errors.  Otherwise, you need to determine what went wrong, which can take up quite some time that could be spent being productive.
With languages that have language-specific package managers, you can generally bypass both problematic issues.  That's not to say there aren't problems, but such problems do tend to be easier to deal with than, say, detecting a recent-enough version in a configure script because there is no associated pkg-config file allowing you to require a version easily.



Jose said:


> Setting aside for the moment the antisocial "I'm gonna download all my own stuff and keep it in a secret place" approach these language-specific package managers take,


This isn't antisocial behavior: if you don't have the access rights to install software in a global location and have no way to temporarily gain access (e.g. sudo, doas), it has to be installed somewhere in your HOME or some other location where you have access.



Jose said:


> what happens when they depend on a C library that some other language also needs?


It fails to work because C isn't a part of that language's ecosystem and can't be downloaded automatically using that tool.  That's the domain of the system's package manager or manual installation in one's HOME since there is no package manager for C (nor should there be; we have "build systems" like Autotools, CMake, Meson, etc. that are used to customize builds out of necessity).  It's usually up to the port maintainers/packagers for such languages to deal with include path resolution and linker paths, and this mostly works.  Occasionally, you might need to add a path or two.  I believe Go and Rust support CFLAGS, CXXFLAGS, LDFLAGS, etc., making it quite simple for them.

Allow me to clarify something: these "package managers" are _not_ intended to replace your system's package manager.  Some are little more than package downloaders that automatically build and install the package in some cache with version information about the package stored somewhere.  The system package manager and the ports tree serve as a convenient way to install software written in any language while the package management solutions intended solely for use by $LANGUAGE are meant to fetch language-specific software in the event that a sufficient version is not installed.



Jose said:


> What happens when there's a version of that library already in the base system?


If such a library is already installed in a known location, and it satisfies the version requirements of whatever software is being built/installed, then it will be used.  Otherwise, the source of that library will be downloaded, built, and installed first, using the language's package manager.  As an example, cc(1) won't look in /usr/local for headers or libraries by default.  If cc acted like the Go compiler, it might try to fetch the dependencies itself, instead of failing to compile, based on some file containing a list of required dependencies.  Alternatively, it might fail to compile, and you'd need to use something like `ccget $URL` to download, build, and install the dependency into a hypothetical "C package cache".



Jose said:


> The approach of downloading all dependencies at install time also guarantees that two installs of the same packages are likely to be different if they were installed a few hours apart. Makes troubleshooting nice and simple.


I'm not certain what you're talking about here honestly, so I can't comment.



Jose said:


> Package management should be provided by the operating system. I'm not interested in your shiny new buzzword-compliant language if it won't play nice with my system's package manager.


That's a problem for system package/port maintainers, not the programmers creating the software.  If there isn't a port for some language-specific package, it's going to need to be downloaded.

audio/spotifyd requires 340 Cargo crates (packages), and audio/spotify-tui requires 219 crates.  Out of those 559 crates that will be built, 343 are unique to either port, and 108 are shared, for a total of 451 crates actually required to be built, meaning 23.947% of the required crates are built twice.  In the entire 2020Q2 ports tree, 9898 total crates are listed with only 2837 crates actually being unique, and "28.662% of the total number of crates listed are unique" is the same as saying "71.338% of the total number of crates listed are potentially built more than once."  That's horribly inefficient.  Of course, nobody builds all the ports, except for the people who do.

Poudriere using a clean environment to build ports is obviously a good idea.  Maybe this problem is unique to ports that require Cargo for dependencies.  Maybe Go ports, Node.js ports requiring www/npm, Ruby ports that use gems, etc. are "less wasteful" compared to Rust ports that require Cargo to download dependencies.  But if not, building ports in bulk would certainly benefit from a shared cache.


----------



## Mjölnir (Jun 28, 2020)

There is no clean solution in an imperfect environment...


----------



## Jose (Jun 28, 2020)

memreflect said:


> I respectfully disagree.  If you can't build software written in a language like C due to a missing dependency, you only have two options:
> 
> Mail the sysadmin requesting the dependency be installed and hope they fulfill your request...


This is an organizational problem. You should have root on some dev system, or better, your own virtual environment if you're writing software. You can't write code to solve organizational problems. Inventing your own little package manager to work around your org's stupid policies only creates more problems.



memreflect said:


> I believe Go and Rust support CFLAGS, CXXFLAGS, LDFLAGS, etc., making it quite simple for them.


I prefer to learn only one package manager, and then I can know instead of guessing, believing, and hoping.



memreflect said:


> Allow me to clarify something: these "package managers" are _not_ intended to replace your system's package manager.  Some are little more than package downloaders that automatically build and install the package in some cache with version information about the package stored somewhere.


How is this not antisocial behavior? I don't want to hunt down the troublesome library in all the little nooks and crannies your idiosyncratic package manager uses. It's always easier to do things your own way than to learn how things are done in order to cooperate with others.



memreflect said:


> I'm not certain what you're talking about here honestly, so I can't comment.


Jose's package manager (jpm) has a feature that allows you to specify minimum versions for dependencies. You can declare, for example, that your package foo depends on library bar version 2.0 or later. I install package foo and get libbar.so.2.0. An hour later, there's a release of libbar 2.1. You install foo and now you get libbar.so.2.1. This new version of bar exposes a bug in foo. You send me a stack trace. I think you're off your rocker 'cause there's no way I can get that stack trace on my install.

Meanwhile, all of libbar's extensive test suite passes with flying colors. The code for foo has not changed. Welcome to a couple of hours in WTF land.

Edit: And this is a very simple example. Imagine something like this happening 3 layers deep in some package with hundreds of gem crates.



memreflect said:


> That's a problem for system package/port maintainers, not the programmers creating the software.  If there isn't a port for some language-specific package, it's going to need to be downloaded.


This is the crux of the problem. If the programmers creating the software can't be bothered to learn their chosen system's packaging tools, they shouldn't be programmers at all.



memreflect said:


> audio/spotifyd requires 340 Cargo crates (packages)...


Thanks for doing this legwork. I will continue to avoid Rust and Spotify.


----------



## memreflect (Jun 28, 2020)

Jose said:


> I don't want to hunt down the troublesome library in all the little nooks and crannies your idiosyncratic package manager uses. It's always easier to do things your own way than to learn how things are done in order to cooperate with others.


You're completely right about cooperation, but it's not always possible.  Many Cargo crates are tagged as v0.X.Y, meaning their interface is unstable; they're effectively usable beta versions.  Individual pieces of software have their own dependencies, and it's better for that software to manage them instead of forcing that burden on maintainers, at least while that software is immature.  That's the entire point of the language-specific package managers: facilitating the use of open-source software in a way that simplifies development.  I can fetch a specific version of a Go project from GitHub for use in my own software easily, but I can't use pkg, apt, yum/dnf, etc. to do anything like that.

As for bug hunting, it's no different compared to any other language.



Jose said:


> Jose's package manager (jpm) has a feature that allows you to specify minimum versions for dependencies. You can declare, for example, that your package foo depends on library bar version 2.0 or later. I install package foo and get libbar.so.2.0. An hour later, there's a release of libbar 2.1. You install foo and now you get libbar.so.2.1. This new version of bar exposes a bug in foo. You send me a stack trace. I think you're off your rocker 'cause there's no way I can get that stack trace on my install.
> 
> Meanwhile, all of libbar's extensive test suite passes with flying colors. The code for foo has not changed. Welcome to a couple of hours in WTF land.


I think I understand your point now.  You're suggesting that because some other package P updated libbar from v2.0 to v2.1, suddenly foo will rely on libbar v2.1 instead of libbar v2.0.  But that's not how it works.  You can have multiple versions installed.  This may seem wasteful, but the problem you're describing is exactly why there's a reliance on minimum versions.  If v2.0 is specified as the minimum version (i.e. >=v2.0), then v2.0 will be used, even if v2.9 is installed.  There is no breakage unless you decide to bump your dependency requirements to a newer version.  Once you do that, it's your job to figure out what went wrong, if anything.

That's where these language-specific package managers differ from a more traditional system package manager: software shouldn't break just because your system suddenly upgraded a library, but if it does, you should be able to use the older version of the library until you figure out what happened because that's the one that worked.  Meanwhile, others are still building using the working code instead of multiple people wasting time hunting a bug down because of a system upgrade that broke the program.



Jose said:


> This is the crux of the problem. If the programmers creating the software can't be bothered to learn their chosen system's packaging tools, they shouldn't be programmers at all.


And if that chosen system is the Debian, Fedora, and Arch Linux distributions as well as FreeBSD?  Just because I can create system-specific packages/ports for each of those systems, that doesn't change the fact that one or more dependencies I need to build that binary package may not be available as system packages or are even the wrong versions (e.g. maybe there's a bug in v2.1 and v2.1.1 hasn't been released yet).  With a language-specific package manager, I can build software with no worries about that because of the version requirements specified, instead of fighting with my development system.  In cases where a system-specific package is needed for me to build my software, it's up to me to deal with that when creating the package/port.



Jose said:


> Thanks for doing this legwork. I will continue to avoid Rust and Spotify.


At this point, I don't expect to convince you that language-specific package managers are somehow better than any package manager meant for installing packages in a system-wide location, but that was never my goal in the first place.  They serve the purpose of dealing with dependency-related problems without the need for a dedicated machine containing the dependencies required, and they are even more useful in the case where you want to add some dependency on package P that was built against libbar v2.1, yet you can only use libbar v2.0 because of a bug in v2.1 or your software.  Whereas a system package manager would upgrade to v2.1, your software can continue to rely upon v2.0 while P relies on v2.1.  Eventually the bug will be fixed, but until that happens, you're stuck at v2.0.


----------



## Mjölnir (Jun 29, 2020)

memreflect said:


> [...] the problem you're describing is exactly why there's a reliance on minimum versions.  If v2.0 is specified as the minimum version (i.e. >=v2.0), then v2.0 will be used, even if v2.9 is installed. [...]


Usually the general assumption is that later versions fix bugs of previous versions, thus later versions are used.  In the example if _minimum version >=2.0_ and 2.9 is installed, 2.9 will be used, but 3.0 wouldn't be used because the ABI will likely have changed (else the new versions would be 2.10, 2.11,...).  You would have to state _minimum version=2.0_ to stick on that version.
Other than this minor flaw, I do enjoy your conversation because of the good quality of your arguments (on both sides).


----------



## Jose (Jun 29, 2020)

memreflect said:


> ...You're suggesting that because some other package P updated libbar from v2.0 to v2.1, suddenly foo will rely on libbar v2.1 instead of libbar v2.0.  But that's not how it works.  You can have multiple versions installed.  This may seem wasteful, but the problem you're describing is exactly why there's a reliance on minimum versions.  If v2.0 is specified as the minimum version (i.e. >=v2.0), then v2.0 will be used, even if v2.9 is installed...


No package manager I'm aware of works in this way*. If foo needs exactly bar 2.0 and baz requires 2.1 or later, both can't be installed on the same system. You'd have to link every binary differently to support what you describe. You can kind-of do this with platforms that use a loader, like Java, but then you need hundreds of lines of shell to launch even the simplest executable.



memreflect said:


> ...That's where these language-specific package managers differ from a more traditional system package manager: software shouldn't break just because your system suddenly upgraded a library, but if it does, you should be able to use the older version of the library until you figure out what happened because that's the one that worked.  Meanwhile, others are still building using the working code instead of multiple people wasting time hunting a bug down because of a system upgrade that broke the program.


All you're doing is kicking the integration can down the road, and probably making it someone else's problem. Given that integration is one of the messiest and least interesting parts of building software, throwing things over the wall like this is extremely antisocial.



memreflect said:


> And if that chosen system is the Debian, Fedora, and Arch Linux distributions as well as FreeBSD?  Just because I can create system-specific packages/ports for each of those systems, that doesn't change the fact that one or more dependencies I need to build that binary package may not be available as system packages or are even the wrong versions (e.g. maybe there's a bug in v2.1 and v2.1.1 hasn't been released yet).  With a language-specific package manager...


If you build something useful and package it for a reasonable platform, support will come. This is one of the great strengths of open source software. Others will help you if you're not too antisocial.

*Gentoo has some limited support for this with "slots". However, slots are hard to maintain, and creating new ones is frowned upon by the Gentoo devs. They're typically only used for things like Php 7.3 vs 7.4, the Python mess, etc. They do come in extremely handy for database upgrades, though. You can have both the old version of Postgresql and the new one installed side-by-side.


----------



## garry (Jun 30, 2020)

Sevendogsbsd said:


> ports-mgmt/synth hasn't had an update in a year but out isn't flagged as "no maintainer" or anything. I used it a few months ago and it worked fine....


Take a look at the source code for `synth` -- I've seldom seen such clean and beautiful code.  Even if developers do prefer to build with the `poudriere` script (and use jails instead of null mounts to allow building malicious/bad-actor ports) every programmer could benefit from learning a little bit from synth's Ada code.  It is really sweet.  And I have to add that I've seldom seen any package receive such good support as `synth` does here.  Mention any problem with `synth` and the author will be show up as if by magic to help...

(Synth is the most straight-forward replacement for portmaster: 

```
synth configure
synth upgrade-system
synth install <port>
```


----------



## Sevendogsbsd (Jun 30, 2020)

I have never had an issue with synth. The times I used it, it worked very well. My use case was ports on a desktop so perhaps not everyone’s situation.


----------



## Lamia (Jun 30, 2020)

Poudriere seamlessly integrates with Buildbot. That's a big plus.


----------



## SKull (Jun 30, 2020)

garry said:


> Take a look at the source code for `synth` -- I've seldom seen such clean and beautiful code.  Even if developers do prefer to build with the `poudriere` script (and use jails instead of null mounts to allow building malicious/bad-actor ports) every programmer could benefit from learning a little bit from synth's Ada code.  It is really sweet.  And I have to add that I've seldom seen any package receive such good support as `synth` does here.  Mention any problem with `synth` and the author will be show up as if by magic to help...
> 
> (Synth is the most straight-forward replacement for portmaster:
> 
> ...


Development here.
That ist exactly why I pressure my teammates to write clean code.
Because bugfixing and new featured are a piece of cake when the codebase is nice and clean.


----------



## wisdown (Sep 17, 2020)

For Years, I have used portmaster to compile all my programs on FreeBSD.

I think the point is:

*Do you need custom options?

If yes, then you will need compile the stuff*, and like handbook advices, do not mix the things (with some exceptions) for compiling stuff way do it for all.

*If not, then use pkg*.

On this exactly moment I am running an test Virtual Machine with poudriere for see if it will save some time for me.

But what's I mean for save some time?

Lets start from beginning...

Usually I use one Virtual Machine per service, so in a minimal network I will have:

1 DNS Server Forwarder
1 DNS Server Authoritative
1 NTP Server
1 Database Server
1 Web Server (inside running Apache on root, jailed node, jailed php, etc...)
1 Mail Server

So from this basic setup I have 6 Virtual Machines...

Working only on DNS servers the compiler time is not a real issue, bind compile fast because there no many dependencies...

But since it is 2 Servers, then I will spend the double of time for the same job (compiling one by one to save resources on server)...

What make me compile dns/bind916?
Usually enterprises looking for performance improvements only will have that with same hardware through tunings... dns/bind916 have this nice option:


```
LARGE_FILE
```

It's off by default, and the requirements to get an real performance boost is only use it when the system have at least 8 GB of RAM or more...

So, it's mean, you need know what you doing when enabling extra options or disabling something because security issues.

On this scenario, if `poudriere` can make an local customized repository to me instead use the default pkg from FreeBSD main server, will be a time save. (Read will compile only one time with my custom options, and the update will be deployed to both servers)

Maybe the real issues only occurs with more complex setups, like on the webserver...

Enabling graphic extra options on graphics/php74-gd will require you compile graphics/gd and others, and those will probably ends in the recursive problem (aka chicken egg) breaking the process.

Using `portmaster` on the webserver scenario, I have the workaround:

I can install the `pkg` version and compile one of the biggest dependencies...

Then I can remove one of `pkg` version downloaded before to fill the requirement, and now compile it from source...

And redo it till get as close of possible from entire compiled system...

For as close possible I mean there situations where only an outdate `pkg` will fix the issue...

From last time I have compiled graphics stuff, if I remember right something related to H264 or H265 was broken, the only way to get this dependency is using an outdate pkg version instead the ports versions...

How `poudriere` will work on previous scenario?

No idea, this is why I am testing now...

But the previous scenario using `portmaster` I was able to get the things up and running.

If `poudriere` can solve the chicken egg problem, so it's the awesome solution...
One big time only recompiling everything when need, and then all your servers can downloading it in the sequence locally.

If `poudriere` fails on my test, there no way instead back to `portmaster` and schedule the recompilations because updates...

How much time we spend doing the things per machine?
Have you tried compile devel/llvm80? Whats the time for do it in 6 machines one by one for example? (Remembering you can have more servers for example an dedicated redis server)

For last, I think there an extra reason to use compiled versions, like the option CPU extra stuff, on default packages that will be always off, compiling from source you will get extra boost from this option, one time your compiled version of the program will have extra functions to use features from your specic processor (this is why is off on generic pkg - read the program does not know what's the processor will enduser have)...

By the way, like many tests around the internet already have reported, llvm versions mostly of time run slowly if compared to gcc version, but this is not the discuss here (read any extra boost to better perfomance is an plus).


----------



## chrcol (Oct 9, 2020)

Poudriere to me would make sense if you want one build machine to provide packages for many machines which are duplicates of each other, so by duplicates I mean same packages installed, same OS version, same port options and same cputype.

The argument you dont need to clutter your system with build dependencies doesnt stick with me, because they will still be on the system, they will just be in the jail directory instead of the base directories, and you actually have more clutter because there will be copies of the binary ports in the jail on top of on the base system as well.

The only other potential advantage is you avoid the potential problem of a half updated system breaking the live configuration, this one I recognise, but I am against the idea of making jails or building on a different machine to my system, so I will stick with what I use now which is primarily portupgrade.  I will probably keep on using it until it stops working.


----------



## leebrown66 (Oct 9, 2020)

chrcol said:


> Poudriere to me would make sense if you want one build machine to provide packages for many machines which are duplicates of each other, so by duplicates I mean same packages installed, same OS version, same port options and same cputype.


I disagree.

I used to use poudriere to build repos for both 386 and amd64, but thankfully that old hardware is no longer alive.

I now use poudriere to build 3 different repos, each with different options, one optimised for my workstation (cpu tweaks, no docs, no examples), one for general puropose servers that have varying CPU's (no cpu tweaks, specific options set, docs, examples) and a third I use for experimental purposes.  That's part of the point with poudriere, different repos can have different option sets and the repo is consistent within itself.  You do get into trouble if trying to mix different repos with different options though.


----------



## Oclair (Oct 10, 2020)

I use Portmaster primarily, and very limited use of pkg.  People who do not use Portmaster should refrain from giving advice regarding the use of Portmaster.  How you update your ports is none of my business, portmaster works great for me, I am grateful dougb contributed it to the community!  I am also grateful there are soo many useful tools to manage ports which facilitate automating updating several installs.  I've used FreeBSD back in 2000 installing 4.2 from a windriver cd.  I just appreciate all the massive talent and expertise who have all helped make our OS and community maintain such a great OS.


----------



## olli@ (Oct 10, 2020)

I’m also one of those who prefer to _not_ use poudriere (nor synth). It has _no_ advantages for my workflow and the way I use the ports.

If poudriere works fine for you, that’s great. But don’t force it on others.


----------



## Francisco Reyes (Oct 11, 2020)

Oclair said:


> People who do not use Portmaster should refrain from giving advice regarding the use of Portmaster.



People who use these forums should actually try to be helpful and polite. Take the other thread for example.



eldaemon said:


> Should I be using a different build tool to make this more bulletproof?



There was more noise from the portmaster purists that could not be bothered to actually  properly read the post  try to understand the actual use case or needs. It is possible, and we may never know due to all the noise, that some other tool may be what the user actually needed. Ultimately the goal should be "how do we help people seeking help to achieve what they are looking to do in FreeBSD".  They may be using a tool and workflow that won't achieve what they need.



Oclair said:


> I am also grateful there are soo many useful tools to manage ports which facilitate automating updating several installs.



And yet  mention of other tools besides portmaster when the requester actually asked if there were other tools seemed to be a problem.



olli@ said:


> I do not like the black&white attitude “poudriere is the only correct way to build ports,



Last I checked we are here to try and help each other in a constructive way. What someone "likes" is up to them to post on *their posts. *To hijack someone else's question with noise about what someone likes is far less helpful or constructive than offering an option to the original poster based on his own question of what other options are available.



olli@ said:


> I’m also one of those who prefer to _not_ use poudriere (nor synth). It has _no_ advantages for my workflow and the way I use the ports.



And that is how it should be. People should use what works for them for their needs. However, they should not assume that their preferred tool is what others need. So when a user asks "are there other options" the tool you use will not always be the right answer, but you can present it as information that then the user can look into and consider.


----------



## olli@ (Oct 11, 2020)

Francisco Reyes said:


> Last I checked we are here to try and help each other in a constructive way. What someone "likes" is up to them to post on *their posts. *To hijack someone else's question with noise about what someone likes is far less helpful or constructive than offering an option to the original poster based on his own question of what other options are available.[


Please stop creating confusion by quoting partial sentences from different threads. My remark that you quoted (partially) was in response to someone who hijacked that other thread in a non-constructive way, and my response was meant to actually _support_ the original poster. You are misrepresenting the facts.

On the other hand, let me remind you of the topic of this thread: “Do not use ports-mgmt/portmaster and other tools who build in the main system”. This is a quite confrontational thesis. It’s no surprise that it provokes a debate like this.


----------



## Mjölnir (Oct 11, 2020)

olli@ said:


> On the other hand, let me remind you of the topic of this thread: “Do not use ports-mgmt/portmaster and other tools who build in the main system”. This is a quite confrontational thesis. It’s no surprise that it provokes a debate like this.


Yes, that was in fact my intention.  My observation is that _portmaster_ causes more bad than good, espc. for newbies.  That's why I wanted a thread where the interested newbie can read about the pros & cons and take an informed decision.


----------



## a6h (Oct 11, 2020)

olli@ is correct. I believe "hijacker" is a reference to me. That conversation was initiated in the other thread:
Thread portmaster-best-practices.77281.

I don't want to cross-post on this thread, therefore I'll continue it on the original one:
Thread portmaster-best-practices.77281/#post-480925


----------



## richardtoohey2 (Oct 11, 2020)

I'm getting dizzy. Think we are almost all agreeing:

1) For new users and maybe _most_ users, pkg is the way to go.  Fast, convenient, less risk of getting lost in dependancy hell or having issues.
2) Sometimes, though, ports are the only way to go.  But there may be dragons here.
3) Mixing ports and packages is not generally speaking a good idea, but some people have success doing so.
4) We have differing views on how best to use ports and what tools to use. poudriere seems to be the one currently recommended (is it safe to say that?) by the FreeBSD developers.  Others prefer (for all its flaws) portmaster and there are yet more options - portupgrade, synth, roll-your-own scripts, etc.
5) Every post about "how do I use portmaster to do this" seems to get replies of (a) "don't do that, it drinks kitten blood", (b) "don't, use pkg" (c) "don't, use synth, it's lovely", (d) "don't, use poudriere, it's the one true way" and the OP's original question gets lost in the expertise/opinions of the port-tool-x fans.  Who then fire up the port-tool-y fans.  And round we go.


----------



## Francisco Reyes (Oct 12, 2020)

How about we work together and come up with pros and cons of tools and then put it all together in a post in the Howtos and moderated forum?
Ultimately, in my opinion, the best we can do is present facts and let people decide what works best for them.


----------



## mickey (Oct 12, 2020)

mjollnir said:


> Yes, that was in fact my intention.  My observation is that _portmaster_ causes more bad than good, espc. for newbies.  That's why I wanted a thread where the interested newbie can read about the pros & cons and take an informed decision.


I have been using portmaster on multiple machines on a daily basis for years now and I can say with confidence that it has done me a lot of good, especially considering the nightmares I was having using portupgrade before that. The use of ports over packages is probably something for more advanced users, but that is not something caused by whatever tool you use to update your system from ports, this is by nature of ports. Anyone who decides to use ports instead of packages probably has a reason to do so, otherwise he/she would not even consider it worth the additional time and effort to do so.

But that aside.... don't you think a thread with such particular title leaves little room for any _pros_ or _informed decisions_?


----------



## Mjölnir (Oct 13, 2020)

mickey said:


> I have been using portmaster on multiple machines on a daily basis for years now and I can say with confidence that it has done me a lot of good, especially considering the nightmares I was having using portupgrade before that. The use of ports over packages is probably something for more advanced users, but that is not something caused by whatever tool you use to update your system from ports, this is by nature of ports. Anyone who decides to use ports instead of packages probably has a reason to do so, otherwise he/she would not even consider it worth the additional time and effort to do so.


To build (a non-trivial large number of) ports on the host requires that the admin is able to cope with issues that _will_ eventually arise, i.e. mainly the so-called _dependency hell:_


mickey said:


> [...]  I have been using the ports tree (and portmaster/portupgrade) almost exclusively from the very beginnings and never had any major headaches I wasn't able to somehow solve by myself. When stuff breaks it is often due to software X version B using a header file or library that software X version A installed on the system, instead of using the correct version that comes bundled with the source. Of course a clean room environment would circumvent such problems, but that is hardly a solution to the problem in itself.


While I do not deny that the scripts & tools to assist building ports on the host work _sometimes_ & save a few keystrokes, your post is one of a few pro-portmaster(8) posts that honestly tell about the fact that this method has inherent issues.  Most others do not come up with arguments other than _"Don't tell me how I have to administer my system"_ & _"portmaster served me well for decades",_ while at the same time embezzling the issues they had to solve manually.  Please note that I'm not talking about building, say, a dozen of ports, but e.g. a GUI and/or the version mismatch (_latest_ ports vs. _quarterly_ packages) you'll run into when building a few selected ports yourself on a system otherwise installed with packages from the official repository.


mickey said:


> But that aside.... don't you think a thread with such particular title leaves little room for any _pros_ or _informed decisions_?


No, because there are obviously very few reasonable pro-arguments to build (a non-trivial large number of) ports on the host unjailed.


----------



## olli@ (Oct 13, 2020)

mjollnir said:


> To build (a non-trivial large number of) ports on the host requires that the admin is able to cope with issues that _will_ eventually arise, i.e. mainly the so-called _dependency hell:_
> 
> While I do not deny that the scripts & tools to assist building ports on the host work _sometimes_


The word “sometimes” is a misrepresentation. Do you have any numbers to back that claim? I dare to say it works fine _most of the time_. I can back that with numbers from my workstation (see below), which is not representative, but not an uncommon kind of setup. Problems caused by ports accidentally using files installed by unrelated ports are a rare exception, not the rule.

Regarding your mention of “a non-trivial number” of ports: I have about 800 ports on my workstation; among them there are large applications with many dependencies (e.g. ffmpeg, mplayer, TeX, Gimp, Scribus, InkScape). I assume this counts as non-trivial. Of those 800 ports there are exactly 2 that have caused dependency problems in the past (one of them is InkScape). This is easy to workaround once you’re aware of it, for example by simply adjusting the order of ports while (re-)building. (My own update script supports the option to build in a jail, only for those few ports that need it. It would probably be a good idea if the portmaster author added such a feature, too. It’s actually not very difficult at all.)


----------



## Mjölnir (Oct 13, 2020)

olli@ said:


> The word “sometimes” is a misrepresentation. Do you have any numbers to back that claim? I dare to say it works fine most of the time.


When I was new to FreeBSD, I tried to build a KDE desktop plus some utilities (avg. workstation use-case) and failed, and I ran into the version mismatch between _latest_ ports tree vs. _quarterly_ packages when I tried to build selected ports with options different from the defaults.  Both issues were not clearly covered by the handbook.


olli@ said:


> [...]  This is easy to workaround once you’re aware of it, for example by simply adjusting the order of ports while (re-)building.


The correct build order should be figured out by make(1).  While I've been critizising _recursive make_ often, I realize that the ports(7) tree as a collection of 3rd party source trees is inherently restive to declare all inter-port dependencies correctly.  Even if it's theoretically possible, there will always be human flaws since naturally the ports tree is a moving target.


olli@ said:


> (My own update script supports the option to build in a jail, only for those few ports that need it. It would probably be a good idea if the portmaster author added such a feature, too. It’s actually not very difficult at all.)


File in a bug report and flag it _wishlist_ or _enhancement_ or such?  For shure an option to build chroot(8)'ed or jail(8)'ed would make perfect sense.


----------



## olli@ (Oct 13, 2020)

mjollnir said:


> When I was new to FreeBSD, I tried to build a KDE desktop plus some utilities (avg. workstation use-case) and failed, and I ran into the version mismatch between _latest_ ports tree vs. _quarterly_ packages when I tried to build selected ports with options different from the defaults.  Both issues were not clearly covered by the handbook.


I see. You are right, mixing latest ports with quarterly packages is a mistake that often leads to pain and suffering. I agree that the Handbook should contain a *big fat* warning not to do that. (Actually I thought it did, but apparently it doesn’t, unfortunately.)



> The correct build order should be figured out by make(1).


Yes, when building a single port (plus its dependencies), that works very well. Otherwise tools like poudriere would have the same problem, because it also relies on the Makefiles of the ports collection.

But when you build two unrelated ports that don’t depend on each other, there is no order of them stored in the Makefiles. In theory the order should not matter. But if one of these ports has a bug causing it to use the wrong include file, for example, then the order of building _does_ matter. In this case poudriere clearly wins because it builds the ports separate from each other, while portmaster doesn’t do that, because it builds everything on the host. Thankfully this happens rather rarely, but still it is annoying, and you have to be aware of it, so you know how to work around it.



> File in a bug report and flag it _wishlist_ or _enhancement_ or such?  For shure an option to build chroot(8)'ed or jail(8)'ed would make perfect sense.


It’s a good idea, but I’m not using portmaster myself, so I have little incentive to spend time with that. (I have to admit this is a selfish attitude, but the day has only 24 hours, so I have to set priorities.)


----------



## Zvoni (Oct 13, 2020)

chipping in with my unprofessional opinion  
Let's be honest with each other:
How many of the "new" FreeBSD-users, who come here to ask for help (and to stay in context: about this particular subject), are your "off-the-shelf"-user, who doesn't have an IT-background and/or longtime-experience with computers?
As in: My Grandma bought a new Laptop to finally get online, and now want's to use FreeBSD?
For that matter: How many of the existing Users here do have such a background?

Point for Honesty: Pretty much none (or a very small percentage).
Meaning: Users asking such questions, do have experience with computers and/or different operating systems.
Meaning: Your off-the-shelf-Winbloze-User is not your regular here, because he wants to try FreeBSD for the heck of it.

Speaking for myself: I came here because i did have my longtime-experience with Windows and Linux, and i had an opportunity to check out a "new"/"different" OS for the Servers of my Skydiving-club.

*In that vein, i would expect experienced FreeBSD-users to advise me, "don't do that, because the script has bugs, that might launch a nuclear rocket in north-korea, if not handled properly".*

Yes, i started out with pkg/binaries from quaterly, until i found out, that to have a feature i needed, i had to compile myself.
First contact with portstree, make config, make, make install
until i run afoul (as it's usual with no experience) with something, and had to research, until i found "make deinstall install".
Additionally, it annoyed the hell out of me, how to update/upgrade the ports.
Nevermind the amount of ports you would have to handle by hand.
So i found portsmaster (since it's mentioned in the handbook).
Yeah, went well, until i stumbled across the pkg/binary vs. "built-from-ports"-trap
Now, some months ago, i found that HowTo-Thread for poudriere (Note: There is no such HowTo for synth, or at least i couldn't find it).

Bottom-Line: I use poudriere, and never looked back (only forward, or until a "better" way presents itself)

As i said: I expect experienced users to tell me the "do's and don'ts", and if a program/script/whatever is flawed (even if it's just flawed in their opinion), i'd expect a warning and/or alternative.
Just because there are users, who don't have problems with it (and i'd bet, those users are at the upper end of the "expert"-scale), doesn't mean the others warning of it are wrong.


----------



## mickey (Oct 13, 2020)

mjollnir said:


> To build (a non-trivial large number of) ports on the host requires that the admin is able to cope with issues that _will_ eventually arise, i.e. mainly the so-called _dependency hell:_


My workstation has 828 packages installed as of now, all built and updated (yes, using portmaster) exclusively from latest ports, usually on a daily basis. Yet I would not want to go as far as to call it a _dependeny hell_.



mjollnir said:


> While I do not deny that the scripts & tools to assist building ports on the host work _sometimes_ & save a few keystrokes, your post is one of a few pro-portmaster(8) posts that honestly tell about the fact that this method has inherent issues.  Most others do not come up with arguments other than _"Don't tell me how I have to administer my system"_ & _"portmaster served me well for decades",_ while at the same time embezzling the issues they had to solve manually.


Updating my systems using portmaster has worked for me more than just sometimes and also saved more than just a few keystrokes. Back in the day when I started using 386BSD 0.1 I believe there wasn't even anything remotely like ports. The usual procedure to install some piece of software was to download the source .tar.Z file, extract it, read and follow the instructions in some README file (if there was one) and hope for the best. Failing that either fix the source manually or complain to the authors. So yes, having ports was kind of a quantum leap. Are there problems? Most likely, but none of the problems I encountered recently had anything to do with portmaster in particular, and fell into one of two categories:

Ports that received an update that turns out to not be quite ready yet. Usually fixed by updating the ports tree again some time later and retrying. A more recent example I can think of was an update to libuv that broke name resolution in my entire network. Was fixed by another update couple hours later.
Ports with a broken build system. One such port that immediately comes to my mind is www/qt5-webengine, which I have seen shoot itself in the foot more than just once, probably because it uses include files installed on the host by a previous version of itself, when in fact it should be using the include files from the source/working directory. Fact is, building qt5-webengine often fails if qt5-webengine is already installed, and again this is not related to portmaster as it fails in exactly the same way when you try building it manually.



mjollnir said:


> Please note that I'm not talking about building, say, a dozen of ports, but e.g. a GUI and/or the version mismatch (_latest_ ports vs. _quarterly_ packages) you'll run into when building a few selected ports yourself on a system otherwise installed with packages from the official repository.


That mixing quarterly and latest is a bad idea is nothing new, but a proven fact, whether you build from ports or use packages. Again that has no relation to portmaster whatsoever. Maybe the handbook and/or other documentation could put more emphasis on this matter if it bothers you that much. Personally I never ran into this issue. As I said, when I started there was hardly any other option _but_ build it yourself, and quite frankly, I wouldn't want it any other way.


----------



## chrbr (Oct 13, 2020)

olli@ said:


> (My own update script supports the option to build in a jail, only for those few ports that need it. It would probably be a good idea if the portmaster author added such a feature, too. It’s actually not very difficult at all.)


Some weeks ago there have been issues with the speed on pkg and/or long unfixed vulnerabilities or so. I do not remember the details. I have looked for a work around. At that time I have tried to create a jail and letting portmaster make packages. Fortunately portmaster has the option to use packages for the build dependencies which saves time. Then I could install the packages from the new source if everything went well. After a few days the pkg servers have been back to normal and I did not follow up the method. I switched back to pkg because I can live with the default settings. But with building in a jail the host is not spoiled and nothing can be overwritten or deleted unintentionally (I think so).


----------



## Oclair (Oct 16, 2020)

Francisco Reyes said:


> the best we can do is present facts



that's factually an opinion


----------



## getopt (Oct 16, 2020)

mjollnir said:


> this is not a question of agreement, but right and wrong


There is no binary answer to this. Trying to force "Is this right?" patterns is prompting for reactance. You will be shown that there is a "depends on".

Polls are attracting attention. In return you get opinions.

A missing item among others in the poll is: "I do not care."


----------



## msplsh (Oct 16, 2020)

olli@ said:


> mixing latest ports with quarterly packages is a mistake that often leads to pain and suffering. I agree that the Handbook should contain a *big fat* warning not to do that.



It would be nice to know _how_ to not do this.  Packages seem to default to quarterly (Which sometimes backports security updates?) and ports using portsnap (as the handbook suggests) defaults to... something very recent?

The hacked up way I've found to keep these in sync is to go to https://svnweb.freebsd.org/ports/branches/ and then figure out what the current quarterly SVN directory revision is, and then feed that into `svnup ports -r` instead of using portsnap.


----------



## Jose (Oct 16, 2020)

getopt said:


> There is no binary answer to this. Trying to force "Is this right?" patterns is prompting for reactance. You will be shown that there is a "depends on".


Note the winky emoji in his post.



getopt said:


> A missing item among others in the poll is: "I do not care."


Why read the thread and post in it if you don't care?



msplsh said:


> It would be nice to know _how_ to not do this.  Packages seem to default to quarterly (Which sometimes backports security updates?) and ports using portsnap (as the handbook suggests) defaults to... something very recent?


I found Vigole's recent post on the subject illuminating. Here's the mailing list message announcing the quarterly branch:


			[FreeBSD-Ports-Announce] [ANNOUNCE] ports 2014Q2 branched


----------



## olli@ (Oct 17, 2020)

msplsh said:


> It would be nice to know _how_ to not do this.  Packages seem to default to quarterly (Which sometimes backports security updates?) and ports using portsnap (as the handbook suggests) defaults to... something very recent?


Portsnap (which is apparently going to be declared obsolete when the switch from Subversion to Git happens) does not support the notion of branches. It always gives you “head”, i.e. the most current state of the ports repository (*). This is the same as when using net/svnup with the “ports” keyword which checks out `ports/head`.

(*) Well, almost. The portsnap data is generated at certain intervals, so it may lag slightly behind the Subversion repository. This doesn’t matter for most practical purposes, though.


----------



## msplsh (Oct 17, 2020)

olli@ said:


> Portsnap (which is apparently going to be declared obsolete when the switch from Subversion to Git happens) does not support the notion of branches.



I put a little more effort into this and found -b

So, maybe the handbook should instead of "here's one way, here's another" instead be like "here's how to get latest (portsnap), here's how to get quarterly (svnup)"

`pkg install svnup
cd /usr/ports
svnup ports -b 2020Q3 -h svn.freebsd.org`

Does that look right?


----------



## olli@ (Oct 17, 2020)

msplsh said:


> `pkg install svnup
> cd /usr/ports
> svnup ports -b 2020Q3 -h svn.freebsd.org`
> 
> Does that look right?


Almost. I think you need `-b ports/branches/220Q4`. You can look up the structure at svnweb.freebsd.org.


----------



## PacketMan (Oct 17, 2020)

garry said:


> Take a look at the source code for `synth` -- I've seldom seen such clean and beautiful code.  Even if developers do prefer to build with the `poudriere` script (and use jails instead of null mounts to allow building malicious/bad-actor ports) every programmer could benefit from learning a little bit from synth's Ada code.  It is really sweet.  And I have to add that I've seldom seen any package receive such good support as `synth` does here.  Mention any problem with `synth` and the author will be show up as if by magic to help...
> 
> (Synth is the most straight-forward replacement for portmaster:
> 
> ...



ports-mgmt/synth is awesome, its all I use. And its creator John Marino is a master of code it seems.  If it were to become unmaintained I would make figure out how to use ports-mgmt/poudriere plain and simple.


----------



## chrcol (Oct 19, 2020)

olli@  I have already removed portsnap of my base system and have a cron'd script to update my quartery branch nightly, it also auto detects when the quarter changes.

I hope a new tool is created when they move to git, as git is so less user friendly and slower vs svn.


----------



## Oclair (Oct 23, 2020)

I wonder if actually all this FUD pointed towards portmaster and portsnap because we're going into a Microsoft owned GIT and away from SVN?


----------



## msplsh (Oct 23, 2020)

Oh geez, please delete this.  Microsoft does not own git and the reason why these have problems is presumably neglect and disinterest.


----------



## Jose (Oct 23, 2020)

Oclair said:


> I wonder if actually all this FUD pointed towards portmaster and portsnap because we're going into a Microsoft owned GIT and away from SVN?


That tinfoil hat doesn't go with the rest of your outfit.


----------



## Jose (Oct 24, 2020)

From the BSD link roundup post from today:


> So yes it is true that we do not yet have FLAVORs nor subpackages but there is some ongoing work to provides those features in the ports tree (actually in final review right now). That work is pretty hard as it breaks the design of some tools that are heavily used by our users like portmaster and portupgrade and which are barely maintained; meaning we need to find an upgrade path which will not break those. We try to always have an upgrade path for users and try to avoid hard breakage. Keep in mind that we do not release a package set, but the ports tree is a rolling release not tied to a FreeBSD release.


----------



## Oclair (Oct 25, 2020)

I remember when FreeBSD 5 was released and when I upgraded it broke all of my 4.11 boxes (yes boxes)


----------



## Oclair (Oct 25, 2020)

msplsh said:


> Oh geez, please delete this.  Microsoft does not own git and the reason why these have problems is presumably neglect and disinterest.


Maybe you might mean assume? as "assumption" suggests there is little evidence supporting your guess. Think carefully before using them interchangeably or you may lose some meaning.


----------



## xtaz (Oct 27, 2020)

Oclair said:


> I remember when FreeBSD 5 was released and when I upgraded it broke all of my 4.11 boxes (yes boxes)


To be fair, back then the development/release model was massively different to how it is today. 5 was in development for several years and had huge very breaking changes to 4. 5.0-RELEASE itself was full of bugs as well. It's the reason DragonFly BSD forked from FreeBSD at that point as they didn't like the direction the 5 code had taken over the 4 code.

Since then the changes between major versions have been much more incremental as there's a new major release every year or so.


----------



## Lamia (Oct 27, 2020)

msplsh said:


> I put a little more effort into this and found -b
> 
> So, maybe the handbook should instead of "here's one way, here's another" instead be like "here's how to get latest (portsnap), here's how to get quarterly (svnup)"
> 
> ...


Yes, it does. We are sometimes unlucky with building all ports needed by our boxes. You might notice it as you switch among 2020Q2, 2020Q3 and even 2020Q4. 2020Q1 works well most of the time. The Q4 showed non-existent ports for all x11-related ports just last week.


----------



## Snurg (Mar 6, 2021)

I have read the whole thread, but didn't find an answer yet for my problem.

SirDice teached me that I can use poudriere to build packages with my own patches, in a way that it keeps/merges these patches through updates.
As far as I understand, the disadvantage of using poudriere is that poudriere cannot just build on demand what you need.
You have either to build ALL gazillions of packages, or to list a fixed set of packages to build.

This is not nice if you want to build _only_ what you _actually_ need.

Could I use synth together with the trick @Mjölnir revealed in this post, to build and install what I need just when I need it, together with the patches?

Aim would be to integrate inofficial patches, hacks and things like stuff being discussed on phabricator, but not yet added in the <current> repo.


----------



## zirias@ (Mar 6, 2021)

Snurg said:


> As far as I understand, the disadvantage of using poudriere is that poudriere cannot just build on demand what you need.
> You have either to build ALL gazillions of packages, or to list a fixed set of packages to build.


Huh? `poudriere bulk` accepts a list of ports on the commandline, being able to pass this list from a file is just convenience. It will only build what you give it and what is necessary as a dependency. If dependencies were previously built in the required version, it will of course NOT rebuild them.


----------



## Mjölnir (Mar 6, 2021)

Snurg, I guess you mean that you want to be able to bypass re-building dependent ports?  That's bogus - think about it...  OTOH, I understand that if you just change an option of a port, you don't want to rebuild all dependents when you know they'll not be affected by that change.  But that's not reflected in the dependency rules in the Makefiles - and it would be very labourous to do so.  Maybe by introducing _tag files_ for each _port-option_ tuple; but that would explode the #rules exponentially...


----------



## Deleted member 30996 (Mar 6, 2021)

Debunked.


Mjölnir said:


> `portmaster -F x11/kde5` (recursively fetch all distfiles needed to build kde5)
> The result is that after more than one hour of uselessly processing the dependency graph, this self-called _master_ still fetches _nothing_.





Trihexagonal said:


> This will do the job, pull in a list of all dependencies it deems necessary for the build, then present them to you for authorization to proceed, or options should you not want to install everything it shows. I go with it unless it's something like IPV6 I've disabled or something I know to watch for to disable during the time it's presented to me.
> 
> `root@jigoku:/ # portmaster x11/kde5`



Don't add a flag to the command and blame the problem encountered by doing so on the program when it doesn't do what you thought it would. That's what Linux people do when they say FreeBSD no worky right for penguin and blame FreeBSD for their own Tuxix Shock Syndrome.


----------



## Mjölnir (Mar 6, 2021)

Trihexagonal said:


> Don't add a flag to the command and blame the problem encountered by doing so on the program when it doesn't do what you thought it would. [...]


Please stay factual.  From the portmaster(8) manpage:
	
	



```
[--force-config|-G] [-aftv] -F
     fetch distfiles only
```
That's _exactly_ what I wanted it to do & I communicated that _accurately_ in the post you referenced.  Please check & verify that this tool applies an O(n^2) (or worse) algorithm where others (i.e. any make(1)-like build tool) has _linear_ runtime: compare

`portmaster  -F x11/kde5` (will process the dep graph for veeeeery looooong time...) versus
`make -C /usr/ports/x11/kde5 fetch-recursive` (starts to download after some sec. or min.)
Conclusion: there's nothing to be debunked.


----------



## Deleted member 30996 (Mar 6, 2021)

What you based your whole theorem of program failure on in using:
`[cmd]portmaster -F x11/kde5[/cmd]`

Was refuted with years of test proven fact and a 100% success rate using the same command for everything built without the -F fag:
`[cmd]portmaster x11/kde5[/cmd]`

The only difference being I don't use KDE and for me it's:
`portmaster x11-wm/fluxbox`

Free your mind and fluxbox will follow.


----------



## zirias@ (Mar 6, 2021)

TBH, I don't understand the manpage here regarding the exact purpose of this mode, but it clearly indicates you can't give a specific port with `-F` (it is not one of the "common flags").

Still I think portmaster is a tool from the past anyways and will (slowly) vanish.


----------



## Mjölnir (Mar 6, 2021)

Trihexagonal said:


> What you based your whole theorem of program failure on in using:
> `portmaster -F x11/kde5`
> Was refuted with years of test proven fact and a 100% success rate using the same command for everything built without the -F fag:
> `portmaster x11/kde5`


Please thoroughly read what I wrote: `portmaster -F` is documented in it's manpage to _"fetch distfiles only"._  Thus what I wrote is not _refuted_, but simply results in so-called _portmaster_ pondering about which files to download for over an hour on a decent modern laptop... whereas the functionally equivalent make(1) command as given above, needs a small fraction of time to decide what to do & starts downloading quickly.


Trihexagonal said:


> Free your mind and fluxbox will follow.


I do like this signature.


----------



## Mjölnir (Mar 6, 2021)

Zirias said:


> TBH, I don't understand the manpage here regarding the exact purpose of this mode, but it clearly indicates you can't give a specific port with `-F` (it is not one of the "common flags").


Then let it run until it has finished calculating the list of files & starts to download => `-F` can handle a target (port to act on).


Zirias said:


> Still I think portmaster is a tool from the past anyways and will (slowly) vanish.


No, it is beeing rewritten -- & hopefully some of it's algorithms redesigned/replaced with appropriate ones, as well -- in Lua for a few years now...


----------



## zirias@ (Mar 6, 2021)

Mjölnir said:


> Then let it run until it has finished calculating the list of files & starts to download => `-F` can handle a target (port to act on).


Well IF this is true (I can't tell cause I don't use portmaster and don't want to install it) then the manpage is wrong. According to the manpage, it isn't possible, which of course leaves me wondering what this option is meant for…



Mjölnir said:


> No, it is beeing rewritten -- & hopefully some of it's algorithms redesigned/replaced with appropriate ones, as well -- in Lua for a few years now...


Maybe there's a usecase. Not for me, though – if you use ZFS anyways, poudriere is simpler AND more robust.


----------



## Deleted member 30996 (Mar 6, 2021)

Mjölnir said:


> Please thoroughly read what I wrote: `portmaster -F` is documented in it's manpage to _"fetch distfiles only"._  Thus what I wrote is not _refuted_, but simply results in so-called _portmaster_ pondering about which files to download for over an hour on a decent modern laptop... whereas the functionally equivalent make(1) command as given above, needs a small fraction of time to decide what to do & starts downloading quickly.



In the area of unnecessary fiddling and fumbling around in wasting time to get the job done it's probably exactly what you wanted in maintaining your level of inexperience in the use of it. A point I tried to talk around but not successful at getting across in my Circumlocution.

I like to avoid that as much as possible. Therefore, use what I know from experience will start the job and finish it in the most efficient manner possible with no wasting of time in my efforts.


----------



## ShelLuser (Mar 6, 2021)

_What a load of nonsense..._  I can't be bothered to read the whole thread but unfortunately I voted 'No' because of my opinion, and then I realized that this post is so poor that it doesn't even manage to clearly explain what I've been voting on. Am I saying "_No, this post is not right_" or is the OP thinking that I'm voting in agreement with whatever he preaches to be not right?

So to avoid any confusion: my "No" vote is directed against the main post itself, _that's_ the part which I think is not right here.

There's *nothing* wrong with ports-mgmt/portmaster and I dare say so because I've been using it for even longer than the main post exists. I started using it ever since I discovered it somewhere around FreeBSD 10 and I've been using it right up to this point. Even during the time when it became unmaintained for a while and it has never failed me.

And I've heard plenty of people share their frustrations with Portmaster over the years and all fair enough because they shared their opinion. But to start boasting how people should not be using it because it's bad is just ridiculous and something I pick up as utter slander, which I think it's really a poor display. Like I always say: only tools blame their tools.


_In fact..._ I dare argue * that it's actually a lot worse to rely on tools like Synth and/or Poudriere. Not because I think they're bad, nonsense, but because of what they do.. See, both environments obfuscate the actual building process. Meaning? If you know how the ports collection works, if you're familiar with things like /etc/make.conf and the several build targets as explained in ports(5) then all that experience will do you little good when working with either environment.

Because you're no longer working with the ports collection anymore, nah, you're now working with the _building environment_ which will be working with the ports collection for you. You don't have to know anymore how to use build targets, nah, you need to know how to use your build environment instead.

Eventually resulting in some users who will become so alienated from the actual process that they'll start to believe that whatever those building environments do is right and everything else to be wrong. How unfortunately however that "everything else" involves the _actual_ FreeBSD build process for the Ports collection, a process that has been in use long before those build environments even existed.

Which is actually one of the reasons I personally dislike both Synth and Poudriere and I don't think they really add much useful to the whole process. But that's only _my opinion_, you will never see me start a rant against their usage. Simply because while I may dislike both programs I still respect the projects themselves.

In fact.. the following argument applies to all three, so _including_ Portmaster: all build environments get the job done. So who am I to start whining about how using Poudriere or Synth could be bad?

TL;DR?

I think the OP is ridiculous, been using Portmaster for almost as long as I've been using FreeBSD itself and I _never_ had issues with it. From setting up servers to KDE powered workstations, things always worked out for me. IMO only tools blame their tools.

</vent>

* (edit): Not really, I definitely can't be bothered to waste more time on this thread, let along argue about a pointless topic as this. Just wanted to clear up my vote which ended up sharing my dislike for the thread.


----------



## zirias@ (Mar 6, 2021)

Well, I didn't bother to read this whole posting. I agree there's quite something wrong with this thread, mixing in unrelated stuff. I disagree there's "nothing wrong" with portmaster: Not building in a clean and defined environment always bears the (small) risk of strange effects. Other tools solve this problem. But that's probably all about it.

*edit:* Of course, you could approach it the other way around. Testing your ports ONLY in e.g. poudriere, you easily miss even the most obvious `CONFLICTS_BUILD` settings. But then, there's no sane way to identify ALL these conflicts. Building in a clean environment at least makes sure you never forget any dependencies. Given how easily poudriere can provide clean build environments with ZFS and jails, I see no reason not to use it, although of course, it should always be possible to build ports on your running system. Just saying it's NOT possible to ensure that for every weird edge case.


----------



## Snurg (Mar 6, 2021)

Sorry, I obviously failed to make understandable what I want.
Please apologize my difficulties in explaining clearly.

*What I want is to build on the respective local computer.
No external build server and all that.

I do not care which build system to be used.
I just need to be able to inject patches where I want. *(possibly using an unionfs for keeping the patches alive through updates, like Mjölnir suggested).

*Ideally I would like some tool that can work the way people are used from  pkg.*

a)
You want to try out some stuff.
You run `pkg install <somestuff>`.
It installs what is needed.
If version conflicts arise, it updates the conflicting stuff and all other packages that depend on this updated stuff (and only these).

b)
When you decide you do not want that particular <somestuff> anymore, you can run `pkg delete <somestuff>`.
bb)
After that, you can run `pkg autoremove` to remove dependencies that are not no longer needed. _This is a must to avoid garbage piling up over time._

*So, I need only three functionalities:
"portinstall" : works like  cd /usr/ports/<somestuff>; make fetch; make patch; make config_recursive +  make install *(case a)
*"portremove" : works like  pkg delete <somestuff> +  pkg autoremove* (case b+bb)
*"portupdate" : updates what is installed, like  pkg update +  pkg upgrade

But, I do not know a solution for bb) when using ports'  make system.
How can I keep the system clean of no longer needed dependencies?*

Having to read and actually understand all the documentation for `portmaster`, `poudriere` and `synth`, only to figure out whether and how I could achieve that task with that particular tool, would take a lot of time.
This is why I am asking.


----------



## zirias@ (Mar 6, 2021)

Snurg said:


> But, I do not know a solution for bb) when using ports' `make` system.
> How can I keep the system clean of no longer needed dependencies?


I don't have an immediate answer, but using poudriere WILL solve this, cause you will actually install with `pkg`, so `pkg autoremove` will work as expected on your own package repository.


----------



## Deleted member 30996 (Mar 6, 2021)

It was bad enough that the very first paragraph in the fine Tutorial you linked to was something someone with experience using it can work out from the Errors shown and information given what happened to stop the build as a normal matter of course, work though it manually then continue the build with portmaster:

"Building of ports as outlined in ports(7) is performed on the live system. Many build failures stem from this when an already installed port or leftover files from an old version causes conflicts with the to be built new version of the port."

I didn't bother with "the well-known Recursive Make Considered Harmful"  link provided when I looked at this the other day. But I followed the link to the article written in 1997 today and it directs to real-linux.org.uk.

"Please consult the well-known _Recursive Make Considered Harmful_ for reasons why such tools do not handle the build process theoretically sound -- they do not use make(1) correctly to compute the graph of dependencies."

Are you kidding me? You're sending FreeBSD people to get advise from an article posted on a Linux site on how to compile ports? The same link shows "the well-known Recursive Make Considered Harmful" article appears on microsoft.com and  gnu.org as well.

EOL


----------



## zirias@ (Mar 6, 2021)

Trihexagonal said:


> I didn't bother with "the well-known Recursive Make Considered Harmful" link provided when I looked at this the other day. But I followed the link to the article written in 1997 today and it directs to real-linux.org.uk.


I think this article does have its merits, and I still tend to structure my own projects to avoid recursive make. But that's what I meant with unrelated stuff. It has nothing to do with building ports. Recursive make is an issue inside a single project, and recursive make can work just fine, but it has its pitfalls. So much about that.


----------



## Snurg (Mar 6, 2021)

Zirias said:


> I don't have an immediate answer, but using poudriere WILL solve this, cause you will actually install with `pkg`, so `pkg autoremove` will work as expected on your own package repository.


Yes, but wouldn't mean using poudriere just moving the garbage pile?
I would prefer a solution without garbage pile.


----------



## zirias@ (Mar 6, 2021)

Snurg said:


> Yes, but wouldn't mean using poudriere just moving the garbage pile?
> I would prefer a solution without garbage pile.


For the packages on your system, `pkg` has you covered with subcommands `autoremove` and `clean`. poudriere itself also offers housekeeping, have a look at subcommands `distclean`, `pkgclean` and `logclean`.


----------



## Mjölnir (Mar 6, 2021)

Trihexagonal, cool down...   that paper is very popular, you'll find it on numerous sites all over the net and AFAIK the author has had no connection to Linux, M$, GNU or whatever, but IIRC he was working at a university?  You find it at Linux/GNU/M$/etc. because they all use make(1) or variants thereof to build software.  That's what make(1) is for: solve dependencies.  That's all about that so far.  Use your intellect, man, instead of jumping on Linux/M$/GNU like a bull on a red flag.  Read the paper so you can judge yourself...


----------



## zirias@ (Mar 6, 2021)

Mjölnir cool down as well please   
Yes, considerations about recursive make and the pitfalls (there be dragons!) are not Linux-specific. But they're not related to the ports tree either, or to a specific tool for building ports. The "naked" ports tree of course uses recursive make, cause there is simply no other option: It's a – let's call it that – "meta buildsystem" meant to build many individual upstream software packages. There is just no way (using `make`) NOT to make that recursive.

*edit:* no objections about the paper in general, as I said before, I think it's indeed a good idea to structure your build system for your own project without using recursion. Immediate benefit is dependencies DO work reliably without special considerations. Recursive approaches always need extra work for that.


----------



## Mjölnir (Mar 6, 2021)

Trihexagonal, can you confirm that I linked a DuckDuckGo query? Yes or No? _"Please press here to change the weather"_. Do you think I moved up M$ & Linux & GNU in thre search results intentionally to cheat you?


----------



## Snurg (Mar 6, 2021)

Zirias said:


> For the packages on your system, `pkg` has you covered with subcommands `autoremove` and `clean`. poudriere itself also offers housekeeping, have a look at subcommands `distclean`, `pkgclean` and `logclean`.


Great!
Does there already exist any ready-to-use script package which takes care of calling poudriere first, and when it finished, calls pkg?
(This script package would then have to keep track what "master" packages poudriere actually needs to make, by editing the package list file which is given to poudriere using the `-f` flag)

What I want is sort of seamless integration of poudriere.
So the user (and the scripts he uses) can just use `pkg` as he is used to, without having to deal with poudriere, after he got poudriere set up?
(I guess this script would have to be put into the path before pkg, so it gets called in place of pkg, and forward/pass to actual
/usr/sbin/pkg after it did the respective poudriere actions)


----------



## zirias@ (Mar 6, 2021)

Sorry Snurg never looked into that direction cause I have a "builder" machine building packages for a whole park of other machines  But seriously, something like this should be pretty simple to script yourself, although triggering a build from issuing a `pkg` command might not exactly follow the principle of least astonishment


----------



## Jose (Mar 6, 2021)

Snurg said:


> Great!
> Does there already exist any ready-to-use script package which takes care of calling poudriere first, and when it finished, calls pkg?


I don't think so. Here's what I do at a high level. I can get into details if you like.

* I keep a branch off the quarterly ports branch in Git that I use to build packages from.
* Single beefy build machine build two separate port sets, one for itself and one for a significantly less beefy server.
* I rsync the ports to the server. I have my private repository on both machines set to a higher priority than the Freebsd quarterly repo. This allows me to install a mix of locally-built and standard packages.

You can build a single port using poudriere-testport(8) but this won't package it, IIRC. The poudriere-bulk(8) command should not clean any ports if you don't give it the `-c` flag.

Edit: Thanks for the correction Mefizto!


----------



## Mjölnir (Mar 6, 2021)

There is poudriere-status(1) & you can RTSL.  I'm sure you can hook in when a port & all it's dependencies are built.  If you don't find the right script/routine to hook into, you can write the author a kind e-mail after streamlining your Q to the bare minimum.  Another method: you can poll for a package to appear in the repository & check it's checksum (don't try to install until the pkg is fully written...). We have filemon(4) (manpage incomplete: it needs to be loaded from loader.conf(5), not via `kld_list`), so you can detect that the contents of some directory has changed?  This is all brainstorming, but IMHO it shows your issue is solvable with the existing toolboxes.


----------



## Deleted member 30996 (Mar 6, 2021)

Mjölnir said:


> Trihexagonal, can you confirm that I linked a DuckDuckGo query? Yes or No?


Are you questioning my Honesty or ability to click on a link and copy and paste a URL?







						Recursive Make Considered Harmful at DuckDuckGo
					






					html.duckduckgo.com
				







Mjölnir said:


> _"Please press here to change the weather"_. Do you think I moved up M$ & Linux & GNU in thre search results intentionally to cheat you?



Cheat me out of what? Are you watching the Weather Channel and multitasking at your age? 

My site isn't even online anymore and my name appeared on 3 of the 5 top ranked listings in a Google search for FreeBSD Desktop Tutorial. It still ranks the #1 spot posted here and the article about it in freebsdnews.com  is #9.

Microsoft, Linux and GNU aren't even in competition with me for ranking.


----------



## Snurg (Mar 6, 2021)

Mjölnir said:


> RTSL.  I'm sure you can hook in when a port & all it's dependencies are built.


Yes, there is the question how to find out when poudriere has finished, and whether that was successful.
I guess that would be difficult and awkward but manageable. But I would like to avoid such kind of hackery.

But, reading the synth intro thoroughly, I get the impression that using *synth might have some advantages for desktop users*:

a) there seems no need to figure out how to check when it is safe to run `pkg`,
b) synth can combine package build and install by just `synth install <somestuff>`
c) when installing stuff via a web interface (postinstaller), the user could watch the build process via Synth's web interface in an iframe and actually see what is happening. This would be MUCH better than any kind of "progress bar".
d) synth is much faster than poudriere
e) and would be usable on Dragonfly BSD, too.

But the best is that synth seems to be designed to make possible to *only build the user-modified packages, and to pass through/use the normal FreeBSD repos for all other packages*.
This would be an extremely lean solution - storage space and compilation overhead reduced to an absolute minimum, just what actually needed.

Thank you very much for the discussion on the various approaches and their pros and cons.
I get more and more the feeling that for me (as a complete retard when it comes to system administration), it might be best to try Synth first, and then see whether I need poudriere at all.


----------



## Deleted member 66267 (Mar 7, 2021)

Snurg said:


> Yes, there is the question how to find out when poudriere has finished, and whether that was successful.
> I guess that would be difficult and awkward but manageable. But I would like to avoid such kind of hackery.
> 
> But, reading the synth intro thoroughly, I get the impression that using *synth might have some advantages for desktop users*:
> ...


I want to make an amendment to your very useful comment. DragonflyBSD still uses Poudriere to build DPorts, though. The information below is controversial, as I can't recall the source where I read it. Don't take it for granted!

The DragonflyBSD people seemed to want Synth but don't want the Ada requirements, so they seemed to rewrite Synth in plain C. I don't know the current status of this effort, or it was even started at all.


----------



## Deleted member 30996 (Mar 7, 2021)

Snurg, having read minimally on the subject of synth I'll address what you see as its possible advantages from the vantage of a ports user.

a) there seems no need to figure out how to check when it is safe to run `pkg`
It's always safe to compile ports and I don't regularly update all my programs as maintenance, only when the need arises from a vulnerability or requirement for another port.

b) synth can combine package build and install by just `synth install <somestuff>`
That's the job of Portmaster and we do it from source.

c) when installing stuff via a web interface (postinstaller), the user could watch the build process via Synth's web interface in an iframe and actually see what is happening. This would be MUCH better than any kind of "progress bar".
So is the login terminal, at a glace from my recliner to see if text is still scrolling or stopped, and if that means done or I have to step in abd fix things before starting the build with the same command and it will pick up from there to finish the job.

d) synth is much faster than poudriere
You have me beat there beyond a doubt. It takes a long time to compile some ports. Nap time.

e) is a moot point.

I would say you're a long way from "retard" status when it comes to FreeBSD Admin, Snurg.


----------



## Mjölnir (Mar 8, 2021)

TWIMC after a PM discussion with Snurg he agreed to open it's own thread on this topic.


----------



## zirias@ (Mar 8, 2021)

Snurg said:


> a) there seems no need to figure out how to check when it is safe to run `pkg`,
> b) synth can combine package build and install by just `synth install <somestuff>`
> c) when installing stuff via a web interface (postinstaller), the user could watch the build process via Synth's web interface in an iframe and actually see what is happening. This would be MUCH better than any kind of "progress bar".
> d) synth is much faster than poudriere
> e) and would be usable on Dragonfly BSD, too.


Uhm, I don't see anything *really* convincing here.

a) there's no need to ever figure that out. poudriere doesn't touch the old state of the repository until it is completely finished.
b) So, `poudriere bulk [...] && pkg install [...]` will achieve this as well.
c) Not sure what the idea behind the iframe is, but poudriere has a web interface as well (you only have to host it, e.g. with nginx).
d) Do you have figures? Nothing can magically speed up the compilation itself, so differences could only be in preparation/cleanup steps, globally and per package. poudriere can be flexibly configured to use your available CPU cores as good as possible.
e) as would poudriere ... uhm ...
Doesn't mean you can't just like synth better for some personal preference of course


----------



## Snurg (Mar 8, 2021)

Zirias, I am not sure you understand the problem.

Imagine a desktop with only a few packages locally modified.

The user wants to save time and storage.
So he wants to use the normal FreeBSD packages.
Technically it is only necessary to build those packages that are different, e.g. patched or non-default build options.

*So the user needs a build manager which only builds the packages that actually need to be built, and builds them just-in-time on-demand, and fetches all other packages directly from the FreeBSD repo.*

What the user (who has no personal preference yet) has read and been told, makes the impression that this is difficult-to-impossible to achieve with poudriere, but very easy with synth.
Is the users' impression correct or not?


----------



## xtaz (Mar 8, 2021)

Poudriere can do the prefetching of packages as of November. https://github.com/freebsd/poudriere/commit/b960cd66e001e1f15c87f089b1d99927705c6406 though not in the stable version yet.


----------



## zirias@ (Mar 8, 2021)

Well, if THAT's what you mean with faster, then yes. But this is a bit misleading cause this is about fetching packages built somewhere else. Still, poudriere only builds what is necessary (and reuses packages once they are built). Yes, if options is all you want to change, this can of course speed things up.


----------



## olli@ (Mar 9, 2021)

also-ran said:


> I want to make an amendment to your very useful comment. DragonflyBSD still uses Poudriere to build DPorts, though. The information below is controversial, as I can't recall the source where I read it. Don't take it for granted!
> 
> The DragonflyBSD people seemed to want Synth but don't want the Ada requirements, so they seemed to rewrite Synth in plain C. I don't know the current status of this effort, or it was even started at all.


They did. It’s in the DragonFly base system since 5.7, see the dsynth(1) manual page. Apparently they have made an effort to make the software as portable as possible, so it should work on FreeBSD, too, but I haven’t tried this.
 
Personally I like Synth much better than Poudriere, for various reasons. But that might be purely a matter of taste. YMMV. If you like Poudriere, then use Poudriere. If you like Synth, then use Synth.


----------



## jb_fvwm2 (Mar 9, 2021)

Is there a way to have synth not build already-installed build or run dependencies?


----------



## zirias@ (Mar 23, 2021)

As there's continued misconceptions around that topic, here's another explanation:

First, "recursive make": `make` is a tool using commands from a "recipe" to process input files to output files in a process that can cascade. So it might produce file_b from file_a, which is needed to produce file_c later. To do that automatically, the input to `make` is a set of dependency rules (here, file_c "depends" on file_b) plus recipes which are short shell scripts required to do the actual processing.

As a consequence, `make` might fail when you forget to list a dependency. In the scenario above, if `make` attempts to create file_c before it created file_b, this will obviously fail. Even worse, `make` uses file timestamps to know which files are outdated and must be recreated. So, if you have a file_b from a previous build and file_c misses the dependency on it, it might be built using an older version of file_b.

Now, enter "recursive make": This is a widespread idiom for organizing a large source tree. Instead of depending on individual files from a subdirectory, you depend on a pseudo-target with a recipe that will just call `make` again inside this subdirectory, that has its own Makefile. This is somewhat simplified, but the main consequence is that you have to be very careful about the dependencies of these pseudo-targets, so the directories are built in the correct order (if you have errors there, they will typically manifest when building in parallel with make's `-j` option), and it will result in whole directories being built even if only a single file would be needed. But it has a few advantages as well: The whole structure is easier to grasp than with ONE huge Makefile (possibly using includes from subdirectories), and you have an easy way to just build one individual directory by calling `make` from in there. So, assuming you get the ordering correct, recursive make still has the one downside that it will build too much when only little parts changed. FreeBSD /usr/src uses recursive make and solves the problem of unnecessary build steps in a different way, see _meta-mode_.

The ports tree also uses recursive make. But this is an entirely different thing, as it doesn't build anything itself, but instead organizes builds of individual software packages, from downloading the distribution files to packaging the result of the builds. For the actual builds, the build systems provided by the upstream packages is used. So, there is just no alternative to recursive make for a thing like the ports tree. Furthermore, this has nothing to do with the handling of dependencies between ports. These are handled in a clever way by the make framework in /usr/ports/Mk, which uses special variables set in the Makefiles like e.g. `BUILD_DEPENDS`. And finally, you almost never use the recursive infrastructure of the ports tree, but instead build individual ports directly from within their directory.

Portmaster also doesn't use the recursive make infrastructure, but parses the special variables in the individual ports' Makefiles and handles these dependencies itself. So, it has *nothing* to do with "recursive make". Also, there's nothing wrong with Portmaster _per se_, it does work as intended.

Now, what's the problem with Portmaster? The same as with building ports manually from the ports tree. It's neither a problem with the ports tree OR with Portmaster itself, but with the build systems of some upstream software packages.

Many software packages have build-time configurations, so they can optionally require additional libraries for optional features. The build systems used offer means for configuration, e.g. with the widespread "GNU autotools", you will have a ./configure shellscript doing that. These configuration tools are meant to do two things: They let the user decide (e.g. with commandline flags) what optional features should be enabled, and they check whether the needed libraries for these features are actually available.

Now, due to either bugs or cluelessness of upstream authors, it can happen that some optional feature is automatically enabled if the build system finds the needed libraries on the system you build on. This is called an "automagic dependency" and should be avoided, but sometimes, upstream build systems fail to do so. It happens mostly when home-brewn build systems (e.g. with large hand-written Makefiles) are used. Say the port/package "frobnicate" has an option to enable feature BAR, which should cause it to link "libbar.so". What *could* happen is that you don't enable that option, so the package you install won't have a dependency on libbar, but the buildsystem would find that libbar.so on your system and link to it nevertheless. You will end up with an implicit dependency `pkg` knows nothing about, and once you remove "libbar" (pkg will have no objections, nothing depends on it!), your "frobnicate" is suddenly broken.

Tools like poudriere or synth solve this problem: They run each build in a clean environment with ONLY the packages installed that are actual and intended dependencies of the port to build. This is again a BIT simplified, but it's basically the main thing…

So, all of this has nothing to do with "recursive make", and Portmaster isn't "broken" either, it's just there are better and more reliable alternatives and you should look into using one of them to profit from it.


----------



## Itsacon (Apr 12, 2021)

Zirias : Thank you for that clear example of why a clean build environment matters. I've been using FreeBSD for something like 20 years, and never was really aware of that.

As for the core of the debate: I started out using portupgrade, since that was what the handbook said to do back in FreeBSD 3. That was a PITA, since you had to either:

Keep watching the build in case something had a config screen that needed to be OK-ed or edited
Use the `--batch` flag, which had its own problems and risks (for example: giving you large numbers of unwanted dependencies, such as with things that try to install x11 by default)
That alone was one of the reasons I started using portmaster: It moves all those config commands to the start of the process, and after that you only have to keep checking if it still looks like the Matrix in that terminal...

(`--list-origins` is also a very useful flag if you're replicating a server.)

I never switched to poudriere, as I never needed it. Sure, portmaster has some problems, but if you know what it's doing, they are easy to deal with. For instance, I switched to security/libressl after Heartbleed, and after the first time updating it, I knew that whenever that needs an upgrade, I need to pass the `-r` flag to portmaster, in case the library number has been incremented (which would cause Apache and other software depending on it be confused at its next restart).

This thread has made me take a good look at synth, and I'm currently giving it a try on one of my servers. If that works, I might integrate it into my workflow.

Interesting thing: building synth using portmaster failed on my system (devel/adacurses installs devel/ncurses, but synth itself wants ncurses from base). Using the ol' `make install clean` worked fine though. This seems to be somewhat related to what Zirias describes above...


----------



## mefizto (Apr 12, 2021)

Hi Jose,



> * I rsync the ports to the server. I have my private repository on both machines set to a higher priority than the Freebsd quarterly repo. This allows me to install a mix of locally-built and standard ports



Could you please elaborate on this point?

As I understand, you are building packages from ports, so I assume that you rsync the built packages, correct?  So should the last sentence not read:



> This allows me to install a mix of locally-built and standard ports packages.



I am not being grammar Nazi, but it seems to be important here, because of the repeated mantra of "not mixing [packages buit form] ports and packages", which is implied.

I further do not understand why would one want to do such.  I have been using ports-mgmt/synth and its options allows to fetch and install a package if the inter-dependencies do not require build it.  So, I give ports-mgmt/synth the port reference and if ports-mgmt/synth does not need to build, ports-mgmt/synth just downloads and - if instructed - installs the package.

Or perhaps I am misunderstanding your work-flow?

Kindest regards,

M


----------



## Jose (Apr 12, 2021)

mefizto said:


> As I understand, you are building packages from ports, so I assume that you rsync the built packages, correct?  So should the last sentence not read...
> I am not being grammar Nazi, but it seems to be important here, because of the repeated mantra of "not mixing [packages buit form] ports and packages", which is implied.


You are correct! Post edited.



mefizto said:


> I further do not understand why would one want to do such.  I have been using ports-mgmt/synth and its options allows to fetch and install a package if the inter-dependencies do not require build it.  So, I give ports-mgmt/synth the port reference and if ports-mgmt/synth does not need to build, ports-mgmt/synth just downloads and - if instructed - installs the package.


I've never tried synth, and don't intend to. I believe it builds and installs ports, not packages? My reasons for avoiding tools like portmaster and synth are:

I don't want to build the same stuff over and over. It's a waste of electricity
I want to set some top-level options and some package level options for the packages I use
I want my custom packages to build in a pristine and self-consistent environment
I like to have one beefy machine that is not on all the time do the builds. The downstream consumers of my builds are on all the time, but are more power-efficient.
HTH


----------



## mefizto (Apr 12, 2021)

Hi Jose,

thank you for the reply.  However, by my introduction of the ports-mgmt/synth, I derailed the conversation.

So, by having your locally-built and standard repository, you do mix packages buit from ports and standard packages, correct?

Kindest regards,

M


----------



## Jose (Apr 13, 2021)

mefizto said:


> So, by having your locally-built and standard repository, you do mix packages buit from ports and standard packages, correct?


That is not how I would describe it. My systems have a combination of packages built by Poudriere using standard options on Freebsd.org hardware and packages built by Poudriere using my options on my hardware.

The details are important because that's where the devil lives. When I run Poudriere, it builds not just the packages I want, but also the entire dependency tree for these pacakges every time (I use the `-c` option to ensure this); and it builds them on a fresh install of 12.2-RELEASE with absolutely nothing else installed in it. Packages built in this way are preferred over standard packages to ensure consistency. I do this before every run of pkg-upgrade(8).


----------



## mefizto (Apr 13, 2021)

Hi Jose,

I appreciate your willingness to explain your workflow, but I think that I am not knowledgeable enough to appreciate the difference.  As I understand all the standard packages, to use your term, are built by Freebsd.org using Poudriere with the standard options.  You are modifying options on some of the packages, but then use these packages and the standard packages.  Thus, mixing packages build [with non-standard options] from ports and packages built with standard options.

Do not worry about my failure to understand, I am glad that it works for you.
Kindest regards,

M


----------



## Jose (Apr 13, 2021)

mefizto said:


> I appreciate your willingness to explain your workflow, but I think that I am not knowledgeable enough to appreciate the difference.  As I understand all the standard packages, to use your term, are built by Freebsd.org using Poudriere with the standard options.  You are modifying options on some of the packages, but then use these packages and the standard packages.  Thus, mixing packages build [with non-standard options] from ports and packages built with standard options.


Yes, but Poudriere ensures that my nonstandard options are propagated to the entire tree of packages.


Consider the following example. I do not use Mysql, and prefer Postgresql. The standard options usually do the opposite, they enable Mysql compatibility, and disable Postgresql. I have the following in /usr/local/etc/poudriere.d/make.conf.

```
OPTIONS_SET+=PGSQL
OPTIONS_UNSET+=MYSQL
```

Say I want to run application _foo_. Application foo depends on library _bar_. They both have Mysql and Postgresql options. I install them out of ports, and I have the correct options in my make.conf. Everything connects to my Postgres and things are happy.

Some time later, I decide to install package _baz_ that also depends on library _bar_. I install it out of packages without thinking. Two things can happen now. I can be annoyed by the large amount of Mysql stuff that gets dragged in and realize my mistake, or I could go blissfully along and then spend some time frustratingly trying and failing to get _baz_ to connect to my database.

This would've all been avoided if I'd just built _bar_ and _baz_ with Poudriere. They, and all the libraries they depend on would've all been built with the correct options.

This is a very simple example of a dependency tree with just one branch and one level of edges. Imagine how quickly this can get out of hand when you have dependency trees with many branches and levels. Imagine further that database options come and go with new application and library versions.

The term I like for servers that are lovingly hand-crafted with make.conf options, src.conf settings and the like is _snowflake server_. They are beautiful and unique, and a nightmare to maintain or migrate away from. Yes, been there, done that. I work hard up front now to make my server builds reproducible. It pays off in terms of time and frustration over the long run.


----------



## mefizto (Apr 13, 2021)

Hi Jose,

thank you for the reply, but it seems that the example that you have given with the _foo_, _bar, _and _baz_ supports my contention of mixing  (packages build from) ports with standard packages,_ i.e._, you build the non-standard package _foo _with Poudriere, and all the non-standard option is propagated through.  Perfect.

But, next you add a package with standard options, which causes the nightmare scenario that you describe.  Thus, is this not what I described as mixing (packages build from) ports with standard packages?  If not what am I missing?

I originally asked about the specific statement because I have been building all the packages from ports, exactly to avoid the scenario described in your message above.  Yes, it results in a lot of rebuilding if it is needed.

Again, the fact that I do not understand does not mean that it does not work well for you.

Kindest regards,

M


----------



## Jose (Apr 13, 2021)

mefizto said:


> thank you for the reply, but it seems that the example that you have given with the _foo_, _bar, _and _baz_ supports my contention of mixing  (packages build from) ports with standard packages,_ i.e._, you build the non-standard package _foo _with Poudriere, and all the non-standard option is propagated through.  Perfect.


This would be true iff ports options never changed. Fact is they do, and you'll wind up with an inconsistent set of options eventually if you mix packages and ports. Upgrades should be a transaction. Either all the ports in a dependency tree are updated at the same time or none are.

If your contention is that you can safely mix packages and ports if you never mess with the standard options; this is probably  true modulo some time jitter in the ports tree, but why would you bother building ports at all then? You're just wasting time and effort, since the best outcome of this approach is that you'll wind up with exactly the same software as if you'd installed the standard package. Also, there are not-so-good outcomes for this approach when the standard ports tree is out of sync with the standard packages set.


mefizto said:


> But, next you add a package with standard options, which causes the nightmare scenario that you describe.  Thus, is this not what I described as mixing (packages build from) ports with standard packages?  If not what am I missing?


I'm confused now. I thought your thesis was that you can mix ports and packages? I'm trying to prove the opposite.


mefizto said:


> I originally asked about the specific statement because I have been building all the packages from ports, exactly to avoid the scenario described in your message above.  Yes, it results in a lot of rebuilding if it is needed.


This is the pain of the ports user, but at least you wind up with a consistent set of software installed. My main criticism of this approach is above, under "snowflake server".


mefizto said:


> Again, the fact that I do not understand does not mean that it does not work well for you.


It sounds like what you're doing (installing everything from ports) works for you, and it's not inherently dangerous. The only reason I don't do that is that I like to have a small, reproducible set of steps to (re)-install and configure any Freebsd machine I maintain. You can come up with a set of steps for a ports-based approach, but it won't be small.


----------



## mefizto (Apr 13, 2021)

Hi Jose,

it is interesting how we could misunderstand each other.

It has never been my thesis that mixing (packages build from) ports with standard packages is a good idea.  In fact, if you look at my initial post (104), I was questioning how can you use both your built packages and the standard packages because that would constitute mixing (packages build from) ports with standard packages, which many posters, you among them, demonstrated to be a bad idea.

Based no the foregoing, I still do not understand how you can use both repositories and not have the problem you described.



> The only reason I don't do that is that I like to have a small, reproducible set of steps to (re)-install and configure any Freebsd machine I maintain. You can come up with a set of steps for a ports-based approach, but it won't be small.



This is another point I do not understand.  You are building with Poudriere, I am building with synth.  So, what is the difference you are alluding to?

BTW, we have moved quite away of the original topic.

Kindest regards,

M


----------



## Jose (Apr 14, 2021)

mefizto said:


> It has never been my thesis that mixing (packages build from) ports with standard packages is a good idea.  In fact, if you look at my initial post (104), I was questioning how can you use both your built packages and the standard packages because that would constitute mixing (packages build from) ports with standard packages, which many posters, you among them, demonstrated to be a bad idea.


But I'm not mixing packages and ports. I'm mixing packages and packages. This can be bad too, if you build one-off packages, but that's not what I'm doing. I'm building consistent package sets.


mefizto said:


> Based no the foregoing, I still do not understand how you can use both repositories and not have the problem you described.


Because my local package repo has higher priority than the standard packages. My local packages will be installed preferentially over the standard ones. This ensures consistency.


mefizto said:


> This is another point I do not understand.  You are building with Poudriere, I am building with synth.  So, what is the difference you are alluding to?


Again, I don't know how Synth works, and I'm not going to find out. Assuming Synth allows you to build ad-hoc package sets based on the current state of the host system, the difference is that Poudriere always builds packages on what is basically a clean install of the build host. In my assumed Synth case, the state of the build host changes over time, which means that builds done today may be different from builds done tomorrow.

I'm also assuming Synth handles so-called "reverse" dependencies correctly. This is not always the case for this kind of tool.


mefizto said:


> BTW, we have moved quite away of the original topic.


I think we're on topic, but am willing to start a new one if you'd like.


----------



## mefizto (Apr 14, 2021)

Hi Jose,



> But I'm not mixing packages and ports. I'm mixing packages and packages. This can be bad too, if you build one-off packages, but that's not what I'm doing. I'm building consistent package sets.


If you re-read my quote, you will notice that I am carefully specifying mixing (_packages build from) ports_ with _standard packages_.



> Because my local package repo has higher priority than the standard packages. My local packages will be installed preferentially over the standard ones. This ensures consistency.


This is exactly the crux of my misunderstanding.  Suppose that you issue a command to install a package. The local package repository is accessed first, but the requested package is not there.  Thus the standard package repository is accessed next, the package is found and installed.

This may or may not result in the disatrous scenario that you described.

As far as I understand it, synth is using the same build logic as Poudriere and. likewise, builds packages in a clean environment.  I do think that it does a great job in handling dependencies because since 2016 when I started using it, I never had a problem.  Of course, as mentioned above, unlike you I do not use a standard package repository, I build everything by synth.

Kindest regards,

M


----------



## Jose (Apr 15, 2021)

mefizto said:


> This is exactly the crux of my misunderstanding.  Suppose that you issue a command to install a package. The local package repository is accessed first, but the requested package is not there.  Thus the standard package repository is accessed next, the package is found and installed.


That cannot happen. Any package that is in the dependency graph of a package for which I want custom options will be built by Poudriere, and therefore will exist in my local package repo.


mefizto said:


> As far as I understand it, synth is using the same build logic as Poudriere and. likewise, builds packages in a clean environment.  I do think that it does a great job in handling dependencies because since 2016 when I started using it, I never had a problem.  Of course, as mentioned above, unlike you I do not use a standard package repository, I build everything by synth.


All I can find on the subject is:


> (Synth) internally creates jail-like environments for each builder...


https://github.com/jrmarino/synth

Contrast with:


> Because Poudriere performs its      building in a clean jail(8) environment and uses      zfs(8) features, it has several advantages over traditional      testing on the host system:
> 
> 
> No pollution of the host environment: No leftover files,      no accidental removals, no changes of existing configuration      files.
> ...











						Chapter 10. Testing the Port
					

Testing a FreeBSD Port




					docs.freebsd.org
				




Even if you could convince me that Synth is objectively superior to Poudriere, I still wouldn't use it. Why? Because Poudriere is used to build the standard Freebsd packages. I want to diverge as little as possible to minimize problems. Also, because so many people have a stake in having Poudriere work, I'm more likely to find solutions to any problems I run into, and have some confidence that bugs will be found and fixed quickly.

I looked at the Synth source, and I have to admit the Ada is beautiful. I'm not a fan of shell, and Poudriere is definitely an ugly duckling here, but I'm still going to use it.


----------



## mefizto (Apr 15, 2021)

Hi Jose,

thank you for your patience, but I still do not understand.


> > mefizto said:
> >
> >
> > This is exactly the crux of my misunderstanding.  Suppose that you issue a command to install a package. The local package repository is accessed first, but the requested package is not there.  Thus the standard package repository is accessed next, the package is found and installed.
> ...


As understand, all the packages that may depend on the package build from ports will also be built.  As I am reading your reply, you will install only packages from your local repository.  Why then do you have the standard repository?

What am I missing?



> Even if you could convince me that Synth is objectively superior to Poudriere, .  .  .


I respectfully submit that none of may statement can be interpreted in this manner.  Lest there be any misunderstanding, I am not trying to convince anybody about anything.

Kindest regards,
M


----------



## Jose (Apr 15, 2021)

mefizto said:


> As understand, all the packages that may depend on the package build from ports will also be built.  As I am reading your reply, you will install only packages from your local repository.  Why then do you have the standard repository?


I use the standard repo packages when the standard options they were compiled with work for me.


mefizto said:


> I respectfully submit that none of may statement can be interpreted in this manner.  Lest there be any misunderstanding, I am not trying to convince anybody about anything.


I didn't mean to imply that you were.


----------



## mefizto (Apr 15, 2021)

Hi Jose,



> I use the standard repo packages when the standard options they were compiled with work for me.


Ah, but how do you know that?  In post #109, you have described a disastrous scenario caused by a "package _baz" _installed from the standard repository. Now, admittedly, you have added a disclaimer "without thinking". 

Maybe, this is what I am missing, that one has to know enough to understand whether or not such a package may cause a problem?

Kindest regards,

M


----------



## Jose (Apr 16, 2021)

mefizto said:


> Ah, but how do you know that?  In post #109, you have described a disastrous scenario caused by a "package _baz" _installed from the standard repository. Now, admittedly, you have added a disclaimer "without thinking".


You are right, my approach would not protect me in that case. The standard package would be installed, and it would not work with Mysql. However, I can never get a package _bar_ that doesn't support Postgresql because that package is in the dependency graph for _foo_ which is in my set of packages I build.


----------



## mefizto (Apr 16, 2021)

Hi Jose,

I agree with the above, and I am glad that we untangled it.   Thank you for your patience.

Kindest regards,

M


----------

