# Questions from the "About ports and (binary) packages" howto



## sackadoo (Feb 28, 2021)

A couple of questions....
1) If I use all default settings while installing from ports, mixing ports and packages isn't a problem?
2) If I install (for instance) bash from the ports collection, installing an unrelated package (x-windows?) shouldn't be a problem?


----------



## ShelLuser (Feb 28, 2021)

sackadoo said:


> 1) If I use all default settings while installing from ports, mixing ports and packages isn't a problem?


Not necessarily, a better description here would be: it may not have to be a problem. But it also remains to be seem how long that status quo is going to last.

See, another issue here is that updates will first appear in the ports collection, so before they appear in the binary repositories. Simple reasoning: those binaries get build from the same ports collection, and building takes time. So if you then start working with an updated ports collection while also having out of date binary packages installed you're in for a nasty dependency mess.

Seriously: if you're just going to use default settings then it's best not to bother with ports at all, rely on binaries. FreeBSD isn't like Linux where several developers seem clueless about optimization. Normally a binary package is already well optimized (as a side note: the same applies to the kernel, FreeBSD also doesn't have this needless demand for kernel building either, it's actually a waste of time if you're not going to apply specific changes. And to make matters worse: even applying specific changes can sometimes be a bad idea ).



sackadoo said:


> 2) If I install (for instance) bash from the ports collection, installing an unrelated package (x-windows?) shouldn't be a problem?


Can't say for sure. Maybe not, maybe so. See...


```
root@vps:/usr/ports/x11/xorg # make all-depends-list | wc -l
     412
```
While x11/xorg may not directly rely on bash, what about any of those 412 dependencies? Or any of the ports which those 412 ports possibly depend on?

And of course: what about ports that are depended on by _both_ Bash and Xorg (or one of its many required ports)?

In the end I wouldn't bother with this, either build the whole thing or rely on binary packages. It's bound to spare you a lot of dependency issues in the upcoming future.


----------



## zirias@ (Feb 28, 2021)

sackadoo said:


> 1) If I use all default settings while installing from ports, mixing ports and packages isn't a problem?


Not sure whether this is already somewhere in this howto, didn't find it quickly:

I have the impression that the biggest source of "mixing problems" for people is mixing quarterly and latest. On a fresh FreeBSD installation, `pkg` is configured to use repositories built from the "quarterly" branch, while tools like `portsnap` will by default fetch a "latest" ports tree. Mixing this is a recipe for getting into trouble very quickly, so either reconfigure pkg to use "latest" packages or check out a quarterly branch of the ports tree for building your own.

I assume other problems are already addressed  In general, once you start mixing, you will of course have more manual maintenance work to do.

*edit:* quick conclusion: it's not so much about build options; for them, follow the simple rule that if you change options for a port, make sure to build all dependent ports yourself as well.


----------



## Deleted member 30996 (Feb 28, 2021)

I like ports and have installed graphics/gimp from pkg as last resort but only because it's a program I use daily. 

pkg didn't make a machineid like it should have to function and pkg installed a version of graphics/openEXR that had an active vulnerability according to `pkg audit -F`. Which is where ports-mgmt/portmaster would throw an error and stop the build, just as I would want it to.

I had to look but the file wasn't hard to create. I deinstalled openEXR manually due to reason stated and Gimp worked fine without it. Relatively easy for someone with experience to work through. Most likely not as easily before I had enough struggle time in figuring things out behind me. 

And that's just an example of something bound to come up now and then using ports as part of the learning process. pkg is no doubt quicker than ports and no sense in making it hard on yourself mixing them when you don't have to. That opportunity will show up when you least expect it.


----------



## dave01 (Feb 28, 2021)

I have successfully mixed ports and packages, but with some caveats.

Most importantly, as mentioned above, make sure pkg is pulling from latest, not quarterly and your portstree is up to date.  Secondly, install from pkg not ports and only then, for the hopefully very few ports you want custom settings for, update rather than install the relevant ports.

I use portinstall which will almost always either build and install the relevant port or stop and list the ports it's going to build.  If it does stop and display a list of ports, hit N for No and check carefully what it's about to do.  Any missing ports, install from pkg.  Any other ports, think carefully if you want portinstall to rebuild/upgrade them.  Personally, I just leave the port alone until after the next pkg update and see if the other out-of-date ports are now in sync.  I will normally ONLY upgrade a port when it builds cleanly with the existing pkgs in place.  Sometimes that means waiting a week or two, but it seems a lot safer to me.

In my case, it's a couple of ports which don't support mp3 OOTB for whatever reason.

Note that if this is a first time use of ports after exclusively using packages, you will likely find lots of build dependancies that you need to add, again preferably from pkg.  You probably don't want GCC, CMake LLVM and/or others all being built from ports just to install a custom build of audio/sox!


----------



## Jose (Feb 28, 2021)

sackadoo said:


> A couple of questions....
> 1) If I use all default settings while installing from ports, mixing ports and packages isn't a problem?


The problem is that it likely won't be a problem, perhaps even for years. When you do stumble into an incompatibility deep in your dependency tree, however, it will be very difficult to find. You may conclude Freebsd is just broken and abandon it. This is why we don't recommend you even try this until you know what you're doing.

For the record, I don't do this. It's not worth the trouble. I'll burn the extra time building Rust in Poudriere just to save myself from wandering around in WTF land.


----------



## Barney (May 1, 2021)

ShelLuser said:


> Hi gang!
> 
> *Introduction* (editorial section)


Where is the source code used to build packages, for when I need to compile something from source but the source doesnt compile and there is no port?

(Mod: removed full quote)


----------



## ShelLuser (May 2, 2021)

Now, first of all a well meant comment: please be more careful with your quoting. You don't always have to quote something to answer a question and well... quoting an entire guide (or message) for a single question (or answer) is quite counter productive and only makes things harder to read. Depending on the size of the original message of course...



Barney said:


> Where is the source code used to build packages, for when I need to compile something from source but the source doesnt compile and there is no port?


I'm not sure I understand your question.

Packages get build from ports, so if you have a package you'll also have access to a port *. The port will grab the source code (if available **) and build it. If you only want to get access to the source code you'd use the `make extract` command within the port directory, see also the ports(7) manualpage.


* Provided that the software was part of the ports collection in the first place. People can distribute packages without the source code, so then you obviously wouldn't get access to it.

** There is also closed-source software within the ports collection, obviously you don't get access to any source code; in these cases the port grabs the software and sets it up on your system.


----------



## outpaddling (Jun 27, 2021)

Mixing is unavoidable if you use binary packages but want something that cannot be packaged for licensing or other reasons, such as libdvdcss.  Until recently, some very common tools like faac, lame, and exfat were also not available as binary packages.  Fortunately, restrictions tend to become more relaxed over the years, but there will always be some things we have to install from source.

The important thing is to keep your installed packages and ports tree in sync.  That means on the same branch (quarterly or latest/head) and up-to-date.  The default FreeBSD install uses quarterly packages by default.  If you stay with that, be sure to clone the matching quarterly ports tree and upgrade it at the start of each quarter.
auto-update-system from sysutils/auto-admin will ensure that your ports tree and packages are always in sync.  It updates packages, ports, and base all at once and makes sure your porta and packages are on the same branch.
I frequently run `auto-update-system --defaults`, then review the output and decide if I should reboot.

sysutils/desktop-installer uses these tools under the hood to ensure that your setup is consistent from the start.


----------



## ShelLuser (Jun 27, 2021)

outpaddling said:


> Mixing is unavoidable if you use binary packages but want something that cannot be packaged for licensing or other reasons, such as libdvdcss.


No it's not. You can easily switch from binary to building your own ports for example.

Or... install the software outside the packaging system, so manually.

And, uhm...


```
libdvdcss-1.4.2
Name           : libdvdcss
Version        : 1.4.2
Origin         : multimedia/libdvdcss
Architecture   : FreeBSD:12:amd64
Prefix         : /usr/local
Repository     : FreeBSD [pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly]
Categories     : multimedia
Licenses       : GPLv2
Maintainer     : jpaetzel@FreeBSD.org
WWW            : http://www.videolan.org/developers/libdvdcss.html
Comment        : Portable abstraction library for DVD decryption
Options        :
```
What made you so sure that libdvdcss wasn't available as a package while it clearly is?


----------



## Crivens (Jun 28, 2021)

ShelLuser said:


> What made you so sure that libdvdcss wasn't available as a package while it clearly is?


It is about some options, which are verboten someplace of the world. Like decss for DVDs or TrueType fonts for libreoffice. That was a patent issue, no idea if it still is.


----------



## Deleted member 30996 (Jun 28, 2021)

outpaddling said:


> I frequently run `auto-update-system --defaults`, then review the output and decide if I should reboot.


I don't do auto Admin anything or run `portmaster -a` as SOP and never update anything unless ports-mgmt/portmaster recommends it as a dependency update. Or if it's a new version of something like www/youtube_dl and I want the latest version.

I've only built a Systenm using pkg once in all the time I've used FreeBSD and that was just a few months ago. I downloaded a snapshot of the ports tree and if there is a vulnerability that pkg hasn't addressed will use portmaster if a fix has hit the ports tree.

I haven't had any problems with it and am using that IBM T43 as my recliner-based text editor at 23 days uptime, located next to my W520 .mp3 player at 56 days uptime, so I can musically multitask with multiple machines.


----------



## outpaddling (Jun 28, 2021)

ShelLuser said:


> No it's not. You can easily switch from binary to building your own ports for example.
> 
> Or... install the software outside the packaging system, so manually.
> 
> ...


I said "If you use binary packages ... mixing is inevitable".  How does the fact that you don't have to use binary packages invalidate this statement?


```
<<<ROOT@barracuda.uits>>> /home/bacon 1035 # pkg install libdvdcss
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'libdvdcss' have been found in the repositories

PORTNAME=       libdvdcss
DISTVERSION=    1.4.3
CATEGORIES=     multimedia
MASTER_SITES=   https://download.videolan.org/pub/${PORTNAME}/${DISTVERSION}/

MAINTAINER=     ports@FreeBSD.org
COMMENT=        Portable abstraction library for DVD decryption

LICENSE=        GPLv2 DMCA
LICENSE_COMB=   multi
LICENSE_NAME_DMCA=      DMCA
LICENSE_TEXT_DMCA=      CSS code may violate the DMCA
LICENSE_FILE_GPLv2=     ${WRKSRC}/COPYING
LICENSE_PERMS_DMCA=     auto-accept
```

libdvdcss used to be packaged, but it isn't anymore.  Please read the post and gather the facts before you respond.


----------



## Jose (Jun 28, 2021)

First of all, this is a guide, you should discuss your opinions elsewhere.

Secondly, you're dispensing bad advice. It is indeed possible to run a system with just binary packages, and mixing ports and packages is a really bad idea.



outpaddling said:


> I said "If you use binary packages ... mixing is inevitable".  How does the fact that you don't have to use binary packages invalidate this statement?


Lastly, do you not see that that's a directly contradictory statement? You state it is "inevitable", yet somehow Shelluser and I have managed to do it. Do you know what the word "inevitable" means?


----------



## Deleted member 30996 (Jun 28, 2021)

Jose said:


> Secondly, you're dispensing bad advice. It is indeed possible to run a system with just binary packages, and mixing ports and packages is a really bad idea.


Right you are.

I only do it as a last resort, and only because I feel confident I can work out any problems mixing ports and pkg is likely to bring with it down the line.

Something a new user would be unable to do, become needlessly frustrated by and might cause them to lose interest in FreeBSD


----------



## jmos (Jun 28, 2021)

Jose said:


> First of all, this is a guide, you should discuss your opinions elsewhere.
> 
> Secondly, you're dispensing bad advice. It is indeed possible to run a system with just binary packages, and mixing ports and packages is a really bad idea.


Opinion? Those cheap sentences are the same as a "shut up & go away"… You know f.e. the option "NO_PACKAGE" in the Makefiles? Here's a list of ports with it:

audio/madfufw
devel/papi
lang/petite-chez
mail/sendmail-devel
mail/sendmail
misc/estic
misc/posixtestsuite
multimedia/sms1xxx-kmod
net/bwi-firmware-kmod
net/bwn-firmware-kmod
net/dimes
net/malo-firmware-kmod
net/opennx
net/vmware-vsphere-cli
security/stunnel
security/tripwire-131
security/truecrypt
security/vpnc

So, let's tell me how to get one of them as binary package - otherwise you should write an apology  outpaddling wrote some really good hints if it comes to mixing ports and packages. And not an opinion.


----------



## outpaddling (Jun 28, 2021)

Reiterating the original statement that's inexplicably under attack here:

"Mixing is unavoidable if you use binary packages but want something that cannot be packaged for licensing or other reasons..."

This is obvious, not controversial.  If there is no binary package, you have to build from source.  So your choices are either mix or don't use binary packages at all.

Users will run into problems with compatibility *IF* their ports tree and installed packages are not in sync.  That's why I wrote auto-update-system and put it in the auto-admin menu, so new users won't get themselves into trouble.  It's a very robust script developed and tested over 10+ years.  Trying to manage a FreeBSD system without tools like this is what scares new users away.

There is nothing inherently bad about mixing ports and packages.  The only problem is people not knowing how to do it safely.   I've been doing it for many years without any problems.  Not only for licensing reasons, but also for optimizing scientific software.  A small number of ports will benefit greatly from building with -march=native on a CPU with AVX extensions, for example.  Building all of the 900+ ports on my scientific workstation would be an absurd waste of time and electricity, though, given that only a handful would benefit from AVX in a meaningful way.   One can just add `CFLAGS+=-march=native` to /etc/make.conf, then `cd /usr/ports/biology/mmseqs2 && make install` to get a binary that's ~30% faster than the binary package.

There are also some ports that will fail to build in the presences of certain installed software, but build fine under poudriere, which always starts with a clean slate.  Hence, using binary packages avoids some problems and falling back on building from source only when necessary is a good strategy.

Lastly, if building something from source with many large dependencies, one can expedite the process greatly by first installing the dependencies as binary packages.  This will often reduce hours of build time to minutes.

Again, if you do use both binary packages and ports, be sure that both your installed packages and ports tree are on the same branch and fully updated.  If you do that, you'll likely never encounter a problem.


----------



## monwarez (Jun 28, 2021)

jmos said:


> Opinion? Those cheap sentences are the same as a "shut up & go away"… You know f.e. the option "NO_PACKAGE" in the Makefiles? Here's a list of ports with it:
> ...
> net/bwi-firmware-kmod
> ...
> So, let's tell me how to get one of them as binary package - otherwise you should write an apology  outpaddling wrote some really good hints if it comes to mixing ports and packages. And not an opinion.


Well, I just had to do
`doas poudriere bulk -j 122amd64 -O overlay -p HEAD net/bwi-firmware-kmod`
Push it to my remote pkg repository
And then I can found it without issue:
`pkg search bwi-firmware-kmod`
Which output

```
bwi-firmware-kmod-3.130.20     Broadcom AirForce IEEE 802.11 Firmware Kernel Module
```


----------



## Jose (Jun 29, 2021)

So 17 out of 44 thousand means that mixing ports is inevitable? I'd never even heard of most of those, and I've certainly never needed them.  If I did, I'd either run a system with just ports or make my own packages.

Saying that mixing ports and packages is "inevitable" is so wrong, it's just plain stupid. It's also bad advice to give.


----------



## SirDice (Jun 29, 2021)

[_Mod: Split the entire thread of questions and discussions from the howto here: https://forums.freebsd.org/threads/guide-about-ports-and-binary-packages.62126/_]


----------



## jmos (Jun 30, 2021)

Jose said:


> So 17 out of 44 thousand means that mixing ports is inevitable? I'd never even heard of most of those, and I've certainly never needed them.  If I did, I'd either run a system with just ports or make my own packages.
> 
> Saying that mixing ports and packages is "inevitable" is so wrong, it's just plain stupid. It's also bad advice to give.


One of a million will turn true into false.

Of course you can use FreeBSD without mixing ports and packages (you can even use it without any port or package). And of course you can set up your own package server and compile all by yourself. But what if you want to go with the official packages, but f.e. need MariaDB as well as LibreOffice and Kdenlive? You need to compile some packages by yourself. And that's not a problem. Done that many years.

Simple example: You don't want the subshell support of Midnight Commander. Where's the problem if you're compiling only that port? None other port depends on it, so nothing can beak. The day MC stops to work will come (but: never came to it), but then you just need to recompile this port. "Quarterly package update" means "quarterly ports update of a few own packages". Much more less complicated, far less tripping hazards, and much more time- and resource-saving than your solution with poudriere. Using poudriere therefore would be "plain stupid" and a "bad advice".

You found your way, and it fits perfect. And you're sharing your experience here with others - and that's really great! But it's not the holy grail for everyone; It's just your recommendation.


----------



## Jose (Jun 30, 2021)

jmos said:


> One of a million will turn true into false.











						False dilemma - Wikipedia
					






					en.wikipedia.org
				





jmos said:


> Of course you can use FreeBSD without mixing ports and packages (you can even use it without any port or package). And of course you can set up your own package server and compile all by yourself. But what if you want to go with the official packages, but f.e. need MariaDB as well as LibreOffice and Kdenlive?


Use ports and not packages. This is not complicated.


jmos said:


> (Snip dead horse flogging)
> You found your way, and it fits perfect. And you're sharing your experience here with others - and that's really great! But it's not the holy grail for everyone; It's just your recommendation.


No. Not mixing ports and packages is not just "my recommendation." It's the recommendation of most knowledgeable Freebsd users.


----------



## Alain De Vos (Jun 30, 2021)

In case of conflicting packages you can also use jails.


----------



## bsduck (Jun 30, 2021)

Jose said:


> Use ports and not packages. This is not complicated.


Not complicated but certainly overkill and not really time-efficient when all you want is to install a few ports that are not distributed as packages.


----------



## Alain De Vos (Jun 30, 2021)

I compile ports in the background. I've set the number of parallel build so I don't even notice (lag) . Then its even ok if compilation would take weeks.


----------



## outpaddling (Jul 1, 2021)

monwarez said:


> Jose said:
> 
> 
> > So 17 out of 44 thousand means that mixing ports is inevitable? I'd never even heard of most of those, and I've certainly never needed them.  If I did, I'd either run a system with just ports or make my own packages.
> ...



Nobody said is was inevitable for every user.  The conditions under which it cannot be avoided have been clearly stated twice now.

So Joe Ubuntu user decides to try out FreeBSD.  He quickly installs most of what he needs using binary packages and is pretty satisfied with the experience.  Now he wants to watch a DVD and finds out that there is no binary package published for libdvdcss.

What are you going to tell him?  Don't install it from ports because mixing ports and packages is "bad"?  Run "pkg delete -a" and reinstall everything from source (which will take days) because that's the right way?  Set up your own package server and populate it with poudriere?

Preaching dogma is no way to provide good advice.  There are pitfalls when mixing ports and packages, most of which result from your ports tree and installed ports/packages being out of sync.  Explaining exactly what those pitfalls are and how to avoid them is a far more intelligent approach.


----------



## Jose (Jul 1, 2021)

outpaddling said:


> What are you going to tell him?  Don't install it from ports because mixing ports and packages is "bad"?  Run "pkg delete -a" and reinstall everything from source (which will take days) because that's the right way?  Set up your own package server and populate it with poudriere?


Absolutely. _Especially_ if he's Joe Ubuntu. He's likely to run into a problem he can't even begin to solve if he mixes ports and packages. He'll conclude that Freebsd is "broken" and stop using it at that point anyway.
Besides, how' Joe Ubuntu going to feel when he discovers what he has to do to watch Netflix?


----------



## outpaddling (Jul 1, 2021)

Jose said:


> Absolutely. _Especially_ if he's Joe Ubuntu. He's likely to run into a problem he can't even begin to solve if he mixes ports and packages. He'll conclude that Freebsd is "broken" and stop using it at that point anyway.
> Besides, how' Joe Ubuntu going to feel when he discovers what he has to do to watch Netflix?


No, he won't have problems with this simple scenario as long as he keeps everything up-to-date:

pkg upgrade
cd /usr/ports && git pull
cd multimedia/libdvdcss && make clean deinstall reinstall

It's that simple and many people do this sort of thing regularly without ever having an issue.

or...

auto-mark-install-from-source multimedia/libdvdcss
auto-update-system

auto-update-system will automatically reinstall libdvdcss if the installed version is older than the one in ports.  I have at least half a dozen ports marked this way on each of my installations.  I've been using this for a long time and never had a single problem result from it.

Time to let go of meaningless generalizations like "mixing ports and packages is always bad".  It obviously depends on the state of your installation.  If everything is kept up-to-date, problems are extremely unlikely.


----------



## shkhln (Jul 1, 2021)

Jose said:


> Absolutely. _Especially_ if he's Joe Ubuntu. He's likely to run into a problem he can't even begin to solve if he mixes ports and packages. He'll conclude that Freebsd is "broken" and stop using it at that point anyway.


Did everyone forget drm-kmod upgrade troubles already? Whatever your preference is, don't bring strawmen "new users" here — keeping a desktop system exclusively using official binary packages is pretty much impossible in practice considering how often they are broken.


----------



## Jose (Jul 1, 2021)

shkhln said:


> Did everyone forget drm-kmod upgrade troubles already? Whatever your preference is, don't bring strawmen "new users" here — keeping a desktop system exclusively using official binary packages is pretty much impossible in practice considering how often they are broken.


So use ports, or get an Nvidia card. This is isn't complicated.


----------



## shkhln (Jul 1, 2021)

Jose said:


> So use ports, or get an Nvidia card. This is isn't complicated.



Nvidia's kmods also tend to be broken from time to time.
Nothing is complex for me specifically, I'm not a newbie.


----------



## PMc (Jul 2, 2021)

Jose said:


> Absolutely. _Especially_ if he's Joe Ubuntu. He's likely to run into a problem he can't even begin to solve if he mixes ports and packages. He'll conclude that Freebsd is "broken" and stop using it at that point anyway.


So, you have a problem with the browser, and want to quickly try a different browser for test. The different browser brings along as prereq its special version of llvm and qt5-webengine. That is already seven hours on my i5 (and one can run FreeBSD as a desktop with slower machines). At that point you have by long forgotten what you actually wanted to test.
This is the problem. Going with ports is all fine in a planned environment and for regular upgrades. It is very different on a desktop where you want to do something, say, with pictures or music or such, and have to try out what software is available and what it can do.
I don't say You're wrong! I say this is a problem and there is apparently no solution. Well, there is a solution: fetch the packages and don't forget to clean up afterwards. But people are rarely fond of clean up - if a configuration is found that works, it gets neither documented nor reproduced - it just stays to be used, until the next upgrade when it will likely break.


----------



## PMc (Jul 2, 2021)

I finally figured out the reason why one cannot mix ports and packages: ports are installed with `pkg add`, packages with `pkg install`.

Then, when mixing ports and packages, we need to install some with `pkg add`, and others with `pkg install`. This does not work: `pkg add` will complain about missing dependencies, instead of fetching them from the network.
We could then do this:

```
pkg add -M <packagename>.t??
pkg check -yd
```

`pkg add -M` should ignore the missing dependencies, and `pkg check -yd` would then supplement them from the repository. But it does not work:

```
+ chroot /mnt sh -c 'pkg add -M /tmp/vb/git-2.31.1_1.t??'
Installing git-2.31.1_1...
pkg: Missing dependency 'curl'
pkg: Missing dependency 'expat'
`-- Installing gettext-runtime-0.21...
pkg: Missing dependency 'indexinfo'

Failed to install the following 1 package(s): /tmp/vb/git-2.31.1_1.txz
```
pkg will indeed ignore the missing dependencies for git-2.31.1_1, but then, when installing the dependencies that are available, it comes across their missing dependencies, and then it fails. The `-M` is not inherited to subsequent invocations.

[usecase]


----------



## bsduck (Jul 2, 2021)

You can use `make install-missing-packages` in a port's folder to install its dependencies as packages.


----------



## Sevendogsbsd (Jul 2, 2021)

PMc said:


> I finally figured out the reason why one cannot mix ports and packages:


I am not sure about that. I am not saying the pkg add/install is wrong, what I mean is I understood that packages are built less frequently (quarterly?) and ports are constantly up to date. So, dependencies for ports could be newer than dependencies for a given build of packages. For example: a shared dependency between package "A" and port "B" would be upgraded when you built port "B" but then that breaks package "A" because the dependency is now newer than what package "A" was built against. I think I am saying that correctly.


----------



## bsduck (Jul 2, 2021)

If you use quarterly packages + quarterly ports, both will have the same versions so you won't get into that kind of trouble.

If you use latest packages + latest ports, you may encounter such a problem from time to time, but it should be solved in a few days.

Mixing quarterly + latest is likely to be problematic.


----------



## Jose (Jul 2, 2021)

bsduck said:


> If you use quarterly packages + quarterly ports, both will have the same versions so you won't get into that kind of trouble.
> 
> If you use latest packages + latest ports, you may encounter such a problem from time to time, but it should be solved in a few days.
> 
> Mixing quarterly + latest is likely to be problematic.


The quarterly ports branch is not as stable as advertised. That plus the fact that it takes several days for the official Poudriere runs to finish means you can still run into trouble even if you stick to the quarterly ports branch and quarterly packages.


----------



## outpaddling (Jul 6, 2021)

shkhln said:


> Did everyone forget drm-kmod upgrade troubles already? Whatever your preference is, don't bring strawmen "new users" here — keeping a desktop system exclusively using official binary packages is pretty much impossible in practice considering how often they are broken.


The drm-kmod issue was actually what prompted me to develop auto-mark-install-from-source and the accompanying functionality in auto-update-system.  This allowed full updates without breaking Xorg via the following:

auto-mark-install-from-source graphics/drm-fbsd12.0-kmod kbi

With this in-place, auto-update-system rebuilds and reinstalls drm-fbsd12.0-kmod from source after upgrading installed packages and updating the ports tree.

Background for those who don't know what it's about: The FreeBSD project aims to maintain full binary compatibility throughout each major release, so binaries built on 13.0-RELEASE should work on all future 13.X installations, for example.  In 12.2-RELEASE, however, there was a change that caused the drm-kmod binaries from 12.1 to fail.  One set of packages is published for each major release and they are built on the oldest supported release.  So for the 3-month overlap of support for 12.1 and 12.2, those of us running 12.2 had to build the drm-kmod modules from source rather than use the binaries from 12.1.  When 12.1 reached EOL and package builds started running on 12.2, this issue went away.

Joe U. is not a straw man, BTW.  A straw man is a non-existent and unrealistic opponent invented to create the appearance of winning a debate.  The scenario faced by Joe U. is very real and quite common.


----------



## mer (Jul 6, 2021)

drm-kmod:
outpaddling has the it as best as I recall from following the mailing lists and other places to complain.  It was the first breakage that I recall in a long time.

Time to build everything and push to all content servers in relation to "new quarterly packages available" is a legitimate concern, one that is being mitigated as best they can.  It's difficult to keep users from peeking early.


----------



## outpaddling (Jul 6, 2021)

Jose said:


> The quarterly ports branch is not as stable as advertised. That plus the fact that it takes several days for the official Poudriere runs to finish means you can still run into trouble even if you stick to the quarterly ports branch and quarterly packages.


Adding a new port to quarterly (the subject of the "not as stable" link) is not a problem.  The thing that is essentially forbidden in a MFH is a change to an existing port that breaks compatibility with other ports in the branch.

Your concern about the few days it takes to generate packages following a commit is valid in theory, but I've never had an issue with it in practice, even working with latest/head.  Most port upgrades don't alter the API in a way that will break anything, so the chances of getting tripped by this is very small.  It is worth watching out for if you mix, though.

Note also that the "few days" rule only applies to x86, for which we have a large build cluster.  powerpc and arm -latest packages are not  kept up-to-date (at the time of this writing), so if you need to mix ports and packages on those platforms, quarterly is much safer at this time.  Hopefully this will change in the near future.


----------



## Jose (Jul 6, 2021)

outpaddling said:


> Quarterly is actually very stable.  Adding a new port to quarterly (the subject of the "not as stable" link) is not a problem.  The thing that is essentially forbidden in a MFH is a *change* to an existing port that alters the API/A


Your reading comprehension is not strong. Here's what the "stable" ports branch is supposed to be


> Quarterly branches aim to receive (on a best effort basis) only the following _Merge From Head_ (_*MFH*_) commits, including, but not limited to:
> 
> 
> Security fixes (that may be version updates, or backports of commits)
> ...


Not new ports.

No nonsense about "altering the API/A" whatever that is.


----------



## AngryChris (Jul 7, 2021)

Your reading comprehension ain't so hot, either.


> including, but not limited to:


How about you tell us, in your own words, what that means.

By the way, it doesn't help your position that your source for "The quarterly ports branch is not as stable as advertised" is literally a link to *your own complaint*. That's not a "source". That's just you jumping up and down and demanding people pay attention to your issue.


----------



## Jose (Jul 7, 2021)

AngryChris said:


> Your reading comprehension ain't so hot, either.
> 
> How about you tell us, in your own words, what that means.


I don't have to. It immediately follows the section you quoted without context. Start reading at "More precisely..."


----------



## astyle (Jul 7, 2021)

PMc said:


> So, you have a problem with the browser, and want to quickly try a different browser for test. The different browser brings along as prereq its special version of llvm and qt5-webengine. That is already seven hours on my i5 (and one can run FreeBSD as a desktop with slower machines). At that point you have by long forgotten what you actually wanted to test.
> This is the problem. Going with ports is all fine in a planned environment and for regular upgrades. It is very different on a desktop where you want to do something, say, with pictures or music or such, and have to try out what software is available and what it can do.
> I don't say You're wrong! I say this is a problem and there is apparently no solution. Well, there is a solution: fetch the packages and don't forget to clean up afterwards. But people are rarely fond of clean up - if a configuration is found that works, it gets neither documented nor reproduced - it just stays to be used, until the next upgrade when it will likely break.


If I know a compilation takes 7 hours (like compiling the kernel before I discovered the -j4 make switch), I just do it overnight.

As for forgetting what I wanted to test - this is why I keep sort of a 'Diary' with what I wanted to test, why, what I did, what I installed, in what order, where I picked up a useful command, what I installed, what version - and what errors I got. That proved incredibly helpful with troubleshooting and asking for help later. And yeah, if I found a config that works, sometimes I bother documenting it, sometimes I don't. I started keeping the diary in 2003, back when I was messing with Linux. Continued with FreeBSD, the journaling habit served me well.


----------



## Deleted member 30996 (Jul 19, 2021)

Over the past month I had a series of 3-4 failed builds I didn't pursue when using ports-mgmt/portmaster to build new programs on this machine already successfully built on others. I did a couple days ago when www/youtube_dl had problems, and they led deep down a rabbit hole in different directions. 

I carefully read /usr/ports/UPDATING, (that didn't help) worked at it and tried a lot of things, including seen mentioned recently in threads. Mixing pkg and ports was the only way I saw to fix it, and I could tell things were about to get worse if I didn't get it fixed.  

I ran the build as usual and when it stopped I used pkg to install the base program that was stopping the rest from building and then restarted the build with portmaster. I used pkg to install four different programs and compiled the rest, but that's what it took. 

I used `shutdown -r now` when I was done and everything is copacetic


----------



## astyle (Jul 19, 2021)

Trihexagonal said:


> Over the past month I had a series of 3-4 failed builds I didn't pursue when using ports-mgmt/portmaster to build new programs on this machine already successfully built on others. I did a couple days ago when www/youtube_dl had problems, and they led deep down a rabbit hole in different directions.
> 
> I carefully read /usr/ports/UPDATING, (that didn't help) worked at it and tried a lot of things, including seen mentioned recently in threads. Mixing pkg and ports was the only way I saw to fix it, and I could tell things were about to get worse if I didn't get it fixed.
> 
> ...


Based on this weekend's study of portmaster(8), I do recall seeing info that ports-mgmt/portmaster can be asked to use  _pkg_ for `build-depends` deps, and even remove them afterwards if they're not needed... 

What I did this weekend is do a dry run with `# portmaster -n x11/kde5`, and it did produce a list of ports that would be installed. I ran that on an empty ports tree. My question here is, would running `# make config-recursive x11/kde5` *before* the dry portmaster run make any difference? 
If it does, I'm gonna be a step closer to solving the mystery of upgrading KDE in-place, without dependency hell.


----------



## Deleted member 30996 (Jul 19, 2021)

I had to install security/p11-kit and dns/libidn2 using pkg. multimedia/ffmpeg was problematic as well and overall, had what I, myself, would classify as a mess to clean up. 

Now I'm updating another that showed an error in the icon path due to a missing .so in graphics/gegl, that has already stopped with the same "FAILED: meson-build" error when compiling as this one did. 

Which is of some import, but can't see where at the moment, and played a part in resorting to use pkg to fix this machine.

It's now successfully compiling starting from the second port listed in the portmaster error in hopes I can move past the fail point and finish in compiling gegl.


----------



## astyle (Jul 19, 2021)

Usually, if meson fails, that's because of race conditions I created myself with the -j4 make flag... In those cases, I remove the work/ port directory, and re-run `make` without the -j4 flag.


----------



## Alain De Vos (Jul 19, 2021)

Of the 2300 ports i compiled i configured manually 40 i.e. 2%. The other 2260 ports have no options file.


----------



## Deleted member 30996 (Jul 20, 2021)

This was the FAIL message I got:

```
FAILED: install script '/usr/local/bin/meson --internal yelphelper install --subdir=help/manual --id=gtk-doc-manual --installdir=share/help --sources=index.docbook@@fdl-appendix.xml --symlinks=true' exit code 1, stopped
FAILED: meson-install 
/usr/local/bin/meson install --no-rebuild
ninja: build stopped: subcommand failed.
*** Error code 1

Stop.
make: stopped in /usr/ports/textproc/gtk-doc

===>>> make stage failed for textproc/gtk-doc
===>>> Aborting update
```
I reinstalled meson but that did not work. It came down to installing textproc/gtk-doc by pkg.

I also was missing dependencies:

```
Updating database digests format: 100%
pkg: gegl has a missing dependency: ilmbase
pkg: itstool has a missing dependency: py37-libxml2
pkg: gimp-app has a missing dependency: ilmbase
pkg: libinput has a missing dependency: py37-pyudev
```

So I complied gimp, itstool and  libinput with portmaster, then ran `portmaster -a` but that gave a failure with this error:

```
===>>> Returning to update check of installed ports

===>>> The misc/qtchooser port has been deleted: No longer supported upstream
===>>> Aborting update

root@bakemono:/ # portmaster misc/qtchooser
make: chdir /usr/ports/misc/qtchooser: No such file or directory

===>>> The misc/qtchooser port has been deleted: No longer supported upstream
===>>> Aborting update
```

I updated what it found to that point and everything seems to be running fine now.


----------



## astyle (Jul 20, 2021)

Yeah, ports-mgmt/portmaster is no AI... It just blindly chases deps until it runs into an error, with no ability to backtrack. Human analysis and judgement  are still necessary to supervise it.


----------



## Deleted member 30996 (Jul 20, 2021)

Running `portmaster -a`:


```
man portmaster 
*snip*
 Features:
     -a  check all ports, update as necessary
```

Let's try that:

```
root@bakemono:/ # portmaster -a
===>>> Gathering distinfo list for installed ports

===>>> Starting check of installed ports for available updates
===>>> Launching child to update nvidia-xconfig-460.67 to nvidia-xconfig-470.42.01

===>>> All >> nvidia-xconfig-460.67 (1/1)

===>>> Currently installed version: nvidia-xconfig-460.67
===>>> Port directory: /usr/ports/x11/nvidia-xconfig

===>>> Launching 'make checksum' for x11/nvidia-xconfig in background
===>>> Gathering dependency list for x11/nvidia-xconfig from ports
===>>> Initial dependency check complete for x11/nvidia-xconfig

===>>> Returning to update check of installed ports


===>>> The misc/qtchooser port has been deleted: No longer supported upstream
===>>> Aborting update

root@bakemono:/ #
```

It's broke in that regard and is up to the current version.

It shows nvidia needs updating and should go much further but aborts due to the deletion of misc/qtchooser from ports.


----------



## astyle (Jul 20, 2021)

Doing a dry portmaster run (with the -n option) inside a `script` session can still generate a text file that can be parsed with `grep` and `sed` for usable analysis.


----------



## Alain De Vos (Jul 20, 2021)

Why not use the , i think better, synth or poudriere ?


----------



## Tieks (Jul 20, 2021)

Trihexagonal said:
			
		

> The misc/qtchooser port has been deleted: No longer supported upstream.


Qtchooser allowed you to choose between two qt versions (4 and 5 iirc). It's no longer needed since we have only version 5 left. If you still haveit installed you can remove with `pkg delete -f qtchooser`.


----------



## Tieks (Jul 20, 2021)

Alain De Vos[/QUOTE said:
			
		

> poudriere ?


I don't like French poodles.


----------



## astyle (Jul 20, 2021)

Alain De Vos said:


> Why not use the , i think better, synth or poudriere ?


Actually, I am working on using portmaster's dry runs to create a list to feed to poudriere... But there's a lot of details to line up. For example, I'm still trying to figure out if `# make config-recursive` makes a difference to the list that portmaster generates.


----------



## astyle (Jul 20, 2021)

Tieks said:


> I don't like French poodles.


I do.


----------



## jmos (Jul 20, 2021)

Alain De Vos said:


> Why not use the , i think better, synth or poudriere ?


How do synth or poudriere remove a runtime dependency on misc/qtchooser from 61 ports?


----------



## Alain De Vos (Jul 20, 2021)

I currently have 2200 ports compiled and installed with poudriere.
For instance Chromium has a build dependency on python2, so i drop it and use firefox instead, i've blacklisted python2.
That is one way to solve dependency problems.
I have qtchooser installed which sits nicely together with the other 2199 ports.


----------



## jmos (Jul 21, 2021)

With portmaster I can simply blacklist a port, too. And as long I leave qtchooser on my installation available, I have no problem with ports using it. So in that case there's no benefit in not using portmaster, and switching to something else wouldn't solve anything…


----------



## Deleted member 30996 (Jul 21, 2021)

Tieks said:


> If you still haveit installed you can remove with `pkg delete -f qtchooser`.


I took the advise of Tieks and issued `pkg delete -f qtchooser`.

I was then able to continue with `portmaster -a`, it completed a full scan, I approved the goahead and it upgraded and/or installed 55 programs with no problem whatsoever. I ran the command again after `portsnap fetch update` just to make sure and it reported all programs were up to date.

I just finished it over the ensuing time period, have invoked `shutdown -r now`, am back up and running and once again things are as they should be.

I've got another going through the same procedure and will probably do the same on all my machines. They have all exhibited the same behavior, whether that's due to my own methodology or not, that's what needs to be done to bring them all back in line.


----------



## Alain De Vos (Jul 21, 2021)

jmos said:


> With portmaster I can simply blacklist a port, too. And as long I leave qtchooser on my installation available, I have no problem with ports using it. So in that case there's no benefit in not using portmaster, and switching to something else wouldn't solve anything…


I seems logical that dependencies cannot be solved by buildtools.
Only blacklisting or disable certain options on certain ports.


----------



## Jose (Aug 16, 2021)

Case study in why you  shouldn't mix ports and packages:








						I need help understanding pkg upgrade
					

I'm tired of running portmaster -ad and taking two or three days to build all the ports, resolving so many issues that pop up along the way. But pkg scares the hell out of me. I manage two servers. I have no backup or test systems to try things out before going into production. Both of these...




					forums.freebsd.org


----------

