# pkg repositories provided by the FreeBSD project needs to grow up (sigh)



## tingo (Jul 21, 2020)

So I like having packages. But far too often I end up in situations like this:

```
tingo@kg-core2$ openscad
Cannot mix incompatible Qt library (5.14.2) with this library (5.15.0)
Abort trap (core dumped)
```
Fine, I'll just upgrade OpenSCAD then

```
tingo@kg-core2$ sudo pkg update -f
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01 
Fetching packagesite.txz: 100%    6 MiB   6.3MB/s    00:01 
Processing entries: 100%
FreeBSD repository update completed. 30777 packages processed.
All repositories are up to date.
tingo@kg-core2$ sudo pkg search openscad
```
Except that it suddenly (for no good reason, seen from a pkg user's perspective) isn't in the package repository anymore. So I  can't. And I'm going to use OpenSCAD now. So do you think that I'm going to spend time trying to install openscad from ports? Hell no, this machine doesn't even have a ports tree. But I have a laptop running Debian in the other room; it has OpenSCAD installed, so I'll go and use that until the package repository gets fixed.

I repeat: FreeBSD's pkg repositories needs to grow up.

Details:

```
tingo@kg-core2$ uname -a
FreeBSD kg-core2.kg4.no 11.3-RELEASE-p10 FreeBSD 11.3-RELEASE-p10 #0: Tue Jun  9 08:49:05 UTC 2020     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
tingo@kg-core2$ pkg -vv| grep url
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/latest",
```


----------



## SirDice (Jul 21, 2020)

tingo said:


> Except that it suddenly (for no good reason, seen from a pkg user's perspective) isn't in the package repository anymore.


That has _nothing_ to do with pkg(8) itself. If a port fails to build (for whatever reason) then a package simply cannot be created. As a  consequence it disappears from the package repositories.

On the last build run 14 ports failed to build and 1318 ports were skipped (probably because a dependency failed).



			https://pkg-status.freebsd.org/builds?jailname=113amd64
		


Digging a little  deeper: http://beefy9.nyi.freebsd.org/build.html?mastername=113amd64-default&build=542302
Looking at that openscad  is  one of the 1318 skipped  ports. The reason it was skipped  was because of qt5-network, which fails on  11.3 due to OpenSSL. 

```
is marked as broken: Qt5 requires Openssl 1.1.1, upgrade to FreeBSD 12.x/13.x or add DEFAULT_VERSIONS+=ssl=[openssl|libressl*] to /etc/make.conf
```

All this has _nothing_ to do pkg(8).


----------



## zirias@ (Jul 21, 2020)

You completely miss the point. This has nothing to do with pkg but with a "rolling release" package repository -- the same will happen on a Linux system as well, cause if a package currently doesn't build, you can't have it, simple as that. You explicitly chose to use "latest", and this is the tradeoff.

If you used quarterly instead, there's very little room for such breakage. It can happen exactly at the times when a new quarterly branch is built.


----------



## SirDice (Jul 21, 2020)

Zirias said:


> If you used quarterly instead, there's very little room for such breakage.


There's no guarantee anything in quarterly isn't broken too. The quarterly branch is just a  latest branch that's more or less frozen for 3 months.


----------



## Alain De Vos (Jul 21, 2020)

Maybe it's just broken and removed from the repository.


----------



## Mjölnir (Jul 21, 2020)

/etc/pkg/FreeBSD.conf:

```
FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}
```
and then wait 2 weeks in Jan/Apr/Jul/Oct before doing `pkg update/upgrade`
EDIT Also keep in mind that most ports are maintained by volunteers, and much of their work is to cope with all those nasty _Linuxisms_ in today's world - not a very thrilling task.  Ironically, many of those can be found in toolkits like Qt or Gtk.


----------



## zirias@ (Jul 21, 2020)

SirDice said:


> There's no guarantee anything in quarterly isn't broken too. The quarterly branch is just a  latest branch that's more or less frozen for 3 months.


The whole purpose of quarterly is to avoid the constant changes (and possible breaks). Yes, it's "just a branch", but it receives fixes for security issues and build errors _and nothing else_ during its lifetime. No, there's no guarantee that a package doesn't "vanish" for build breakage, but it normally only happens at times a new quarterly branch is built.


----------



## shkhln (Jul 21, 2020)

tingo said:


> But far too often I end up in situations like this



That probably means you like partial updates. Pkg doesn't track package versions at all, so that's quite dangerous.


----------



## shkhln (Jul 21, 2020)

Zirias said:


> Yes, it's "just a branch", but it receives fixes for security issues and build errors _and nothing else_ during its lifetime.



Quarterly receives whatever it receives, including updates; this is left entirely to maintainer discretion.


----------



## Mjölnir (Jul 21, 2020)

tingo said:


> [...] Hell no, this machine doesn't even have a ports tree. [...]


You may want to.  There are some fine little scripts in the ports tree that do not have a package equivalent.
`portgrep -x -s 'NO_(ARCH|BUILD)=[[:blank:]]*yes' | cut -d / -f 1,2 -s | less` Most interesting are in _sysutils_.


> I repeat: FreeBSD's pkg needs to grow up.


As noted above by all others, it's a matter of the build process.  When 150 of 30,000 ports fail to build, that's 0.5%


----------



## tingo (Jul 21, 2020)

I consider the package repositories a vital part of pkg - without them the tool pkg(8) would be useless. Yes, I could set up my own packages, but that is beside the point; pkg and the packages / package repositories that FreeBSD provide needs to grow up.
As for the latest versus quarterly issue; why couldn't I just go with quarterly? At the time this machine was installed, I needed a drm-kmod package that wasn't in quarterly. So "damned if you do, damned if you don't".

Yes - I'm well aware that work for ports and packages is provided by volunteers - I've been with FreeBSD for a while now (as regulars here will know).


----------



## SirDice (Jul 21, 2020)

tingo said:


> Yes, I could set up my own packages, but that is beside the point; pkg and the packages / package repositories that FreeBSD provide needs to grow up.


It's perfectly fine if you have complaints or suggestions for improvement, but you're blaming the wrong part of the system.

The reason there's no package for openscad is because it depends on qt5-network. This dependency fails to build on 11.4 because of a change in qt5-network, it stopped supporting the version of OpenSSL that's in 11.4. This in turn causes a cascade of other ports failing too.

If you want to play the blaming game, blame qt5-network.


----------



## tingo (Jul 21, 2020)

SirDice said:


> It's perfectly fine if you have complaints or suggestions for improvement, but you're blaming the wrong part of the system.


I have changed the subject. Happy now?


----------



## SirDice (Jul 21, 2020)

No, you're still placing the blame on the wrong things.


----------



## zirias@ (Jul 21, 2020)

shkhln said:


> Quarterly receives whatever it receives, including updates; this is left entirely to maintainer discretion.


You should also have a look at the associated PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244212

This is a _security update_. These will be approved for merge of course. Quarterly will never receive any update, just because a maintainer thinks it would be a good idea. The rule is, as I said, security fixes and fixes for serious issues like broken builds.


----------



## shkhln (Jul 21, 2020)

Zirias said:


> This is a _security update_. These will be approved for merge of course. Quarterly will never receive any update, just because a maintainer thinks it would be a good idea. The rule is, as I said, security fixes and fixes for serious issues like broken builds.



Fine. Here is a straightforward update: https://github.com/freebsd/freebsd-ports/commit/9d952e13336c5c901bc61f89172415e18cd48474. One can always claim it improves stability anyway.


----------



## eldaemon (Jul 21, 2020)

Why are you worried about rolling release behavior when you're not even on 12.1? If you updated to 12.1 it sounds like this would not have been an issue.

Perhaps it's fixed in 11.4 as well, I am not sure.

I get that it's frustrating. I also use openscad. Pretty cool software.


----------



## shkhln (Jul 21, 2020)

shkhln said:


> Fine. Here is a straightforward update: https://github.com/freebsd/freebsd-ports/commit/9d952e13336c5c901bc61f89172415e18cd48474. One can always claim it improves stability anyway.



Ah, I see. That's pre-split. Something like https://github.com/freebsd/freebsd-ports/commit/89ac24d902611d08c5b8cb43a83235a6b50c32d2 then?


----------



## zirias@ (Jul 21, 2020)

shkhln said:


> Fine. Here is a straightforward update: https://github.com/freebsd/freebsd-ports/commit/9d952e13336c5c901bc61f89172415e18cd48474. One can always claim it improves stability anyway.


This commit was done on head, before the quarterly branch was created. Otherwise, it would have a commit message saying MFH and referencing the original commit to head


----------



## zirias@ (Jul 21, 2020)

shkhln said:


> Ah, I see. That's pre-split. Something like https://github.com/freebsd/freebsd-ports/commit/89ac24d902611d08c5b8cb43a83235a6b50c32d2 then?


This MFH has "Approved by:    portmgr (bugfix blanket)". This is for fixing a serious issue. Unfortunately, it doesn't tell what exactly the serious issue was. But it still stands, merges to quarterly are only for security fixes and fixes of serious issues. Well, there's a strange exception for browsers and related software. See also here:
https://www.freebsd.org/doc/en_US.I...rs-guide/ports.html#ports-qa-misc-request-mfh


----------



## 20-100-2fe (Jul 21, 2020)

Zirias said:


> the same will happen on a Linux system as well



No.
In 25 years using different Linux distributions, I've never seen such behaviors.
I've been using Void - a rolling release distribution - for 2+ years now and it never happened either.

Why do you think there are so many recurring complaints on this forum about pkg and repositories?
Such issues only happen on BSD OSes (not just FreeBSD).
The BSD way of handling packages and repositories is broken by design, that's all.
Reciting the same old mantras to every OP will not fix it.


----------



## shkhln (Jul 21, 2020)

Well, at least the version check is designed into pkg. This feature is also partially implemented, it just doesn't work properly for whatever reason.


----------



## SirDice (Jul 21, 2020)

20-100-2fe said:


> The BSD way of handling packages and repositories is broken by design, that's all.
> Reciting the same old mantras to every OP will not fix it.


Ok, so lets turn this around. What do you suggest should happen when a port fails to build and thus a package cannot be created? The port fails and there's no simple way to fix it for a particular legacy release version. The port works fine on newer releases. What do you suggest as a possible solution?


----------



## Mjölnir (Jul 21, 2020)

20-100-2fe said:


> [...] Why do you think there are so many recurring complaints on this forum about pkg and repositories?
> Such issues only happen on BSD OSes (not just FreeBSD).
> The BSD way of handling packages and repositories is broken by design, that's all. [...]


What are the differences that would show up to be relevant here? Or is it just the policy of when and which _"ports-bundles"_ are updated (together)?


----------



## gnath (Jul 21, 2020)

20-100-2fe said:


> The BSD way of handling packages and repositories is broken by design


No it is not broken , but how it works in this way. I have not much experince on Linux as a hole , but in Debian/Devuan the upgrades are pushed in to the repository when all the dependency chain as a hole is resolved. So there is no problem and also dist-upgrade for remaining unresolved issue. FreeBSD ports are modified individually and commited into the port tree. Any change in options or breakage in any build dependency makes the problem. Poudriere makes those incomplete build considering the hole head brunch. So _as and when "ports-bundles" or "total 30k+ ports" are ready then the repository may be updated. The second one require lot of coordinated maintainers. Or I can wait for my "ready-port" when available. 
My thought._


----------



## zirias@ (Jul 21, 2020)

20-100-2fe said:


> In 25 years using different Linux distributions, I've never seen such behaviors.
> I've been using Void - a rolling release distribution - for 2+ years now and it never happened either.


So, if you tell me that there are rolling release distributions that never experience package build errors, I simply don't buy it.
If you just tell me they handle it differently (by just keeping the latest successful build), that's a different story, and it's kind of "hiding" the problem. Whether that's better is probably open for debate -- it sure avoids the typical temporary unavailability of packages, but the cost could be subtle incompatibilities with "funny" results at runtime.


----------



## zirias@ (Jul 21, 2020)

gnath said:


> in Debian/Devuan the upgrades are pushed in to the repository when all the dependency chain as a hole is resolved.


There are a lot of independent package updates in Debian sid. There are coordinated port upgrades (for stuff like Qt, KDE, ...) in FreeBSD ports head. Not sure where you see the conceptual difference...


----------



## Mjölnir (Jul 21, 2020)

To go back to the genuine topic: One can `pkg help lock` vital packages and every 3 month or so update a jail or VM to see if such problem comes up.  Then wait to upgrade the main workstation/server.


----------



## tingo (Jul 21, 2020)

mjollnir said:


> To go back to the genuine topic: One can `pkg help lock` vital packages and every 3 month or so update a jail or VM to see if such problem comes up.  Then wait to upgrade the main workstation/server.


From a user point of view that is a workaround (band-aid) for a system which is not working.


----------



## jmos (Jul 21, 2020)

gnath said:


> I have not much experince on Linux as a hole , but in Debian/Devuan the upgrades are pushed in to the repository when all the dependency chain as a hole is resolved. So there is no problem and also dist-upgrade for remaining unresolved issue.


Debian knows stable, testing and sid ("unstable" / "still in development"). On sid or testing you can get unmet dependencies. In testing this happens seldom to the commonly used packages, but it happens. And stable has a freeze on packages, so there are months to solve those problems. And that's the price of it: Older software.


----------



## Mjölnir (Jul 21, 2020)

So it turn out to be a matter of policy.  The mantainers do the `pkg lock` for you.  That's not the philosophy of FreeBSD's handling of the ports collection.  If you want that, maybe one of the FreeBSD-based distributions like Ghost/Fury/MidnightBSD better fits your needs.  PC_BSD/TrueOS had sandboxed application bundles (different library versions can co-exist), you might be able to find the sources for that framework or maybe it's in the ports?


----------



## zirias@ (Jul 21, 2020)

As already said, the quick solution to avoid these problems is using _quarterly_. It's a sane default for using packages. It's not nearly as old as Debian stable, and most of the time, there's no attempt to "backport" upstream fixes, but *if* necessary, just import the new version. Plus when the branch is still new, build problems are more likely, so maybe just don't upgrade the first few days of every quarter. But apart from that, it should be at least as "stable" as Debian testing. Definitely a reasonable choice.


----------



## gnath (Jul 22, 2020)

I think problem is non-availability of some packages from repository. It is not in debian (all 3 branches). Devuan took almost one year to unchain tangled dependency due to one systemd. Even security update took some time.
qt5-network depends on three and required by three ports. OpenSSL is now missing. Now you settle the situation keeping OpenSSL and think similar situation as whole.


mjollnir said:


> As noted above by all others, it's a matter of the build process. When 150 of 30,000 ports fail to build, that's 0.5%





SirDice said:


> On the last build run 14 ports failed to build and 1318 ports were skipped (probably because a dependency failed).





SirDice said:


> What do you suggest as a possible solution?


No quick solution except clearing the tangle. What I understand that quarterly and latest at that day are build from same revision of head 'branch'.


----------



## 20-100-2fe (Jul 22, 2020)

SirDice said:


> Ok, so lets turn this around. What do you suggest should happen when a port fails to build and thus a package cannot be created? The port fails and there's no simple way to fix it for a particular legacy release version. The port works fine on newer releases. What do you suggest as a possible solution?



The possible solution I suggest is to use the same principles as those adopted everywhere else.
I don't mean just Linux package managers, but also maven, npm, composer...
There is a reason for all of them being built on the same principles.
Maybe this reason is that it's the best strategy we've found so far to handle dependencies and repository integrity, don't you think?
Or are all of them wrong?
Why should we deprive FreeBSD of the benefits of the learnings and findings of other developers?


----------



## zirias@ (Jul 22, 2020)

20-100-2fe You don't explain what *you* think the key difference would be (and how this is "better")? I don't see any deficiencies with dependency handling in pkg. The occassional problems people run into are ~90% using _latest_ and some packages are temporarily missing for build failures. So, asking again, is your suggestion to just keep the latest successful build of each individual packages? As also mentioned before, this has other drawbacks. If you see any drawbacks regarding _dependencies_, care to explain?


----------



## 20-100-2fe (Jul 22, 2020)

Zirias said:


> 20-100-2fe You don't explain what *you* think the key difference would be (and how this is "better")? I don't see any deficiencies with dependency handling in pkg. The occassional problems people run into are ~90% using _latest_ and some packages are temporarily missing for build failures. So, asking again, is your suggestion to just keep the latest successful build of each individual packages? As also mentioned before, this has other drawbacks. If you see any drawbacks regarding _dependencies_, care to explain?



The other package management systems keep all versions of the packages, so when a build fails (or is not finished) you don't have the same problems as with FreeBSD, i.e. not being able to install the package, or having pkg upgrade delete your installed package because it is no longer in the repository.

With the other systems, a package is only upgraded when all its dependencies are available in the repository.
As long as they aren't, you keep - or are able to install - the previous version.

Doing this for FreeBSD would probably not require much work : adding tests in pkg and keeping previous versions of the packages in the repository.
And it would also trivially solve the issues we have with drm-kmod at each release.


----------



## Mjölnir (Jul 22, 2020)

The drawback would be that a single or few ports can delay the whole progress by several month, s/t even a year.  In the meantime, the maintainers have to provide at least security fixes for the versions in the repository, while also working to fix the new version's dependencies.  Who's going to do that additional work?


----------



## zirias@ (Jul 22, 2020)

20-100-2fe said:


> The other package management systems keep all versions of the packages, so when a build fails (or is not finished) you don't have the same problems as with FreeBSD, i.e. not being able to install the package, or having pkg upgrade delete your installed package because it is no longer in the repository.


That's possible, but has other drawbacks -- it can lead to "funny" bugs at runtime. Still, you could argue about what's worse for the end user.


20-100-2fe said:


> With the other systems, a package is only upgraded when all its dependencies are available in the repository.


"The other systems" also have some form of package builds, and for this "source", having all the dependencies upfront is the only way this could ever work -- no different with FreeBSD ports. Of course, in a binary repository, you should also only put packages when all their dependencies are available as well -- which is btw exactly what poudriere is doing. I get the feeling you are confusing something here, it's all coming back to the question how you deal with unexpected build fails. Poudriere just doesn't add that package to the repository, and as a consequence, all dependent packages aren't added either. The only alternative would be to keep the latest successful build, but see above ...

Build fails happen, and of course, they happen for e.g. Debian sid as well. Probably, Debian takes the approach of just keeping the previously built packages in their binary repository. But that's trading the problem of temporarily unavailable packages for the problem of unexpected runtime instability.

If you want a more stable set of packages, FreeBSD offers that in the quarterly repository built from the current quarterly branch.


----------

