# Teach me package management



## dpx (Aug 13, 2017)

What am I doing wrong since this doesn't make sense?

I have almost all packages installed with pkg as binary. I have very few packages installed from ports - vim without all gtk dependencies but with everything else and nvidia without linux dependencies.

I am using pkg upgrade to well upgrade binary packages.


```
# pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking for upgrades (4 candidates): 100%
Processing candidates (4 candidates): 100%
The following 5 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   linux-c6-xorg-libs: 7.4_9
   linux-c6-fontconfig: 2.8.0_3
   linux-c6-expat: 2.0.1_5
   linux_base-c6: 6.9

Installed packages to be REINSTALLED:
   nvidia-driver-340-340.102 (options changed)

Number of packages to be installed: 4
Number of packages to be reinstalled: 1

The process will require 296 MiB more space.
68 MiB to be downloaded.

Proceed with this action? [y/N]:
```

Both me and FreeBSD know that I installed nvidia from ports. Even versions match so the only difference is (options changed) part. No I don't want linux_* thank you.

Why does pkg upgrade try to mess with package installed from ports? I need pkg upgrade to well upgrade everything I installed as binary packages, as needed.

What am I missing? I don't appreciate pkg upgrade trying to upgrade nvidia, since it obviously makes no sense to do that. What when some other packages need to be upgraded too? Is the only option to install again one by one and avoid pkg upgrade that way?

Even if sources and ways of building differ, binary packages and ports work well together. I bet it is very common to have 99% installed as binary and several specific packages build. It makes no sense to have to either stick with the ports or binaries completely, I don't want't to build xorg or firefox from source, than you.

So please teach me some package management, I may be doing something wrong here.


----------



## dpx (Aug 13, 2017)

What's also strange is that pkg upgrade offers to install binary nvidia but I have also built vim which is 'fine' by pkg? Is that because of linux_* libraries that binary nvidia requires?

I am trying to understand userland packaging logic since it looks like I am doing something wrong or it needs some love before it gets better.


----------



## getopt (Aug 13, 2017)

dpx said:


> Teach me package management: What am I doing wrong since this doesn't make sense?



Start reading FreeBSD Handbook 4. Installing Applications: Packages and Ports. If you did read it already do it again. Get known to the Pkg subcommands you use like `pkg help upgrade`.



dpx said:


> I have almost all packages installed with pkg as binary. I have very few packages installed from ports



It has been discussed en length here on the forums, as a lot of people run into problems mixing ports and packages. The distillate from this is that packages on the FreeBSD-repositories are built with default port options. Therefore you may mix ports and packages as long as you stay with the default options when building ports.

Now if you start using custom port options things might get really complex in some cases because of (chaines of) the dependencies. While trying to use `pkg lock` it may help to tell Pkg not touching some custom ports, but do not expect that it works in every case. 

So general advice to newbies is: Do NOT mix ports and packages unless you have to do it for a good reason. 

Also note that there is editors/vim-lite which is built for console use without the gtk stuff.


----------



## dpx (Aug 13, 2017)

Thanks getopt for your help.



getopt said:


> Start reading FreeBSD Handbook 4. Installing Applications: Packages and Ports. If you did read it already do it again. Get known to the Pkg subcommands you use like `pkg help upgrade`.



I did and it is not completely clear. For example, when it comes to using pkg to upgrade, it should either upgrade binary packages only or binary + ports that I built. It should *not* offer to replace port that I built with stock binary package especially if it is obvious that I used different build options. I have 25+ years of professional programming experience, with every language imaginable, so I should be able to understand the logic quickly. Sooner I do sooner I would be able to contribute.



getopt said:


> It has been discussed en length here on the forums, as a lot of people run into problems mixing ports and packages. The distillate from this is that packages on the FreeBSD-repositories are built with default port options. Therefore you may mix ports and packages unless you stay with the default options when building ports.



Can you please clarify this? I did built nvidia with different options than what binary package offers (no linux, acpi only). So I did stay away from the default options (if that's what you suggest). If this is expected behavior I don't understand it. If it is not let's fix pkg -- please let me know how to contribute to its development.



getopt said:


> Now if you start using custom port options things might get really complex in some cases because of (chaines of) the dependencies. While trying to use `pkg lock` it may help to tell Pkg not touching some custom ports, but do not expect that it works in every case.



Pkg lock is interesting but it looks like really ugly fix for something that should be fixed elsewhere. I'll try using it but it doesn't seem like clean or long term way I'd love to use.

I have examined some chains of the dependencies and I don't really understand why so much of them are version-hardcoded. I suspect that causes at least half of all the mixing problems. I understand that you want to limit lowest version number, but insisting on 'something 2.24.1' and not accepting either 2.24.2 or 2.24.0 seems wrong.

If there is any proper way to discuss this (forum is fine but I can switch to mailing list if that's more appropriate) please let me know. From my short experience with FreeBSD, it is amazing system that needs some work with packaging in general. Not even much work, just better way to fit things together. I am open to jump in and help with code or discuss stuff if there is a way to do so.


----------



## dpx (Aug 13, 2017)

Example:
I have installed xmobar as binary package but it didn't offer xft fonts so I tried switching to ports.
I have ghc-7.10.2_3 (haskell compiler) installed as binary package, because I installed xmonad and xmonad-contrib as packages.
x11/hs-xmobar/Makefile incudes ../../lang/ghc/bsd.cabal.mk which in turn requires building ghc-8.0.2. Then ghc pulls in gcc (I think) and so on.

There is absolutely no need for this kind of mess. xmobar definitely doesn't care if I use ghc 7.x or ghc 8.x. I ended building xmobar using whatever gcc I have installed (7.10.2) with xft support in short 10 seconds. Pulling in dependencies with *specific* version number causes ton of problems, both in number of packages that have to be built and in building time. It is sure way to cause LONG chains of dependencies and ton of recompiling.

Same happened with rxvt (package vs ports) with perl version dependency. Some binary package pulled in perl5-5.24.1_1 as a dependency. Then ports x11/rxvt requests whatever latest perl version is in the ports, so it has to build it. There is absolutely no need for this.

Having hs-xmobar depend on the specific 8.0.2 version is just wrong. Same is all over the ports. We can symlink stuff and don't care at least about minor version. There is absolutely no reason why binary packages and ports can't mix well unless there are hard coded versions used.


----------



## getopt (Aug 13, 2017)

dpx said:


> Pkg lock is interesting but it looks like really ugly fix


True. But mixing ports and packages is ugly too. If you need custom options for more than 1 box consider building all on ports-mgmt/poudriere in bulk and have your own local repository. Then use Pkg for deploying your custom or default built packages in any combination you need. 

In contrary to ports-mgmt/portmaster Poudriere builds in a clean environment not affecting the life system on which you build. That is what I do and I'm happy with that.


----------



## dpx (Aug 13, 2017)

Thanks. I am thinking about reducing number of things I have to build, not increase. Poudriere looks interesting as a last resort option but I'd rather avoid more caching and more building if I can.



getopt said:


> True. But mixing ports and packages is ugly too.



Why? That's what I can't understand yet. It doesn't have to be ugly. For at least 99% of dependencies you don't really care what minor version number is, if it is 4.2 or 4.1, often even you don't care if it is 7.x or 8.x (see my xmobar post above). For a very limited set of packages you would need to limit lowest version number but rarely or never maximal version number.


----------



## forquare (Aug 13, 2017)

dpx said:


> Why? That's what I can't understand yet. It doesn't have to be ugly. For at least 99% of dependencies you don't really care what minor version number is, if it is 4.2 or 4.1, often even you don't care if it is 7.x or 8.x (see my xmobar post above). For a very limited set of packages you would need to limit lowest version number but rarely or never maximal version number.



If you’ve built a package and linked against $dependancy 7.x, if you upgrade $dependancy to 8.x will your package break without rebuilding/relinking?

Ports are just Packages you have to compile, they aren’t really separate things.


----------



## obsigna (Aug 13, 2017)

dpx said:


> Thanks. I am thinking about reducing number of things I have to build, not increase. Poudriere looks interesting as a last resort option but I'd rather avoid more caching and more building if I can.
> 
> 
> 
> ...



Beauty lies in the eye of the beholder. I won't submit the following solution to a beauty contest, however, it has the big advantage that it works, which by its own weighs out any opinions about aesthetics.

```
#!/bin/sh

### the list of the ports that shall be updated from sources
### in the future include 'devel/pecl-pthreads' before 'textproc/pecl-yaml'
portslist="\
devel/apr1 \
devel/subversion \
lang/php71 \
mail/postfix \
mail/roundcube \
net/mDNSResponder \
net/netatalk3 \
www/mod_php71 \
www/squid \
\
textproc/pecl-yaml \
archivers/php71-phar \
archivers/php71-zip \
archivers/php71-zlib \
converters/php71-iconv \
converters/php71-mbstring \
databases/php71-pdo \
databases/php71-pdo_pgsql \
databases/php71-pgsql \
databases/php71-sqlite3 \
devel/php71-intl \
devel/php71-json \
ftp/php71-curl \
graphics/php71-exif \
math/php71-bcmath \
net/php71-sockets \
security/php71-filter \
security/php71-hash \
security/php71-openssl \
sysutils/php71-fileinfo \
textproc/php71-dom \
textproc/php71-xml \
www/php71-session"

### fetching updates of the FreeBSD ports tree
/usr/bin/printf "\nFetching updates of the FreeBSD ports tree...\n"
/usr/sbin/portsnap fetch update
/usr/sbin/pkg version -v
/usr/sbin/pkg updating -d `date -v-2w +%Y%m%d`

### ask and in case of y|Y run the updating processes
/usr/bin/printf "\nDo you want to continue (y/n)? "
save_stty_state=$(stty -g); stty raw -echo; answer=$(head -c 1); stty $save_stty_state
if echo "$answer" | grep -iq "^y" ; then
   /usr/bin/printf "\n\n"
   /usr/sbin/pkg update

   portmake=""
   pkgslist=`/usr/sbin/pkg version -o | /usr/bin/cut -f1 -w`
   for port in $portslist ; do
      for pkg in $pkgslist ; do
         if [ "$pkg" == "$port" ] ; then
            continue 2
         fi
      done

      portmake="$portmake $port"
   done

   pkgslist=""
   outdated=`/usr/sbin/pkg version -ol\< | /usr/bin/cut -f1 -w`
   for outd in $outdated ; do
      for port in $portslist ; do
         if [ "$port" == "$outd" ] ; then
            portmake="$portmake $port"
            continue 2
         fi
      done

      pkgslist="$pkgslist $outd" 
   done

   pkgupgrd=""
   outdated=`/usr/sbin/pkg version -RU -ol\< | /usr/bin/cut -f1 -w`
   for outd in $outdated ; do
      for pkg in $pkgslist ; do
         if [ "$pkg" == "$outd" ] ; then
            pkgupgrd="$pkgupgrd $pkg"
            continue 2
         fi
      done
   done

   /usr/bin/printf "\nUpdating binary packages...\n"
   if [ "$pkgupgrd" != "" ] ; then
      /usr/sbin/pkg upgrade -U $pkgupgrd
   else
      echo "All installed packages are up-to-date."
   fi

   /usr/bin/printf "\nUpdating ports...\n"
   if [ "$portmake" != "" ] ; then
      cwd=$PWD

      for port in $portmake ; do
         cd /usr/ports/$port
         /usr/bin/make clean deinstall
         /usr/bin/make install clean
      done

      cd "$cwd"
   else
      echo "All installed ports are up-to-date."
   fi

   /usr/bin/printf "\nCleaning up...\n"
   /usr/sbin/pkg clean -y

else

   /usr/bin/printf "\n\n"

fi
```
The above script does what pkg(8) is not able to do, namely, ignore some ports of a software installation in the pkg upgrade process.

At the head of the script we need to inform the ports which must not be upgraded from the binary repository but shall be build from the ports tree -- of course the list of ports $portslist needs to be adapted for other machines, and maintained whenever a new port with custom options is installed on the system.

In the first stage it fetches the latest updates of the FreeBSD ports tree by the way of portsnap(8), and it shows a list of all installed software with indication of the version status. It will also show the entries in /usr/ports/UPDATING related to the installed software of the last two weeks. Depending on that information the user might decide to stop or to proceed.

In case of proceed it updates the local copy of the repository catalogue(s). You want to let pkg use the latest repository, by default it works against the quaterly repository which is with a higher probability out of synch with the ports tree.

Now the script compiles 3 lists, the list of all outdated software and more 2 divisions, namely outdated packages to be upgraded $pkgupgrd and outdated ports to be made $portsmake. BTW, pkg lock does not work as advertised.

Finally it calls the command pkg upgrade -U $pkgupgrd which takes care only for the outdated packages, and then it loops through the list of the outdated ports, stepping into each of the respective port's directory and calling the usual make deinstall/make install combo.

This script works now on 6 FreeBSD servers very well. All well and good, what about the risks? If you are not satisfied, then continue reading. ...



... In the case of deviations from the common feature set, people recommend to built everything on the local machine from the port's sources, and those people continue telling you that it is too risky to intermix pre-built packages with your customized port-builds.

In a certain sense, this is even correct, namely, without a serious risk assessment the best educated guess of the experts is always that many things may go wrong. And of course the conclusion from this is that Dual Mode updating of ports, i.e. mixed installation from locally built and pre-built packages is too risky for even trying it.

*Risk assessment and Risk separation*

A package with the common set of options can be created locally from the corresponding port with the make package command and by leaving the default options in the configuration dialog in place. The thus locally created package would be exactly identical to the corresponding package of the same version in the public repository of pre-built packages.

This identity of locally built vs. pre-built packages is one of the essential basis for the further discussion. The other basis is that the installation phase is agnostic to the origin of the package -- a package is a package is a package is ...

Any procedure of Dual Mode installation and updating of ports must stay on these basis, and given this constraints the risks can be named and classified into 2 groups:

1. First installation of a port/package

It may happen, that a selection of ports and options may conflict with installed packages. These conflicts must be resolved beforehand utilizing above script, by calling pkg delete on the infringing package and adding it to the list of ports at the head of the script.


2. Updating of the installed ports/packages collections

Ususally (amd64) the ports tree is ahead to the latest package repository by 2 to 3 days. In rare cases, it happens that a custom port depends on a new version of pre-built package which didn't made it yet into the remote repository. In this case we may wait for more 2 or 3 days and start the script again, or we manually run a local onetime build of the dependency before executing the software update script.

Special occasions, which usually show-up in the /usr/local/UPDATING file, need special actions before we can let the script do its automatic job. For example the recent switch from mail/dovecot2 to mail/dovecot needed to be done outside of the normal updating sequence.

The command pkg upgrade shows a list of actions to be done, and it asks whether to proceed [y/n]. If pkg wants to reinstall some of your custom ports because of changed options, then we need to tell n(o) here, because this means, that package dependencies have changed compared to our first installation. In order to reolve this conflict, we would need to customize the package, i.e. call pkg delete on it and put it into the list of the ports at the head of our script.


----------



## dpx (Aug 13, 2017)

forquare said:


> If you’ve built a package and linked against $dependancy 7.x, if you upgrade $dependancy to 8.x will your package break without rebuilding/relinking?



In most cases major version number will make the difference, minor will not. Not to mention unfortunate xmobar example that demands specific compiler version, up to the third version sequence identifier -- which simply makes no sense.

Haskell does/did offer dynamic and static linking, I think they have switched to dynamic only. If it is dynamic build it makes sense to be careful with version numbers but it won't break at third sequence identifier x.x.this. Most of the compilers may differ with major version number (so Makefile can demand <=8.x) but hard linking package with the full compiler version won't achieve anything.

Systemd distros have fixed this problem long ago, they keep major version symlink and all the packages compile using symlink. If my package needs Python 3.x it usually won't matter if I have 3.1 or 3.2. Even if it does require 3.1 and won't work with 3.0 it will surely work with 3.2 too. My makefile should demand 'at least 3.1' and be at ease with it.

If we have Python2 and Python3 packages it makes no sense that any particular port requires exactly Python 3.6.2. and refuses to build with Python 3.6.3. Then because it does require Python 3.6.3 we go to great length to compile and provide that exact version. That's nonsense. What FreeBSD needs instead is:

Both binary packages and ports create major version symlinks for other packages to use.
For all the ports, require minimal version that makes sense, in xmobar case that would be ghc 7.x or even 6.x. *Not* 8.0.2. and nothing but that exact version.

Anything that builds from ports follows symlinks and doesn't care what exact third version sequence identifier is. Not unless it breaks package and even then only by limiting lowest version that works. This would make packages and ports equal and play well with each other.
By doing this we don't care about exact minor versions, ports build using "what's already there", so we don't compile everything in the tree to build the package. I believe it can be done without much effort, if packages care about "I am the latest one installed" instead of "I am exactly 3.6.2 and I stay in my own folder".

Most importantly, small modification of Makefiles will make binary packages and ports play well with each other.

If there are any obstacles in doing this please let me know what they are. In this case it doesn't make sense to modify/submit one by one package -- it requires community discussion, decision and effort.



forquare said:


> Ports are just Packages you have to compile, they aren’t really separate things.



Of course they are, that's why I try to understand FreeBSD userland building process and see if we can make it better. Ports are great, their hard coded dependencies are not.

FreeBSD is so far mindblowing for me as a long time 'nix user. Packages and ports on the other hand need to be fixed before they are truly usable.


----------



## dpx (Aug 13, 2017)

I am seriously suggesting this for a consideration, it is invented many times over:
https://en.wikipedia.org/wiki/Dependency_hell

Especially this part:


> Version numbering
> A very common solution to this problem is to have a standardised numbering system, wherein software uses a specific number for each version (aka _major version_), and also a subnumber for each revision (aka _minor version_), e.g.: *10*.1, or 5.*7*. The major version only changes when programs that used that version will no longer be compatible. The minor version might change with even a simple revision that does not prevent other software from working with it. *In cases like this, software packages can then simply request a component that has a particular major version, and any minor version (greater than or equal to a particular minor version). As such, they will continue to work, and dependencies will be resolved successfully, even if the minor version changes.*


Also:


> Some package managers can perform smart upgrades, in which interdependent software components are upgraded at the same time, thereby resolving the major number incompatibility issue too.
> Many current Linux distributions have also implemented repository-based package management systems to try to solve the dependency problem. These systems are a layer on top of the RPM, dpkg, or other packaging systems that are designed to automatically resolve dependencies by searching in predefined software repositories. Examples of these systems include Apt, Yum, Urpmi, ZYpp, Portage, Pacman and others.


----------



## dpx (Aug 13, 2017)

getopt said:


> Unfortunately it does make sense if it is known that 3.6.3 would break building for whatever reason. Although the restriction may be a temporary one.



If that ever happens it can only be very rare edge case. How many real world cases are there, that you know of? I can't think of any although I am aware it may _rarely_ happen.

Edge cases that are so rare that they may never happen should not prevent ports from being more flexible.

Edit: or alternatively makefiles should have min, max, skip versions. Min would be 3.0, max would be 3.9999, skip may be array of versions that break building.


----------



## dpx (Aug 13, 2017)

Also we are talking about two software sources only, binary packages and ports. I am sure if 3.6.3 breaks anything it will be skipped in both sources, right?


----------



## drhowarddrfine (Aug 13, 2017)

dpx said:


> Also we are talking about two software sources only, binary packages and ports. I am sure if 3.6.3 breaks anything it will be skipped in both sources, right?


I have not read through this thread, I'm just a poet and don't know it,  so I apologize if this is out of context but, to be clear, packages are pre-compiled ports. So version X of a package is equal to version X of a port with all defaults set.


----------



## dpx (Aug 14, 2017)

drhowarddrfine said:


> I have not read through this thread, I'm just a poet and don't know it,  so I apologize if this is out of context but, to be clear, packages are pre-compiled ports. So version X of a package is equal to version X of a port with all defaults set.



They are the same in theory but being versions apart they can't communicate well. They could talk with small amount of effort and some amount of good will.

Tens of posts down the road and I still haven't heard or don't understand:


Why pkg wants to install binary nvidia package if it knows I have built the package from the ports? Bug or logic I don't understand?
Why ports require exact version numbers of all the dependencies?
Again if there is better place to discuss or at least get this answers please let me know. So far I have only seen pretty much circular reasoning that leads nowhere. Thank you.


----------



## dpx (Aug 14, 2017)

Just as an example, here is ArchLinux makefile for xmobar-git. I have picked that example because it downloads source code and compiles everything for that package -- equivalent of x11/hs-xmobar port on FreeBSD.



> # Maintainer: Vlad M. <vlad@archlinux.net>
> # Contributer: euclio
> # Contributer: Cedric Girard <girard.cedric@gmail.com>
> # Contributer: Colin Woodbury <colingw@gmail.com>
> ...



There is *not a single version dependency*, in all three dependency sections (makedepends, depends, optdepends). So it compiles with any ghc as it should.

runhaskell script only invokes latest whatever haskell installed.

We can do the same for FreeBSD ports. There are no downsides except for some effort to get there.


----------



## Deleted member 30996 (Aug 14, 2017)

dpx said:


> Tens of posts down the road and I still haven't heard or don't understand:
> 
> 
> Why pkg wants to install binary nvidia package if it knows I have built the package from the ports? Bug or logic I don't understand?
> ...So far I have only seen pretty much circular reasoning that leads nowhere. Thank you.



You're issuing a command for pkg to install/upgrade a package and don't know why it wants to install a package?



drhowarddrfine said:


> ...to be clear, packages are pre-compiled ports.


----------



## dpx (Aug 14, 2017)

Trihexagonal said:


> You're issuing a command for pkg to install/upgrade and don't know why it wants to install a package?



Yes, it makes no sense.



> # pkg info nvidia-driver-340
> nvidia-driver-340-340.102
> Name           : nvidia-driver-340
> Version        : 340.102
> ...





> # pkg upgrade
> Updating FreeBSD repository catalogue...
> FreeBSD repository is up to date.
> All repositories are up to date.
> ...



Pkg upgrade knows I have changed options, see parts in bold, therefore it can safely assume I have installed from ports if there is no such direct info in DB. Pkg can offer to rebuild the port I used (I don't expect this and I know it would be complicated) or to ignore packages with "(optons changed)" that were obviously not installed as binary packages. What it shouldn't do under any circumstances is to demand all-or-nothing upgrade that overwrites all port installations. I wouldn't say a word if pkg has command to skip any options-changed package.

The way pkg and the way ports are structured, they offer all or nothing install. There is no reason for this, they could mix well and its not outlandish from user to expect such behavior. It is not like there are really big obstacles for mixing them well.

So in order to pkg upgrade sudo, sqlite3, firefox and curl I have to upgrade nvidia which was clearly not part of the pkg based installation? Edit: and 'upgrade' it offers is not even an upgrade, versions are the same, the only 'reason' for upgrade is because I haven't used defaults options and binary package.

This is not rocket science and yet I am not sure if what I am saying gets relayed that way. If I should clarify anything please let me know.


----------



## Deleted member 30996 (Aug 14, 2017)

dpx said:


> Yes, it makes no sense.



With all due respect, it makes perfect sense.

That is the function of pkg. To install packages.


There is a vulnerability in graphics/OpenEXR, which is included in the build of graphics/gimp.

If you try to build graphics/gimp with ports-mgmt/portmaster it will stop the build at graphics/OpenEXR due to the vulnerability and you would have to issue `cd /usr/ports/graphics/gimp && make IGNORE_VULNERABILITIES=yes clean install` to continue the build.

However, if you use `pkg install gimp` it will install it along with graphics/OpenEXR with no regard to the vulnerability.

Then you can `cd /usr/ports/graphics/OpenEXR && make deinstall clean` and graphics/gimp will still run without it.

And you thought you were confused before...


----------



## dpx (Aug 14, 2017)

As I said before I have 25+ years of programming under the belt. If I am confused that's because something doesn't check with logic.



Trihexagonal said:


> That is the function of pkg. To install packages.



Respect & all but this is circular reasoning. Of course it does install packages. Grass is green because it is green. Let me rephrase:

- Why does pkg touch something that's clearly not installed by pkg?
- Why there is no switch in pkg to say 'upgrade everything'/'upgrade pkg installed packages only'?

Lack of this simple switch denies lots of flexibility that would be there if there is no lack of such switch. <- no intention to be difficult or go in circles but I can't be more clear.

The only reason, from the pkg/binary packages side of the story, for binary packages and ports not mixing well, is that pkg tries to overtake what's not its job to do, and there is no switch to prevent such behavior.

The only reason, from the ports side of the story, for binary packages and ports not mixing well, is that ports demand specific dependency version number.

Both can only be defended by 'things are that way because things are that way'. I am linux refugee that recognized how much better FreeBSD core is. Userland is far less quality and the reason I keep ask questions and try to understand is because I intend to keep using FreeBSD. It all can be fixed unless I hit the wall of circular reasoning.

Does anybody understand what I am saying and what the problems are?


----------



## forquare (Aug 14, 2017)

dpx said:


> Pkg upgrade knows I have changed options, see parts in bold, therefore it can safely assume I have installed from ports if there is no such direct info in DB.



I _think_ that is a slightly wrong assertion. ports-mgmt/pkg knows that the options between the package installed and the package in the repository differs.
To assume that means said package must have been compiled by Ports assumes that the options haven’t been changed by the Port maintainer (I’m not 100% sure when such a change would require a “_x” version bump, and it could be forgotten).

I think you might find better, more complete answers on
freebsd-pkg@freebsd.org
It is more likely that they will come across a developer who can explain the reasoning to you, or tell you where to jump in to help, or perhaps be inspired by your answers.


----------



## dpx (Aug 14, 2017)

forquare, thanks for the proper address to ask. I have sent email already.

You are right we can't be 100% sure where nvidia came from, unless that info is stored somewhere, but we can be 100% sure it wasn't pkg installed package.

It is not only nvidia, let's assume I am happy with 100 binary packages but I need to customize five of them. So I go to ports, make config and make install them. It all means that I put some effort into picking options and compiling. Pkg knows it is not its product '(options changed)' but it tries to overwrite my hard customization work. I can't see logic in doing that.

Edit: in such rare case where ports maintainer changes flags, I am sure I can deinstall package once then reinstall new one. I don't expect that to happen often and even if it does rest of the upgrade cycle would behave as expected.


----------



## SirDice (Aug 14, 2017)

dpx, packages are nothing more than pre-compiled ports. As a matter of fact, ports build packages and it's those packages that get installed. So, from the system's point of view, there is no difference between a port and a package once they are installed. That's where your confusion lies, you assume ports and packages are different when they are not. And that is why a pkg-upgrade(8) will upgrade everything, including the stuff you built from ports.


----------



## dpx (Aug 14, 2017)

SirDice, thanks. I understand the mechanism.

When you customize 'make config' and change options that port is definitely not the same as stock pre-compiled port. I understand the logic but it makes no sense -- if I modify package settings I would probably want package to stay that way, not to be overwritten with newer/same version stock package. Because settings definitely differ, right?

If I upgrade package with changed config, I would either like to keep it manually upgraded or not touched at all. Either way it makes sense that I want my config to stay, right?

It is 'make config' & 'make install' not 'make config-until-next-pkg-update'. Overwriting manually customized ports with stock packages just makes no sense.


----------



## dpx (Aug 14, 2017)

> #pkg info | wc -l
> 606





> # pkg query -e '%a = 0' "%n;%v;%sh;%R;%sb" | grep -i unknown-repository | sort --field-separator=';' -g -k5 | cut -f 1,2,3,4 -d';' | column -s';' -t
> dialog4ports       0.1.6     24.3KiB  unknown-repository
> pkg                1.10.1    11.4MiB  unknown-repository
> vim                8.0.0867  29.2MiB  unknown-repository
> nvidia-driver-340  340.102   121MiB   unknown-repository



So I have 606 packages in total, two of them (vim and nvidia) installed from ports. Since pkg upgrade can't at least skip those installed from ports all the suggestions so far is to either stick with the binary packages or with the ports. Since I want my vim customized I would have to compile other 602 (604 minus dialog4ports and pkg) packages too? That's plain silly, there is no need for that if pkg knows not to touch those two I need.

Also unknown-repository should be a pretty good indication for pkg not to touch that package?


----------



## drhowarddrfine (Aug 14, 2017)

dpx said:


> if I modify package settings I would probably want package to stay that way, not to be overwritten with newer/same version stock package.


You can't modify packages. You can only change settings if you use ports. Now I'm confused by what you're asking. You don't use `pkg upgrade` to upgrade ports.


----------



## SirDice (Aug 14, 2017)

dpx said:


> When you customize 'make config' and change options that port is definitely not the same as stock pre-compiled port.


Yes, but it's the same package you end up with. Although the package that was created from that port will have different options set. This is the main reason why mixing ports and packages is generally a bad idea. To take the example of nvidia-driver, it's still the nvidia-driver package that going to be installed, regardless of the options you enabled/disabled.



dpx said:


> if I modify package settings I would probably want package to stay that way, not to be overwritten with newer/same version stock package. Because settings definitely differ, right?


You can't change package settings, once a package is built its options and dependencies are 'set in stone'.  So for the nvidia-driver all it sees is a nvidia-driver package that's installed and will therefor try to "upgrade" that package regardless if you built it from ports or not.


----------



## dpx (Aug 14, 2017)

It said 'package settings'. Not 'packages.' Meaning 'make config' then nice dialog pops up, then I pick options I need. Then 'make install'. Settings.

Package config modified this way definitely differs from stock binary package, right?

Please read the thread. I don't use pkg upgrade to upgrade ports but that's what pkg upgrade insists on.


----------



## SirDice (Aug 14, 2017)

dpx said:


> It said 'package settings'. Not 'packages.' Meaning 'make config' then nice dialog pops up, then I pick options I need. Then 'make install'. Settings.


Those are port settings. Package settings/options cannot be changed.



dpx said:


> I don't use pkg upgrade to upgrade ports but that's what pkg upgrade insists on.


Yes, and I'm trying to tell you there is no difference between a port and a package once they are installed, from the system's point of view they are all packages.


----------



## dpx (Aug 14, 2017)

SirDice, are you a programmer in any form? It seems this is semantics discussion while it should be very obvious that my vim built from ports (with python and without gtk2/gtk3) differs as a result from stock package. They only share name and version but pkg query very well knows it is installed from 'unknown-repository'. So there are no logical/code obstacles to add switch to pkg that handles both variants well.

Reason I keep asking the same question is that fixing it would take five minutes, it offers more flexibility and it doesn't break anything. I am bit puzzled that simple logical stream of deduction can't get trough but I don't mind asking again if we can have better OS as a result.


----------



## dpx (Aug 14, 2017)

SirDice said:


> Those are port settings. Package settings/options cannot be changed.



Ah semantics. Ok. So customizing port setting makes resulting package (since that's how pkg sees it, right?) different than what stock package offers.



SirDice said:


> Yes, and I'm trying to tell you there is no difference between a port and a package once they are installed, from the system's point of view they are all packages.



That's plain not true. Their functionality and their repository are different. Even pkg knows this. I wonder if pkg@ guys can make some sense into this. I have sent email.


----------



## SirDice (Aug 14, 2017)

dpx said:


> Reason I keep asking the same question is that fixing it would take five minutes, it offers more flexibility and it doesn't break anything.


In that case I suggest you submit patches for pkg(8): https://github.com/freebsd/pkg

I really do understand what you are trying to say but the functionality simply isn't there.


----------



## tobik@ (Aug 14, 2017)

> Why does pkg touch something that's clearly not installed by
> pkg?


Ok, let me first repeat what others have said in this thread already:
ports are installed by pkg(8).  It's the underlying mechanism to
install them and installed ports are recorded into the package
database.  There is no conceptual distinction between a package
installed from ports or a package installed from the FreeBSD package
repository.



dpx said:


> Also unknown-repository should be a pretty good indication for pkg not to touch that package?


It would say that changing pkg's behavior on the basis of the
repository name (or lack thereof) of the _currently installed package_(!) would be
insane and a POLA violation.



> So in order to pkg upgrade sudo, sqlite3, firefox and curl I
> have to upgrade nvidia which was clearly not part of the pkg based
> installation?



pkg(8) will do its best to upgrade your system with the
available information it has.  It handles package upgrades on the
basis of the configured package repositories.  Since you clearly have
only setup the FreeBSD repository (or are simply using the defaults)
it will only use it to do so.  A package's options can change in the
repository and pkg(8) naturally wants to follow that change
and wants to reinstall the package.

I bet the solution to your problem would be to setup your own package
repository and point pkg(8) to it with a higher priority than the FreeBSD one.

`make package` will create a package of a port and put it
into /usr/ports/packages (only if the directory exists,
so create it first).  `pkg repo /usr/port/packages` will then
create a repository from them.  And to make pkg aware of it, create
/usr/local/etc/pkg/repos/local.conf with 
	
	



```
local: {
	url: "file:///usr/ports/packages",
	priority: 10,
	enabled: yes
}
```
Now reinstall the desired packages from the local repository to make
them stick to it during upgrades (see pkg-repository(8)):
`pkg install -f -r local nvidia-driver-340 vim`

It's likely that you will end up in dependency hell with this at some
point though if you don't take care to keep the local package versions
in sync with the FreeBSD repository.


----------



## jb_fvwm2 (Aug 14, 2017)

I concur with dpx's questions, and am facing a 'do not upgrade ANYTHING' without triple workaround hassles, [ though my fault for using CURRENT and too many ports... ] and terse messages from pkg as to why operations on a subset of packages want an all-or-nothing approach. 
  From years of back and forth, I think it is a lack of coders, as well as not too many persons in the same boat as dpx and I.
  Thankfully, this is FreeBSD where it should all work out in the end.
/edit/
  As I typed this post, the well-thought out guide right above it appeared.  If I had read it before posting this, I might be well on my way to such an implementation.  
/end edit/


----------



## dpx (Aug 14, 2017)

tobik@ said:


> It would say that changing pkg's behavior on the basis of the
> repository name of the _currently installed package_(!) would be
> insane and a POLA violation.



Thanks tobik@. I was about to clone pkg and send small PR that would make pkg much more user friendly without sacrificing any security. Are you saying that's not acceptable in any form? I have no intention to keep local patched pkg version only for myself so if it can be accepted OS wide I won't do it.

Here is what I had in mind:

Keep pkg and pkg upgrade working as is
Add switch -d <repo name> which means "don't touch packages installed from this repository". Maybe allow multiple -d too.
So for everybody not interested they can keep using pkg upgrade as usual, nothing is changed. For those like me who want 602 binary packages and only two or so built from ports, I could use 'pkg upgrade -d unknown-repository' and have all the functionality I need.

If it is futile effort and can't be accepted please let me know.



tobik@ said:


> ...



That's helluva path to follow, even for me. Not in terms of if I know how to do it but in terms of time/effort needed. I understand the logic but man the overhead.

So if it is official policy that proposed change can't be accepted please let me know.


----------



## tobik@ (Aug 14, 2017)

dpx said:


> If it is futile effort and can't be accepted please let me know.


No, by all means submit it. I do not speak for the pkg developers. The post is my own personal opinion only.



dpx said:


> That's helluva path to follow, even for me. Not in terms of if I know how to do it but in terms of time/effort needed. I understand the logic but man the overhead.


Hmm, it takes like 10 minutes to setup


----------



## dpx (Aug 14, 2017)

tobik@ said:


> No, by all means submit it. I do not speak for the pkg developers. The post is my own personal opinion only.



Thanks will do. It should be small patch anyway so if I intend to stay here better test the waters with something tiny. I'll have PR in a couple of days.



tobik@ said:


> Hmm, it takes like 10 minutes to setup



I know but I need to setup repo then keep it in sync for two small packages. Even if there is ten of them it is easier to 'make deinstall/reinstall' when something breaks then to keep repo in shape. I understand the need for bigger more-than-one-machine setups.


----------



## SirDice (Aug 14, 2017)

dpx said:


> I know but I need to setup repo then keep it in sync for two small packages. Even if there is ten of them it is easier to 'make deinstall/reinstall' when something breaks then to keep repo in shape.


If you have a good repository there would be no need to rebuild something from ports (because there's a good package in your repository). Build once, install many


----------



## dpx (Aug 14, 2017)

tobik@ said:


> I bet the solution to your problem would be to setup your own package
> repository and point pkg(8) to it with a higher priority than the FreeBSD one.



Actually, more that I think about it, this is starting to make sense. I'll experiment with local repo first then see if there is any need to modify pkg.


----------



## SirDice (Aug 14, 2017)

I would actually recommend disabling the FreeBSD repository completely and only use your own. If your repository is missing something you'll find out quickly enough and mixing custom packages (with custom options) with the official packages (which all use the default settings) is bound to cause conflicts. In my view it's better to have something crash and burn because your repository is missing a package than being completely oblivious and end up with mixed options/defaults and a bunch of weird conflicts.


----------



## rufwoof (Aug 14, 2017)

dpx said:


> Thanks tobik@. I was about to clone pkg and send small PR that would make pkg much more user friendly without sacrificing any security. Are you saying that's not acceptable in any form? I have no intention to keep local patched pkg version only for myself so if it can be accepted OS wide I won't do it.



https://www.freebsd.org/doc/en/articles/linux-users/startup.html


> Since FreeBSD is developed as a complete operating system, user-installed applications are not considered to be part of the “base” system. User-installed applications are generally installed using Packages or Ports.



https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/why-port.html


> In FreeBSD, anyone may submit a new port, or volunteer to maintain an existing unmaintained port. No special commit privilege is needed.


----------



## dpx (Aug 14, 2017)

SirDice said:


> I would actually recommend disabling the FreeBSD repository completely and only use your own. If your repository is missing something you'll find out quickly enough and mixing custom packages (with custom options) with the official packages (which all use the default settings) is bound to cause conflicts. In my view it's better to have something crash and burn because your repository is missing a package than being completely oblivious and end up with mixed options/defaults and a bunch of weird conflicts.



I am trying to avoid that, especially since I have such disproportion between things I have to customize and those that work for me as stock builds. As long as I need up to ten-ish compiled ports I can live with occasional breakdowns. Everything is not process critical, including nvidia drivers.

I would avoid compiling _everything_ at all cost, that simply doesn't look appealing to me. I have run away from systemd mess on linux, even if there is Gentoo distro that offers not-systemd (OpenRC) init. It would work for me but Gentoo compiles everything from sources - big no for me. Otherwise, if we ignore compiling, it is one of the few systemd-free distros left.

FreeBSD (OS) has better performance than linux on my computer so I'd love to stay around. I have set local repo, for vim and nvidia only, and so far so good, we'll see how it behaves when new updates come.

Since I am using mutt-offlineimap-vdirsyncer-khal-rxvt it is all very lightweight and easy to set, if I ever need to change OS again. I hope I won't have to do that any time soon.

Please don't make me compile everything, I'll miss ZFS badly


----------



## drhowarddrfine (Aug 14, 2017)

dpx The typical reason for compiling from ports is one needs some feature that isn't in the defaults or won't make the package for some time down the road. In my case, for example, I needed a feature in nginx that was only available in ports with different flags set (SPDY then HTTP2 if your familiar). Many, maybe most, don't need to do that.


----------



## tankist02 (Aug 14, 2017)

dpx Have you looked at Synth to create a local repo and its option to use precompiled packages to reduce recompilations? I read this thread with great interest - I'd also love to have a few custom ports and the rest as precompiled packages.


----------



## SirDice (Aug 16, 2017)

dpx said:


> I would avoid compiling _everything_ at all cost, that simply doesn't look appealing to me.


I have set up poudriere for a client. I'm not building _everything_ (by everything I mean all 26000+ ports), I'm only passing a list of ports/packages I need. Sure, it needs to build 200-300 packages (due to all sorts of dependencies) but it only builds them all from scratch the first time. Most of the time the builds are incremental and really only rebuild things that require it. It does a fairly good job of it. Sometimes there's something odd going on and I just force it to rebuild the whole list from scratch.

In the end my repository only has the stuff we need/use. So there's no Gnome or KDE for example in my repository because we have no need for it on our servers.


----------



## dpx (Aug 18, 2017)

tankist02 said:


> dpx Have you looked at Synth to create a local repo and its option to use precompiled packages to reduce recompilations? I read this thread with great interest - I'd also love to have a few custom ports and the rest as precompiled packages.



Not yet but it looks interesting, thanks.

I am using local repo as tobik@ suggested and it works in terms of pkg not bugging me to install stock packages with each pkg update. I am expecting to have to recompile both packages when version bumps up but for only two packages so far that's not a big deal.


----------

