# Inconsistencies Between pkg <arg>, pkg_* and portmaster



## rtwingfield (Oct 28, 2013)

I cannot understand the inconsistent nuances between `pkg`ng and `portmaster`.  With apologies, this is a long and complicated post (I actually have over forty typed pages of examples, test results, and notes accumulated over a period of five man-days.  Very frustrating!)

NOTICE:
`pkgng` is installed on this system
`portmaster` is installed on this system
`portupgrade` is not installed on this system.​
Yes . . .I have read the `man` pages and Handbook, et al.

`pkgng` is the next generation replacement for the traditional FreeBSD package management tools, offering many features that make dealing with binary packages faster and easier.

`pkgng` is not a replacement for port management tools like ports-mgmt/`portmaster` or ports-mgmt/`portupgrade`. These tools can be used to install third-party software from both binary packages and the Ports Collection, while * pkgng installs only binary packages*. 

To ensure that the FreeBSD Ports Collection registers new software with `pkgng`, and not the traditional packages format, FreeBSD versions earlier than 10.X require this line, 
	
	



```
WITH_PKGNG= yes
```
 in `/etc/make.conf`: 

Once `pkgng` is installed, the package database must be converted from the traditional format to the new format by running the command. `pkg2ng`.​
Unfortunately, there is not much information in the Handbook, _Chapter 5.6.3.1. Upgrading Ports Using Portmaster_ (see http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-using.html) regarding specific examples of how to use the application.

There are two commands that can generate lists of installed ports and packages:
`pkg_info
	pkg info`​
Additionally, `ls` listings of the /var/db/pkg and /var/db/ports databases do not agree with the lists generated by `pkg_info` and `pkg info` commands.

(From the man pages) the output of `PKG_INFO(1)` is either extracted from package files named on the command line, or from already installed package information in /var/db/pkg/<pkg-name>.

There are apparently three databases effecting the results:

/var/db/pkg
/var/db/ports
/usr/ports/packages​

/usr/ports/packages is the default directory pointed to by the PACKAGES variable of `PORTMASTER(8)` and is used to store new and backup packages.  The content of this database seems to be volatile or transient.  

I'm not sure how the /var/db/ports database is maintained or updated, but I do know that to check for stale entries in /var/db/ports, `# portmaster -v --check-port-dbdir` apparently tries to clean up the database.  For example:

Ran `make deinstall` in /usr/ports/lang/perl5.14
. . .still has corresponding sub-dir in /var/db/ports that contains an options file.   Why?
Running `portmaster -v --check-port-dbdir` prompted to delete the db for perl5.14.

Note that two versions existed for python:

===>>> Checking lang_python27: Ok
	===>>> Checking lang_python33: Ok

python33 is the latest version.  How to remove python27?  (Used `make deinstall`.)
Again, had to run `portmaster -v --check-port-dbdir` to remove from /var/db/ports.​


The `portmaster --delete-packages` argument sounds dangerous.  If there is . . .what is the correct syntax to instruct `portmaster` to delete only one specific port?  If portmaster can delete only one specific port, will it update the /var/db/ports database with having to run `portmaster -v --check-port-dbdir`?  (Every time I've tried to delete a specific single port with `portmaster`, it launches into a (re)install of the port.)  The only way I've been able to delete an installed port is to `make deinstall`, and apparently the make process does not update the /var/db/ports database (or others uniformily).



Regardless, I have produced a four-column report listing the output of ls /var/db/pkg, ls /var/db/ports, pkg info, and pkg_info.  None of them agree with each of the others!  This document is a 56K PDF and this forum will not let me upload it, so here is a URL to the same:  http://archaxis.net/htdocs/RTWingfield/htdocs/FreeBSD/ports-pkgs/FreeBSD_Port_and_Package_Management_Lists.pdf.

If you look at (relative) page 15 of 19, you will see four columns:
*ls /var/db/pkg* (360 total) -- *ls /var/db/ports* (84 total) -- *pkg info* (376 total) -- *pkg_info* (358 total)
(The numbers are the number of rows or objects in each column.)

*MySQL Example:*  Neither the mysql-server or mysql-client is listed in the ls /var/db/ports column. 
The versions, 5.6.14, are correct for both client and server in the ls /var/db/pkg and pkg_info columns; however, the pkg info column identifies the mysql-server-*5.5.33* version!  Where did it get this???


*Continuing re. Perl:* I've tried to install perl5.18 via portmaster, but it seems to insist on reinstalling perl5.14. 

Only perl5.14 is listed in the ls /var/db/pkg and pkg_info columns; doesn't appear in either the ls /var/db/ports or pkg info columns.  I've just restarted (~5:20 PM) using `portmaster -i /usr/ports/lang/perl5.18`.  This will take a while.  It will be interesting to see what changes.  *See my followup post*.


*Final Example re. PHP5:*  This has been a real "bear"!  I'm going to try a "remake" using `portmaster` instead of `make deinstall` and `make build install clean`. Currently, the only php5-* objects show up  in the pkg info column as "extensions".  I will start this once the Perl compiles and (hopefully) installation finish.  It will be interesting to see what changes.

It is very annoying that the number and versions in the example columns do not balance.  

Why are they so different?

*What I suspect:*  Some make files (when run from a `make` command) do not consistently update the databases as per `pkgng` and `portmaster` hopefully do.  

How to synchronize the results?


----------



## rtwingfield (Oct 29, 2013)

*perl5.18 portmaster results*

Perhaps this test results will illustrate why this is so frustrating!

Just completed  `portmaster -i /usr/ports/lang/perl5.18`

The results are as follows:

`ls /var/db/pkg`
perl5.14-5.14.4_2 _. . .why is this still here?_
perl5.18-5.18.1_2 _. . .this was added as a result of the portmaster operations.
_
`ls /var/db/ports`
lang_perl5.18 _. . .this was added as a result of the portmaster operations.
_
`pkg info`
perl5.18-5.18.1_2 _. . .this was added as a result of the portmaster operations.
_
`pkg_info`
perl5.14-5.14.4_2_  . . .why is this still pointing to perl5.14-5.14.4_2?_​
Ran the following, but Perl reported as "corrupt" . . .why?:

`# pkg_version -l "<"|less`

```
alsa-plugins                        <
bash                                <
bison                               <
ca_root_nss                         <
cairo                               <
desktop-file-utils                  <
dialog4ports                        <
en-freebsd-doc                      <
gettext                             <
gmp                                 <
hal                                 <
libSM                               <
libX11                              <
libXaw                              <
libXmu                              <
libXpm                              <
libXrandr                           <
libXv                               <
libgsf                              <
libidn                              <
libltdl                             <
libvpx                              <
m4                                  <
nspr                                <
nss                                 <
pciids                              <
pkg_version: the package info for package 'perl5.18-5.18.1_2' is corrupt
pixman                              <
webp                                <
xf86-video-intel                    <
xf86-video-vesa                     <
xinput                              <
xkeyboard-config                    <
xorg-server                         <
xterm
```

Ran the following, but *NO* mention of Perl
(lines numbers provided by `vi`)

`# portmaster -L --index-only | egrep '(ew|ort) version|total install' > newv_available.txt`


```
1         ===>>> New version available: dialog4ports-0.1.5_2
      2         ===>>> New version available: en-freebsd-doc-42697,1
      3         ===>>> New version available: libvpx-1.2.0
      4         ===>>> New version available: ca_root_nss-3.15.2_1
      5         ===>>> New version available: gmp-5.1.3
      6         ===>>> New version available: m4-1.4.17,1
      7         ===>>> New version available: nspr-4.10.1
      8         ===>>> New version available: pciids-20131023
      9         ===>>> New version available: pixman-0.30.2
     10         ===>>> New version available: cairo-1.10.2_6,2
     11         ===>>> New version available: desktop-file-utils-0.22_1
     12         ===>>> New version available: gettext-0.18.3.1
     13         ===>>> New version available: hal-0.5.14_21
     14         ===>>> New version available: libSM-1.2.2,1
     15         ===>>> New version available: libXaw-1.0.12,2
     16         ===>>> New version available: libXmu-1.1.2,1
     17         ===>>> New version available: libXrandr-1.4.2
     18         ===>>> New version available: libXv-1.0.10,1
     19         ===>>> New version available: libgsf-1.14.28
     20         ===>>> New version available: libidn-1.28_1
     21         ===>>> New version available: php5-gd-5.4.21
     22         ===>>> New version available: php5-pdo-5.4.21
     23         ===>>> New version available: php5-xml-5.4.21
     24         ===>>> New version available: webp-0.3.1_1
     25         ===>>> New version available: xf86-video-intel-2.7.1_5
     26         ===>>> New version available: xf86-video-vesa-2.3.3
     27         ===>>> New version available: xinput-1.6.1
     28         ===>>> New version available: xkeyboard-config-2.9_1
     29         ===>>> New version available: xorg-server-1.7.7_11,1
     30         ===>>> New version available: xterm-297
     31         ===>>> New version available: alsa-plugins-1.0.27_1
     32         ===>>> New version available: bash-4.2.45
     33         ===>>> New version available: bison-2.7.1,1
     34         ===>>> New version available: mysql55-server-5.5.34
     35         ===>>> New version available: nss-3.15.2
     36         ===>>> New version available: php5-filter-5.4.21
     37         ===>>> New version available: php5-json-5.4.21
     38         ===>>> New version available: php5-mbstring-5.4.21
     39         ===>>> New version available: php5-mcrypt-5.4.21
     40         ===>>> New version available: php5-mysql-5.4.21
     41         ===>>> New version available: php5-mysqli-5.4.21
     42         ===>>> New version available: php5-openssl-5.4.21
     43         ===>>> New version available: php5-pdo_sqlite-5.4.21
     44         ===>>> New version available: php5-posix-5.4.21
     45         ===>>> New version available: php5-session-5.4.21
     46         ===>>> New version available: php5-simplexml-5.4.21
     47         ===>>> New version available: php5-snmp-5.4.21
     48         ===>>> New version available: php5-sqlite3-5.4.21
     49         ===>>> New version available: php5-tokenizer-5.4.21
     50         ===>>> New version available: php5-xmlreader-5.4.21
     51         ===>>> New version available: php5-xmlwriter-5.4.21
     52         ===>>> New version available: php5-zip-5.4.21
     53         ===>>> New version available: php5-zlib-5.4.21
     54 ===>>> 377 total installed ports
     55         ===>>> 53 have new versions available
```


----------



## wblock@ (Oct 29, 2013)

portmaster is for upgrading ports.  pkg is a binary package manager that keeps track of installed packages and can (optionally) install binary packages.  Think of it as the database that tracks which packages are installed.  It is a replacement for the old pkg_install programs.

After switching to pkg, stop using the old pkg_install programs.  The data they use is no longer updated after the switchover.

There are equivalents to all the pkg_* commands, they are just subcommands given after pkg.  For instance, the old `pkg_info` is `pkg info`.


----------



## rtwingfield (Oct 29, 2013)

Hello Warren,

With sincere thanks for your reply, I have read your blog post, http://www.wonkity.com/~wblock/docs/html/portupgrade.html#_updating_the_ports_collection, if I may quote myself (from FreeBSD Handbook The FreeBSD Documentation Project, Revision: 42953, Copyright Â© 1995-2013 The FreeBSD Documentation Project, _Chapter 5.5. Using pkgng for Binary Package Management_ http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/pkgng-intro.html, ):


> `pkgng` is not a replacement for port management tools like `ports-mgmt/ portmaster` or `ports-mgmt/ portupgrade`. These tools can be used to install third-party software from both binary packages and the Ports Collection, while `pkgng` installs only binary packages.


 I understand this; however, once a reader drills down into the lexicon, it seems that many of the "port management" tools start to refer to functions as "`pkg-this`" or "`pkg-that`"; very confusing.  IMHO, there should be a clear distinction between `pkg delete <args>` and `portmaster delete <args>`.  The fact that some of these `portmaster` operations are executed as `pkg-whatever` contributes to (my) perception that the commands should only pertain to installed _packages_.

Arguably (I suppose) . . .once a port is downloaded, compiled, and installed, then the executables are essentially a _package_.

Regarding you comment,


> After switching to pkg, stop using the old pkg_install programs.


I do not (never have) use `pkg_install`.  I am an _old_ programmer and prefer to compile in-place native code for the platform's OS.

your comment:





> There are equivalents to all the `pkg_*` commands, they are just subcommands given after `pkg`. For instance, the old `pkg_info` is `pkg info`.


 . . .so should the recommendation be to not "trust" the `pkg_info` command?  (I'm concluding . . .apparently so.)

. . .moving on:  

`pkd info` lists mysql-server-5.5.33 when in fact, the version installed and executable is/are:
`# mysqladmin -V`

mysqladmin  Ver 8.42 Distrib 5.6.14, for FreeBSD9.1 on i386

`# mysql -V`

mysql  Ver 14.14 Distrib 5.6.14, for FreeBSD9.1 (i386) using  5.2​

Note that `pkg_info` "got it right"; however, `pkg info` still displays mysql-server-5.5.33.

*QUESTION:*  Once `portmaster` and `pkgng` are installed, should one ever simply execute a command line `make build install clean` scenario?  I'm still concerned how reliable legacy make files are regarding this _new_ database scenario.


----------



## kpa (Oct 29, 2013)

@wblock@ is trying to tell you that `pkg_info` should not be used at all after the conversion to PKGNG packages, the old and the new package tools do not use the same database and you're guaranteed to see inconsistencies if you try to use both.



> QUESTION: Once portmaster and pkgng are installed, should one ever simply execute a command line make build install clean scenario? I'm still concerned how reliable legacy make files are regarding this new database scenario.



No, when you are using PKGNG packages the ports Makefiles are not used any differently than before. All packages are still built using the same ports(7) infrastructure as before, only the packaging and package registration part has changed to use the new pkg(8) tools. Stop thinking that PKGNG has somehow magically replaced everything in the ports system when it really hasn't.


----------



## wblock@ (Oct 29, 2013)

rtwingfield said:
			
		

> I understand this; however, once a reader drills down into the lexicon, it seems that many of the "port management" tools start to refer to functions as "`pkg-this`" or "`pkg-that`"; very confusing.  IMHO, there should be a clear distinction between `pkg delete <args>` and `portmaster delete <args>`.  The fact that some of these `portmaster` operations are executed as `pkg-whatever` contributes to (my) perception that the commands should only pertain to installed _packages_.
> 
> Arguably (I suppose) . . .once a port is downloaded, compiled, and installed, then the executables are essentially a _package_.



Not essentially, it *is* a package.  A port builds into a package, which is then installed.  Information is kept on where the files are installed and what it depends on.  That information in handled by either the old pkg_* programs or the new pkg.



> I do not (never have) use `pkg_install`.  I am an _old_ programmer and prefer to compile in-place native code for the platform's OS.



I meant to say pkg_add.  All of those individual programs with the names starting with pkg_ are part of the old system, replaced by pkg.



> so should the recommendation be to not "trust" the `pkg_info` command?  (I'm concluding . . .apparently so.



More than that: the information given by `pkg_info` is coming from a database that is no longer being used (text files in /var/db/pkg).  When you switch to pkg, it keeps all the information in a SQLite table.  At that point, looking at the old information is, at best, misleading.  It does not apply any more.  It's an old, stale copy of information.



> *QUESTION:* Once `portmaster` and `pkgng` are installed, should one ever simply execute a command line `make build install clean` scenario?



Yes.  The ports system builds a package, and pkg records the installation in the database.



> I'm still concerned how reliable legacy make files are regarding this _new_ database scenario.



Port builds, installs, deinstalls, and other operations are the same.  It's just the database layer underneath that has been rewritten.  To use a database metaphor, the old pkg_* programs were MySQL, and the new pkg system is PostgreSQL.  The end user (portmaster, or manual port builds or installs) does not see a difference, only the database engine has changed.


----------



## rtwingfield (Oct 29, 2013)

This was in response to kpa's comments:

. . .then why does `pkg info` _insist_ on displaying that mysql-server-5.5.33 is installed, when in fact, mysql56-server-5.6.14 is the current version, ergo:


```
# mysqladmin -V
mysqladmin  Ver 8.42 Distrib 5.6.14, for FreeBSD9.1 on i386

# mysql -V
mysql  Ver 14.14 Distrib 5.6.14, for FreeBSD9.1 (i386) using  5.2
```


----------



## wblock@ (Oct 29, 2013)

That one is harder to answer.  Was the conversion to pkg done completely, importing the old package data to the new database?  Maybe a port was installed that depended on the old version of MySQL and installed it because the pkg database did not show that the newer version was already installed.


----------



## rtwingfield (Oct 29, 2013)

Re:  Warren Block's comment:


> More than that: the information given by `pkg_info` is coming from a database that is no longer being used (text files in /var/db/pkg). When you switch to `pkg`, it keeps all the information in a *SQLite table*. At that point, looking at the old information is, at best, misleading. It does not apply any more. It's an old, stale copy of information.



I've found no reference to this nuance in any man or Handbook article.  How would you know this?

Regardless, entries in /var/db/pkg appear to be updated.

I would prefer that only one scenario be enforced, i.e., installed.  For example, if `pkg` is installed, then all legacy systems should be removed, waxed, deep-six'ed,   . . .along with all the `pkg_*` commands.  

/var/db/pkg should be deleted.

This is way too confusing.


----------



## rtwingfield (Oct 29, 2013)

wblock@ said:
			
		

> That one is harder to answer.  Was the conversion to pkg done completely, importing the old package data to the new database?  Maybe a port was installed that depended on the old version of MySQL and installed it because the pkg database did not show that the newer version was already installed.



. . .well, I did run `pkg2ng` based on the following documentation:

To ensure that the FreeBSD Ports Collection registers new software with `pkgng`, and not the traditional packages format, FreeBSD versions earlier than 10.X require this line, 
	
	



```
WITH_PKGNG= yes
```
, in /etc/make.conf: 

Once `pkgng` is installed, the package database must be converted from the traditional format to the new format by running the command. `pkg2ng`.​
Regarding port dependencies, I was working with Perl and PHP . . .both huge applications with (I'm sure) many dependencies.

. . .so I assume that `pkg2ng` converted /var/db/pkg to the MySQLite db?  (I haven't looked for it.)


----------



## rtwingfield (Oct 29, 2013)

wblock@ said:
			
		

> That one is harder to answer.  Was the conversion to pkg done completely, importing the old package data to the new database?  Maybe a port was installed that depended on the old version of MySQL and installed it because the pkg database did not show that the newer version was already installed.



. . .perhaps an installed version of phpMyAdmin, Perl or PHP.  A lot of inter-dependencies.


----------



## throAU (Oct 29, 2013)

Would it not be sane to simply have pkg_ commands removed (or otherwise hobbled to simply report "You need to use the PKGNG tools!" or similar once the package database is migrated to PKGNG?

If the commands are going to be at best misleading, and possibly dangerous, they should be disabled (or at least require an over-ride switch), in my opinion.


----------



## rtwingfield (Oct 29, 2013)

Exactly my point!


----------



## kpa (Oct 29, 2013)

The old `pkg_*` tools are not installed at all by default in FreeBSD 10.


----------



## wblock@ (Oct 29, 2013)

rtwingfield said:
			
		

> I've found no reference to this nuance in any man or Handbook article.  How would you know this?



@bapt gave a presentation at BSDCan 2012:
http://www.youtube.com/watch?v=4Hxq7AHZ27I&list=PL4A10C737F224D0DC


----------



## Beastie (Oct 29, 2013)

rtwingfield said:
			
		

> Regardless, entries in /var/db/pkg appear to be updated.


I may be wrong, but in a system that is fully upgraded to use the new *pkg* tool, there shouldn't be any of the old directories with the +COMMENT, +CONTENTS, etc. files - whether updated or not.
You should only find SQLite databases in there:
`ls /var/db/pkg`

```
local.sqlite
local.sqlite-journal
repo-packagesite.sqlite
```

I used *pkg2ng* months ago, so I don't remember whether it clears the old database or not. So, again, I may be wrong.

When I want to use the new tool, I just remove everything myself and start from scratch:
`# rm -R /usr/local`
`# rm -R /var/db/pkg`
`# pkg`





			
				throAU said:
			
		

> Would it not be sane to simply have pkg_ commands removed (or otherwise hobbled to simply report "You need to use the PKGNG tools!" or similar once the package database is migrated to PKGNG?


If you want to start using the new *pkg* tool, then actually *using* it makes perfect sense. That's the whole point, no?


----------



## wblock@ (Oct 29, 2013)

No, it's a fair point.  The old programs should not be left around after the conversion, it gives the user an option to make an unnecessary mistake.  But they are part of the base system, so it's different than just being a port that can be uninstalled.  It might be possible to have the pkg_* programs detect pkgng and refuse to run, which would be an improvement.  Anyone want to enter a PR?


----------



## junovitch@ (Oct 31, 2013)

So, have the pkg_ tools just look for a sqlite database and toss up a "I hope you know what you are doing " warning if it's there?  Something like this perhaps?


```
#include <stdio.h>
#include <sys/stat.h>
#include <fcntl.h>

struct stat buffer;
int main()
{
if(stat("/var/db/pkg/local.sqlite", &buffer) != 1)
        printf("WARNING: Found /var/db/pkg/local.sqlite\nUse of pkg_ tools not recommended when pkgng is in use\n");
}
```


----------



## wblock@ (Oct 31, 2013)

That sort of thing, yes.  I'd hope it could be a little more elegant, but it's the idea.


----------



## junovitch@ (Oct 31, 2013)

Yes, that was a very quick and ugly example. I can poke at it a bit this weekend to make it a legit patch and submit a PR. It does bring a question to mind though, since 10.0-RELEASE will be on PKGNG tools will the 9.3-RELEASE that follows be as well?  Or will the 9 branch stay with old pkg_ tools until EOL?


----------



## wblock@ (Oct 31, 2013)

There may be a common place to patch all of the old pkg_* programs.  I don't know what the plans are for FreeBSD 8 and 9.


----------



## kpa (Oct 31, 2013)

http://lists.freebsd.org/pipermail/freebsd-pkg/2013-October/000107.html


----------

