# Problems with local pkg repo catalog



## ssw01 (Aug 3, 2021)

I recently did a poudriere run on a 13.0-RELEASE-p3 system to build a couple of ports.
I expected the newly built packages to be added to the corresponding repository because I could've
sworn that was my past experience.  Instead, running `pkg install <some pkg>` for
"some pkg" already in the repo produced a message that "some pkg" wasn't in the repo.
The packagesite.txz seemed to contain only the most recently built couple of
packages.

So I decided I needed to regenerate the catalog and ran `pkg repo <repo path>`
which produced the output

```
pkg: No package files have been found
Cannot create repository catalogue
```

The version of pkg on this machine is 1.17.1.

I tried running the same `pkg repo` on a 12.2 machine with pkg 1.15.10 specifying the
same repo tree via NFS and it  seemed to work fine.

Does this call for a bug report or am I doing something wrong?

----
Edit:

So I force installed pkg-1.16.3 with `pkg add -f <path to pkg-1.16.3.txz>` and tried `pkg repo <repo path>`
again which succeeded.

Running `pkg install apache24` then worked after reinstalling pkg-1.17.1.  So I'm back in business but this
seems peculiar.


----------



## Alain De Vos (Aug 3, 2021)

Two files which might shed a light

```
/etc/pkg/FreeBSD.conf
```
And the contents of

```
/usr/local/etc/pkg/repos
```


----------



## ssw01 (Aug 3, 2021)

Here they are.  These have been working fine for ages.

Thanks!


----------



## Alain De Vos (Aug 4, 2021)

If i'm correct your directory,
/var/poudriere/data/packages/r130amd64-muqchem-default
should contain:

```
meta.conf
meta.txz
packagesite.txz
[ And directories:]
All/
Latest/
```


----------



## ssw01 (Aug 4, 2021)

It does indeed, although I won't have access to the machine involved until tomorrow so can't
provide an actual listing at the moment.  Recently, files of the form packagesite.pkg
which are links to or from the corresponding .txz files have appeared as well.

I unpacked packagesite.txz and looked at the yaml file.  It contained only the most recently built
packages which brings up the first part of my question.  Suppose I run this sequence:


```
poudriere bulk -j <jailname> -f <file containing list of ports>
poudriere bulk -j <jailname> <origin1/port1> <origin2/port2>
```

Would the corresponding repository be expected to contain the union of what's in the
file from the first invocation and the two packages from the second or just the latter?

I could have sworn it was the union but maybe I dreamed that.  The actual
contents of <repo root>/.latest/All include the union of all the .txz files
but the metadata seems to have other ideas.

In any case, running `pkg repo <path>` with rev 1.17.1 fails while the
same command on the same directory using rev 1.16.3 works, so that seems
like a problem.

Thanks!


----------



## SirDice (Aug 4, 2021)

Some things changed with pkg(8) version 1.17.0. You need to clean out your repository to get rid of the old packages. These seem to get a preference over the newer packages. Just run `poudriere pkgclean -j <jailname> -f <file containing list of ports>` to have it remove all the old packages.


----------



## ssw01 (Aug 4, 2021)

Thanks SirDice,

I'm a bit confused.  The man page for poudriere-pkgclean(8) says the -f option
specifies a list of ports to *keep*.  When I ran `poudriere pkgclean -j <jail> -f <my port list>`,
poudriere(8) reported that it wanted to remove 1304 ports it had recently
spent a week building.  So it looks as if it considers everything I built using the -f option
in the example in message #5 to be stale.

Do I really have to do this to get things back in sync?


----------



## SirDice (Aug 4, 2021)

ssw01 said:


> The man page for poudriere-pkgclean(8) says the -f option
> specifies a list of ports to *keep*.


That is correct.


ssw01 said:


> When I ran `poudriere pkgclean -j <jail> -f <my port list>`,
> poudriere(8) reported that it wanted to remove 1304 ports it had recently
> spent a week building.


Look closely, those are old versions. You can collect a whole bunch of old packages if you continuously update the builds. The repository will keep all those old versions around. So after a while you have multiple (old) versions of various packages. poudriere-pkgclean(8) will clean those out.


----------



## ssw01 (Aug 4, 2021)

Here's the output of `ls -l /var/poudriere/data/packages/r130amd64-std-default/.latest/All | tail -5`:

```
-rw-r--r--  1 nobody  wheel     227392 Jul 28 17:23 zip-3.0_1.txz
-rw-r--r--  1 nobody  wheel    4616668 Jul 31 21:27 zsh-5.8.txz
-rw-r--r--  1 nobody  wheel     595452 Aug  2 15:55 zstd-1.5.0.pkg
lrwxr-xr-x  1 nobody  wheel         14 Aug  2 15:55 zstd-1.5.0.txz -> zstd-1.5.0.pkg
-rw-r--r--  1 nobody  wheel     106176 Jul 27 07:37 zziplib-0.13.72_1.txz
```

Here are the final few lines of output from `poudriere pkgclean -j r130amd64-std -f /var/poudriere/portlist-std.txt`:

```
/var/poudriere/data/packages/r130amd64-std-default/.latest/All/zip-3.0_1.txz
/var/poudriere/data/packages/r130amd64-std-default/.latest/All/zsh-5.8.txz
/var/poudriere/data/packages/r130amd64-std-default/.latest/All/zstd-1.5.0.txz
/var/poudriere/data/packages/r130amd64-std-default/.latest/All/zziplib-0.13.72_1.txz
[00:01:34] Removing these 1304 stale packages will free: 3.50 GiB
[00:01:34] Proceed? [y/N] n
[00:03:25] Cleaning up
[00:03:25] Unmounting file systems
```

This was actually a pretty new repo without much cruft because I created it fresh after updating to 13.


----------



## SirDice (Aug 4, 2021)

Don't use dashes (`-`) in the name of the jail (r130amd64-std) or the ports tree name. That causes a bunch of weird issues. Underscores are fine. That problem seems to stem from the fact poudriere uses those dashes as a divisor.


----------



## ssw01 (Aug 4, 2021)

Oooh, never knew that.  Thanks!


----------



## SirDice (Aug 4, 2021)

ssw01 said:


> Oooh, never knew that. Thanks!


Me neither when I first started with poudriere. It's not in the manual either, but I noticed some really weird behavior if you use them. It might be the reason why poudriere-pkgclean(8) wants to remove perfectly good packages in your case.


----------



## ssw01 (Aug 4, 2021)

OK, since I seem to be able to make the existing repos usable by running `pkg repo` with a pre-1.17
version of pkg(8), I'll limp along that way until the time comes for a ports update.  I'll then create shiny
new jails following your naming advice and rebuild everything.

Thanks again.


----------



## Jose (Aug 4, 2021)

SirDice said:


> Don't use dashes (`-`) in the name of the jail (r130amd64-std) or the ports tree name. That causes a bunch of weird issues. Underscores are fine. That problem seems to stem from the fact poudriere uses those dashes as a divisor.


Doc bug?


----------



## Alain De Vos (Aug 4, 2021)

Poudriere puts itself a  "-" between the jailname and portsname.


----------



## SirDice (Aug 5, 2021)

Alain De Vos said:


> Poudriere puts itself a "-" between the jailname and portsname.


Yeah, that's why I think having a dash in the jail name or ports tree causes a failure to properly identify the jail or ports tree. Options didn't get properly applied, make.conf files didn't work and a whole bunch of other weird behavior.


----------



## grahamperrin@ (Aug 7, 2021)

Jose said:


> Doc bug?











						poudriere naming conventions for jails and ports trees · Issue #897 · freebsd/poudriere
					

Prerequisites ☑ Have you checked for an existing issue describing the issue? From https://forums.FreeBSD.org/threads/81582/post-525675: Don't use dashes (-) in the name of the jail … or the por...




					github.com


----------



## brd@ (Aug 7, 2021)

SirDice said:


> Don't use dashes (`-`) in the name of the jail (r130amd64-std) or the ports tree name. That causes a bunch of weird issues. Underscores are fine. That problem seems to stem from the fact poudriere uses those dashes as a divisor.


I have been using dashes in my jail names for many years and have had no issues.  If you have found a bug please open an issue and we would be happy to fix it.


----------



## brd@ (Aug 7, 2021)

ssw01 said:


> OK, since I seem to be able to make the existing repos usable by running `pkg repo` with a pre-1.17
> version of pkg(8), I'll limp along that way until the time comes for a ports update.  I'll then create shiny
> new jails following your naming advice and rebuild everything.
> 
> Thanks again.


What version of poudriere are you using?


----------



## grahamperrin@ (Aug 8, 2021)

With poudriere-devel-3.3.99.20210720 on FreeBSD 14.0-CURRENT,


```
root@mowa219-gjp4-zbook-freebsd:~ # grep -v \# /usr/local/etc/poudriere.d/make.conf
WITH_DEBUG_PORTS += multimedia/webcamd sysutils/bsdisks

ICA_CERTS=QuoVadisEuropeEVSSLCAG1.crt

LICENSES_ACCEPTED += commercial

WITHOUT_LLVM_TARGET_ALL=
root@mowa219-gjp4-zbook-freebsd:~ #
```

With `-` in the name of a jail:

`poudriere jail -c -j 13-0-release -v 13.0-RELEASE`
`poudriere jail -u -j 13-0-release`
`poudriere bulk -b latest -j 13-0-release -Ct sysutils/lsblk`
`poudriere bulk -b latest -j 13-0-release -Ct multimedia/webcamd sysutils/bsdisks`
From the manifest of the resulting package for webcamd:


```
"options":{"COMPAT32":"on","DEBUG":"off","DVB":"on","INPUT":"on","KEYBOARD":"off","MOUSE":"off","RADIO":"on","V4L2LOOPBACK":"on","VT_CLIENT":"off","VT_SERVER":"off","WEBCAM":"on"}
```

– debug _off_. PEBKAM? Something wrong with my /usr/local/etc/poudriere.d/make.conf maybe?

With my usual jail (for CURRENT) I ran: 

`poudriere bulk -b latest -j main -Ct multimedia/webcamd`
– then installed the package and: 


```
root@mowa219-gjp4-zbook-freebsd:~ # pkg info webcamd | grep -i debug
        DEBUG          : off
root@mowa219-gjp4-zbook-freebsd:~ #
```

– again, _off_ but I want it _on_. What am I missing?


----------



## ssw01 (Aug 9, 2021)

brd@ said:


> What version of poudriere are you using?


poudriere-devel-3.3.99.20210521.

I've deleted my old jails and created a new one with no '-' in the name and am in the
middle of a build.  It has a few days yet to run, so I won't know for a while whether
the behavior is different.


----------



## ssw01 (Aug 11, 2021)

Well, my build finished early because lang/rust failed which caused a bunch of stuff to get skipped but that's neither
here nor there for present purposes.  The build command was `poudriere bulk -f /var/poudriere/plists/full.txt -j r130amd64full`.

I then ran `poudriere pkgclean -f /var/poudriere/plists/full.txt -j r130amd64full` to see what would happen.  Here's the
tail end of the output:


```
/var/poudriere/data/packages/r130amd64full-default/.latest/All/zip-3.0_1.txz
/var/poudriere/data/packages/r130amd64full-default/.latest/All/zsh-5.8.txz
/var/poudriere/data/packages/r130amd64full-default/.latest/All/zstd-1.5.0.txz
/var/poudriere/data/packages/r130amd64full-default/.latest/All/zziplib-0.13.72_1.txz
[00:01:38] Removing these 1153 stale packages will free: 
[00:01:38] Proceed? [y/N] n
[00:02:08] Cleaning up
[00:02:08] Unmounting file systems
```

I then ran `pkg upgrade ports-mgmt/poudriere-devel` to see what upgrading to 20210720 would do.  The output:


```
Updating local repository catalogue...
pkg: Repository local has a wrong packagesite, need to re-create database
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
Fetching packagesite.pkg: 100%  276 KiB 282.2kB/s    00:01    
Processing entries: 100%
local repository update completed. 1153 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
    ca_root_nss: 3.63 -> 3.68
    poudriere-devel: 3.3.99.20210521 -> 3.3.99.20210720

Number of packages to be upgraded: 2

Proceed with this action? [y/N]: n
```

So given the message about "wrong packagesite", I ran `pkg repo /var/poudriere/data/packages/r130amd64full-default`,
which produced:


```
pkg: No package files have been found
Cannot create repository catalogue
```

Also:


```
> ls -al /var/poudriere/data/packages/r130amd64full-default
total 12
drwxr-xr-x  3 root  wheel  512 Aug 11 08:36 .
drwxr-xr-x  9 root  wheel  512 Aug  6 15:02 ..
lrwxr-xr-x  1 root  wheel   18 Aug 11 01:38 .buildname -> .latest/.buildname
lrwxr-xr-x  1 root  wheel   20 Aug 11 01:38 .jailversion -> .latest/.jailversion
lrwxr-xr-x  1 root  wheel   16 Aug 11 01:38 .latest -> .real_1628660288
drwxr-xr-x  4 root  wheel  512 Aug 11 01:38 .real_1628660288
lrwxr-xr-x  1 root  wheel   11 Aug 11 01:38 All -> .latest/All
lrwxr-xr-x  1 root  wheel   14 Aug 11 01:38 Latest -> .latest/Latest
lrwxr-xr-x  1 root  wheel   17 Aug 11 01:38 meta.conf -> .latest/meta.conf
lrwxr-xr-x  1 root  wheel   16 Aug 11 01:38 meta.pkg -> .latest/meta.pkg
lrwxr-xr-x  1 root  wheel   16 Aug 11 01:38 meta.txz -> .latest/meta.txz
lrwxr-xr-x  1 root  wheel   23 Aug 11 01:38 packagesite.pkg -> .latest/packagesite.pkg
lrwxr-xr-x  1 root  wheel   23 Aug 11 01:38 packagesite.txz -> .latest/packagesite.txz
-rw-r--r--  1 root  wheel    0 Aug 11 08:36 packagesite.yaml
```

Anything else I can try to help track this down?


----------



## SirDice (Aug 11, 2021)

ssw01 said:


> ```
> Updating local repository catalogue...
> pkg: Repository local has a wrong packagesite, need to re-create database
> ```


You get that message if the URL to your repository is changed.


----------



## ssw01 (Aug 11, 2021)

SirDice said:


> You get that message if the URL to your repository is changed.


Ah, gotcha.

So I went ahead and ran the `pkg upgrade ports-mgmt/poudriere-devel`, so now running
poudriere-devel-3.3.99.20210720.

Then tried the `poudriere pkgclean` again and got the same result (it wants to delete the
entire repository).


----------



## SirDice (Aug 11, 2021)

ssw01 said:


> Then tried the `poudriere pkgclean` again and got the same result (it wants to delete the entire repository).


That's odd as it shouldn't do that, poudriere-pkgclean(8) works as expected for me. I do have a similar issue with poudriere-logclean(8). If I run `poudriere logclean 14` it always want to delete _all_ build logs when it should only delete the ones older than 14 days.


----------



## ssw01 (Aug 11, 2021)

So I reran `poudriere pkgclean` with the "-vv" option and captured the voluminous output.  I can attach
the file if desired but even compressed, it's over 400 kB.

Lots of lines of the form


```
[00:00:56] Found incorrect format file: /var/poudriere/data/packages/r130amd64full-default/.latest/All/zziplib-0.13.72_1.txz
```

seem suspicious.

I'm working on grokking pkgclean.sh where the file suffix is being checked but maybe this will ring a bell
with someone.


----------



## ssw01 (Aug 11, 2021)

OK, after spending some time spelunking through the pkgclean logic, I think I understand what's
supposed to happen.  The only files that are getting called out to be fed into the shredder are *.txz files.
In the new scheme, these are symlinks to *.pkg files and it looks like the *.pkg files
are safe.

Had I gone ahead with what `poudriere pkgclean` wanted to do, everything *probably* would have
been fine.

Still a bit afraid to try it though!


----------



## grahamperrin@ (Aug 12, 2021)

ssw01 said:


> afraid to try it



What's the fear?


----------



## ssw01 (Aug 12, 2021)

grahamperrin said:


> What's the fear?


The fear is having to spend another week rebuilding everything if it got blown away after all.

Of course, the computer has to do the actual work.  I just have to wait...


----------



## grahamperrin@ (Aug 13, 2021)

ssw01 said:


> poudriere-devel-3.3.99.20210521.





ssw01 said:


> … week rebuilding everything …



It should not take so long with binaries. 


```
-b name  Specify the name of the binary package branch to use to prefetch
              packages.  Should be "latest", "quarterly", "release_*", or url.
              With this option poudriere will first try to fetch from the
              binary package repository specified the binary package prior to
              do the sanity check if the package does not already exist.  When
              -t is used along with -C, or -c, then listed packages will not
              be fetched.

              See PACKAGE_FETCH_BRANCH, and PACKAGE_FETCH_URL, in
              poudriere.conf.sample.
```


----------



## grahamperrin@ (Aug 13, 2021)

ssw01 said:


> … Had I gone ahead with what `poudriere pkgclean` wanted to do, …



Here, yesterday: 


```
root@mowa219-gjp4-zbook-freebsd:~ # pkg clean -n
Nothing to do.
root@mowa219-gjp4-zbook-freebsd:~ # pkg clean -a -y
The following package files will be deleted:
        /var/cache/pkg/kmix-21.04.3.pkg
        /var/cache/pkg/kmix-21.04.3~0adcfd78b4.pkg
        /var/cache/pkg/volumeicon-0.5.1_1~dd3f2e06cc.pkg
        /var/cache/pkg/volumeicon-0.5.1_1.pkg
The cleanup will free 1 MiB
Deleting files: 100%
All done
root@mowa219-gjp4-zbook-freebsd:~ # poudriere pkgclean
poudriere pkgclean [options] [-f file|cat/port ...]

Parameters:
    -A          -- Remove all packages
    -a          -- Keep all known ports
    -f file     -- Get the list of ports to keep from a file
    [ports...]  -- List of ports to keep on the command line

Options:
    -j jail     -- Which jail to use for packages
    -J n        -- Run n jobs in parallel (Defaults to the number of
                   CPUs times 1.25)
    -n          -- Do not actually remove anything, just show what would be
                   removed
    -N          -- Do not build the package repository when clean completed
    -p tree     -- Which ports tree to use for packages
    -R          -- Clean RESTRICTED packages after building
    -v          -- Be verbose; show more information. Use twice to enable
                   debug output
    -y          -- Assume yes when deleting and do not confirm
    -z set      -- Specify which SET to use for packages
root@mowa219-gjp4-zbook-freebsd:~ # poudriere pkgclean -a -j 13-0-release
[00:00:00] Gathering all expected packages
[00:00:00] Creating the reference jail... done
[00:00:57] Mounting system devices for 13-0-release-default
[00:00:57] Mounting ccache from: /var/cache/ccache
[00:00:57] Mounting ports from: /usr/local/poudriere/ports/default
[00:00:57] Mounting packages from:
[00:00:57] Mounting distfiles from: /usr/ports/distfiles
[00:00:57] Copying /var/db/ports from: /usr/local/etc/poudriere.d/options
[00:00:57] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/13-0-release-default/ref/etc/resolv.conf
[00:00:57] Starting jail 13-0-release-default
[00:00:58] Loading MOVED for /usr/local/poudriere/data/.m/13-0-release-default/ref/usr/ports
[00:00:58] Ports supports: FLAVORS SELECTED_OPTIONS
[00:01:23] Gathering ports metadata
[00:05:42] Calculating ports order and dependencies
[00:06:08] Sanity checking the repository
[00:06:08] Unqueueing existing packages
[00:06:10] Sanity checking build queue
[00:06:12] Looking for unneeded packages
[00:06:12] Calculating size for found files... done
[00:06:12] These stale packages will be deleted:
/usr/local/poudriere/data/packages/13-0-release-default/.latest/All/bsdisks-0.26.txz
/usr/local/poudriere/data/packages/13-0-release-default/.latest/All/lsblk-3.6.txz
/usr/local/poudriere/data/packages/13-0-release-default/.latest/All/webcamd-5.13.2.6.txz
[00:06:12] Removing these 3 stale packages will free: 1.50 KiB
[00:06:12] Proceed? [y/N] y
[00:17:54] Removing files... done
[00:17:54] Deleting stale symlinks... done
[00:17:54] Deleting empty directories... done
[00:17:54] Creating pkg repository
Creating repository in /tmp/packages: 100%
Packing files for repository: 100%
[00:18:01] Cleaning up
13-0-release-default: removed
13-0-release-default-n: removed
[00:18:01] Unmounting file systems
root@mowa219-gjp4-zbook-freebsd:~ #
```

Packages remain: 


```
% du -hs /usr/local/poudriere/data/packages/13-0-release-default/All/
130M    /usr/local/poudriere/data/packages/13-0-release-default/All/
%
```


----------



## ssw01 (Aug 13, 2021)

grahamperrin said:


> It should not take so long with binaries.
> 
> 
> ```
> ...


I'm having some difficulty finding info on exactly what the "-b" option does.  I find the man page
description a bit on the ambiguous side.  Is there a doc somewhere that goes into more detail?

Suppose my list of ports includes www/firefox and I have non-standard options set for it
(which is, of course, my main reason for using poudriere in the first place).  If I run
`poudriere bulk -f <list of ports> -j <jail name> -b latest`, will poudriere pull down binary
packages for all dependencies but then build firefox locally according to my options?

If the answer to the above is "yes". then if I have

```
OPTIONS_UNSET+=<stuff I don't want>
```
in /usr/local/etc/poudriere.d/make.conf, is the system smart enough to only pull
down packages that don't conflict with that?


----------



## grahamperrin@ (Aug 13, 2021)

Lazily:

`poudriere bulk -b latest -j r130amd64full -Ct www/firefox`

Downside of the laziness:



```
-t       Add some testing to the specified ports.  Add -r to recursively
              test all port dependencies as well.  Currently, the behavior is
              to uninstall a port, and to disable parallel jobs for make(1).
              When used with -a then -rk are implied.
```

If you want parallel jobs, edit a .conf accordingly then (instead):

`poudriere bulk -b latest -j r130amd64full -C www/firefox`

*Postscript*: see below …


----------



## bdrewery@ (Aug 13, 2021)

I have just pushed this change to Poudriere git (not ports yet) to clarify what `-b` will fetch. It's smart within reason.



> -b name  Specify the name of the binary package branch to use to prefetch packages.  Should be "latest", "quarterly", "release_*", or url.  With this option poudriere will first try to fetch from the binary
> package repository specified the binary package prior to do the sanity check if the package does not already exist.
> 
> poudriere will only use packages that:
> ...



During the migration of pkg from .txz to .pkg it may fetch some packages that it then does not use. This isn't worth documenting or coding for as it is a temporary period.

About the `-` issue I've also blocked creating jails and ports trees with '-' and warn for set names. It doesn't necessarily cause build problems unless the various names have overlap, but it can break `poudriere status` parsing.

As for this...


> -t       Add some testing to the specified ports.  Add -r to recursively
> test all port dependencies as well.  Currently, the behavior is
> to uninstall a port, and to disable parallel jobs for make(1).
> When used with -a then -rk are implied.


It is not correct. `-t` has no direct impact on whether or not make jobs are used. `bulk` does that normally unless a package is in `ALLOW_MAKE_JOBS_PACKAGES`. `testport` on the other hand *enables* make jobs for the port being tested.
I've added this blurb to poudriere-bulk(8):



> poudriere will disable make jobs for packages not listed in ALLOW_MAKE_JOBS_PACKAGES, unless ALLOW_MAKE_JOBS is set in poudriere.conf.  The number of jobs can be controlled in make.conf with the ports(7)
> flags MAKE_JOBS_NUMBER and MAKE_JOBS_NUMBER_LIMIT.  Beware that there is no global job control so each builder created from -J (or the default PARALLEL_JOBS) will spawn as many jobs as the ports framework
> allows.


And this to poudriere-testport(8):


> poudriere will allow make jobs for the port being tested.  See poudriere-bulk(8) for more details on make jobs.


----------

