# pkg repo



## balanga (May 8, 2021)

I'm thinking about setting up my own local pkg repository and was advised to look at pkg-repo() but am not very clear about how it works. Do you need to assemble a load of pkgs first in somewhere like /var/cache/pkg and then run this utility? What does it output?


----------



## SirDice (May 8, 2021)

balanga said:


> What does it output?


It generates the required meta.conf, meta.txz and packagesite.txz files that are required for pkg-update(8) to get information from that repository.


----------



## balanga (May 9, 2021)

So when I've run pkg-repo() what do I need to do to make `pkg install` try to retrieve pkgs from this repository?

I saw a post which suggested using a web server to make pkgs accessible but not sure if that is the way to do it...


----------



## SirDice (May 9, 2021)

You need to set up a custom repository in /usr/local/etc/pkg/repos/. While a web server is certainly the easiest solution if you have more hosts on your network, you can actually use any kind of URL, even file://:

```
my-repo: {
  enabled: yes
  url: file:///some/local/directory
}
```
See pkg.conf(5).


----------



## jmos (May 9, 2021)

Basically you can do this:

Create all packages you've got installed on your package server with `pkg create -a` inside the desired directory (unfortunately you'll get all packages in one directory). Root shouldn't be necessary, but some backages will fail without…

Afterwards run `pkg repo path/to/that/dir/` - your meta files will be created.

Then set up a simple webserver, f.e. lighttpd: You configuration could look like this:


```
server.document-root = "/path/to/dir"
server.port = 1120
server.dir-listing = "enable"
mimetype.assign = ()
```

Note the high port number I've used that makes it possible to run the server without root. Now start your webserver: `lighttpd -f configfile.conf`

(The last weeks I began writing and testing a simple, but more comfortable tool to turn a given machine into a package server as easy as possible and with low dependencies.)


----------



## balanga (May 9, 2021)

SirDice said:


> You need to set up a custom repository in /usr/local/etc/pkg/repos/. While a web server is certainly the easiest solution if you have more hosts on your network, you can actually use any kind of URL, even file://:
> 
> ```
> my-repo: {
> ...


Once I've done that, is it a matter of imply `pkg install pkgname` and pkg will be directed by via the directive in /usr/local/etc/pkg.conf...?

Not entirely sure where your my_repo is located or how pkg finds it... or is your example part of your pkg.conf?


----------



## SirDice (May 9, 2021)

balanga said:


> s it a matter of imply `pkg install pkgname` and pkg will be directed by via the directive in /usr/local/etc/pkg.conf...?


Put your custom repos in /usr/local/etc/pkg/repos/<repo>.conf. Then check with `pkg -vv`:

```
Repositories:
  dicelan-server: {
    url             : "file:///usr/local/poudriere/data/packages/13-stable-server/",
    enabled         : yes,
    priority        : 0
  }
```
Disable the FreeBSD repo by creating a /usr/local/etc/pkg/repos/FreeBSD.conf:

```
FreeBSD: {
  enabled: no
}
```


----------



## balanga (May 16, 2021)

I've created a test-repo.conf and copied /var/cache/pkg/* into it just to get a feel for how this works, and now when I attempt something like`pkg search lfm` I get:-


```
pkg: Repository test-repo missing. 'pkg update' required
pkg: wrong architecture: FreeBSD:11:amd64 instead of FreeBSD:13:amd64
pkg: repository test-repo contains packages with wrong ABI: FreeBSD:11:amd64
```

Do I need to sort out these pkgs into different repos? How does FreeBSD handle different ABI's?


----------



## Alain De Vos (May 16, 2021)

With poudriere it works. How did you created the repo ?


----------



## balanga (May 16, 2021)

I just copied /var/cache/pkg/* from some system to it. Don't really know how a repo is supposed to look....


----------



## Alain De Vos (May 16, 2021)

There is a certain structure.
It should contain directories All,Latest
It should countain meta.conf, meta.txz, packagesite.txz.
Why and how I don't know. Hope to find out.
meta.conf contains how it is packaged.
packagesite.txz is a large yaml of all the packages available in the repo.
It contains general fields like, name,size, etc ...
The packages are stored as .txz file in directory All.
Copying /var/cache/pkg is not enough. You miss the meta information.
It's like a reduced version of only the directory All

Don't you need compat for different abi's ?


----------



## balanga (May 16, 2021)

I don't know how all this is structured... just learning...

pkg-repo() creates the meta.conf, meta.txz packagesite.txz, all of which I have, but obviously something is missing.


----------



## ShelLuser (May 16, 2021)

A really easy ("quick and dirty") way to set up a repository would be....

`# pkg create -ao /usr/ports/packages/All`
`openssl genpkey -algorithm RSA -out private.key`
`openssl pkey -in private.key -out public.key -pubout`
`# pkg repo /usr/ports/packages private.key`
(as a sidenote... openssl isn't necessarily getting better; in the 'old days' you could create both a private _and_ public key in one go)

Anyway... this would grab all your installed packages and publish them into /usr/ports/packages/All. Then you create the repository manifest and sign that with your RSA key.

The next step would be to copy your public key onto the other server(s) which need to use your repo and then inform them about the repo and specify the key (using "PUBKEY" in your config file).

As said: this is a very bare and "dirty" way, but it'd work.[/file]


----------



## balanga (May 16, 2021)

This doesn't sound like 'bare minimum' to me  . I'm using
*signature_type: "none",*
and don't need any keys, that just gets in the way for me.


I don't really see what I've done differently apart from not creating the All directory. I'm not building any pkgs myself, I just want a single local repository on my LAN, so I don't need to keep fetching the same pkgs off the Internet.


----------



## SirDice (May 17, 2021)

balanga said:


> I don't really see what I've done differently apart from not creating the All directory.


You still need that directory structure. The packages should go in an All directory. 


```
All/  # Contains the packages
meta.conf
meta.txz
packagesite.txz
```


----------



## jmos (May 17, 2021)

SirDice said:


> You still need that directory structure. The packages should go in an All directory.


Package file can be in structured in subdirectories - the pkg-repo command crawls through them, and adds the paths to the meta files. I'm splitting my packages in subdirs as the ports tree does, and: It works. There's no need for an "ALL" directory, you can use whatever you want. Example:


```
jo@freya ~>  mkdir mypackages
jo@freya ~>  mkdir mypackages/misc
jo@freya ~>  pkg create mc-nox11
Creating package for mc-nox11-4.8.26
jo@freya ~>  mv mc-nox11-4.8.26.txz mypackages/misc/
jo@freya ~>  pkg repo mypackages/
Creating repository in mypackages/: 100%
Packing files for repository: 100%
```

Now I can use the "mypackages" directory for a package server.


----------



## SirDice (May 17, 2021)

jmos said:


> There's no need for an "ALL" directory, you can use whatever you want.


All is often used because the old package repository structure used it. But yes, it can be any name nowadays. There does need to be _a_ directory:

```
To create a package repository catalogue, specify the top-level directory
     beneath which all the packages are stored as repo-path.  pkg repo will
     search the filesystem beneath repo-path to find all the packages it
     contains.  Directories starting with ‘.’ or named Latest are not
     traversed.
```


----------



## Alain De Vos (May 17, 2021)

Shouldnt you be able to set up a pkg repository without using openssl or openssl installed ?
I remember using linux and this key thing getting out of hand.


----------



## jmos (May 17, 2021)

Alain De Vos said:


> Shouldnt you be able to set up a pkg repository without using openssl or openssl installed ?


Yes, it is optional (f.e. as my own network is trusted I don't use it).


----------



## balanga (May 17, 2021)

I think that depends on what you have fo signature_type, eg:=

*signature_type: "fingerprints",*


----------



## balanga (May 17, 2021)

Is it OK to lump together pkgs for different releases of FreeBSD including amd64 and i386 because these errors suggest that pkgs for different releases should be segregated in some way.



> pkg: Repository test-repo missing. 'pkg update' required
> pkg: wrong architecture: FreeBSD:11:amd64 instead of FreeBSD:13:amd64
> pkg: repository test-repo contains packages with wrong ABI: FreeBSD:11:amd64



I have simply lumped together all the pkgs that have been installed over the last couple of years during which time FreeBSD has been updated, but I would still like to service requests from systems that are running older versions.


----------



## balanga (May 17, 2021)

To reiterate what I have...

Client 

/usr/local/etc/pkg/repos/test-repo.conf:-


```
test-repo: {
    url: "file:///mnt/pkg-repo/",
    signature_type: "none",
    enabled: yes
}
```

Server

The repository is in:-

/var/tmp/pkg

This contains copies of all the *.txz files that are in /var/cache/pkg/ and in addition has packagesite.txz meta.txz meta.conf which were produced when I ran `pkg repo`.

There are no directories in this directory. Do I need to create an All directory and move all the *.txz files there?

Getting a bit confused by all this...


----------



## jmos (May 18, 2021)

balanga said:


> Is it OK to lump together pkgs for different releases of FreeBSD including amd64 and i386


That can't work. The package system expects them to be in their own repository: amd64 for FreeBSD 11 has to be an own repository as well as i386, and both again for FreeBSD 12 (and 13). F.e. there's a reason why you have to rebuild or upgrade all packages after a major FreeBSD update (and that doesn't mean "install the old packages again", but "install new packages that are not build for the old major version"). Also you cannot have two different versions of one package in one repository.


----------



## balanga (May 18, 2021)

So what should the directory structure look like?

Like this?

repo/
       i386/
               10.0/
               11.0/
               12.0/
               13.0/
       amd64/
                  10.0/
                  11.0/
                  12.0/
                  13.0/

and should pkg-repo() be run in each directory


----------



## SirDice (May 18, 2021)

To give you an example:

```
root@molly:~ # ll /usr/local/poudriere/data/packages/
total 90
drwxr-xr-x  7 root  wheel  15 May 15 15:06 122-release-server/
drwxr-xr-x  7 root  wheel  15 May 15 11:22 13-stable-server/
drwxr-xr-x  7 root  wheel  15 May 15 23:25 130-release-desktop/
drwxr-xr-x  7 root  wheel  15 May 15 14:18 130-release-server/
lrwxr-xr-x  1 root  wheel  18 Nov  7  2020 FreeBSD:12:amd64@ -> 122-release-server
lrwxr-xr-x  1 root  wheel  18 Feb 15 21:24 FreeBSD:13:amd64@ -> 130-release-server
```


```
root@molly:~ # ll /usr/local/poudriere/data/packages/130-release-desktop/
total 118
lrwxr-xr-x  1 root  wheel  18 Feb 11 17:14 .buildname@ -> .latest/.buildname
lrwxr-xr-x  1 root  wheel  20 Feb 11 17:14 .jailversion@ -> .latest/.jailversion
lrwxr-xr-x  1 root  wheel  16 May 15 23:25 .latest@ -> .real_1621113913
drwxr-xr-x  4 root  wheel   9 May 10 00:25 .real_1620599154/
drwxr-xr-x  4 root  wheel   9 May 11 01:21 .real_1620688887/
drwxr-xr-x  4 root  wheel   9 May 12 08:19 .real_1620800359/
drwxr-xr-x  4 root  wheel   9 May 14 16:27 .real_1621002475/
drwxr-xr-x  4 root  wheel   9 May 15 23:25 .real_1621113913/
lrwxr-xr-x  1 root  wheel  11 Feb 11 17:14 All@ -> .latest/All
lrwxr-xr-x  1 root  wheel  14 Feb 11 17:14 Latest@ -> .latest/Latest
lrwxr-xr-x  1 root  wheel  17 Feb 11 17:14 meta.conf@ -> .latest/meta.conf
lrwxr-xr-x  1 root  wheel  16 Feb 11 17:14 meta.txz@ -> .latest/meta.txz
lrwxr-xr-x  1 root  wheel  23 Feb 11 17:14 packagesite.txz@ -> .latest/packagesite.txz
```


```
root@molly:~ # ll /usr/local/poudriere/data/packages/130-release-desktop/All/
total 3626564
-rw-r--r--  1 root  wheel    7539288 May 15 15:22 ImageMagick6-6.9.11.6_3,1.txz
-rw-r--r--  1 root  wheel    2957088 May 15 15:39 RetroArch-1.9.0.txz
-rw-r--r--  5 root  wheel     141928 Apr 16 20:03 aalib-1.4.r5_13.txz
-rw-r--r--  2 root  wheel    9191964 May 14 06:45 adwaita-icon-theme-3.38.0.txz
-rw-r--r--  3 root  wheel     455708 May 12 02:34 alsa-lib-1.2.2.txz
-rw-r--r--  5 root  wheel    3456612 May  5 02:30 aom-3.1.0.txz
-rw-r--r--  2 root  wheel     480564 May 14 01:37 apr-1.7.0.1.6.1_1.txz
-rw-r--r--  5 root  wheel      34496 Apr 16 13:24 argp-standalone-1.3_4.txz
{... and a whole bunch more, I won't bore you with a complete list ..}
```

I don't have i386 repositories because I have no 32 bit machines and thus no use for 32 bit packages.


----------



## balanga (May 18, 2021)

Wow!

I'm trying figure out exactly how to make a mini version of this..

Presumably your /usr/local/etc/pkg/repos/my-repo.conf contains a URL pointing at /usr/local/poudriere/data/packages/ where you have the directories you mentioned such as /130-release-desktop/ and in that you have /All/ where you put your pkgs. So do you run pkg-repo ()on every subdirectory? And when a client runs `pkg install` how does the server know which directory to get its pkgs from? Is that via /usr/local/etc/pkg/repos/my-repo.conf on the client?

When you normally run `pkg install`, FreeBSD pkg server decides which version to give you without you specify the ABI, although that is probably done behind the scenes and does not need to be done explicitly..

Apologies for so many questions. Enquiring minds need to know


----------



## Alain De Vos (May 18, 2021)

When you run "pkg install" it goes first looking to all files in /etc/pkg and then in /usr/local/etc/pkg/
The contents of these files contain the info on where and how to find the packages.
A line like "url: "file:///poudriere/data/packages/poujail-pouports" says it is on this pc with this path.


----------



## balanga (May 18, 2021)

Looking at /etc/pkg/FreeBSD.conf it says:-



> # To disable this repository, instead of modifying or removing this file,
> # create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
> #
> #   mkdir -p /usr/local/etc/pkg/repos
> #   echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf


So it must first look to see if such a file as /usr/local/etc/pkg/repos/FreeBSD.conf exits

There is also /usr/local/etc/pkg.conf which may or may not exist.

Somewhere along the line an ABI is passed to the pkg server which must direct it to the corresponding directory to service this ABI. Don't know how or if the repository should be structured to cope with the ABI.


----------



## Alain De Vos (May 18, 2021)

There is a variable ${ABI}. I don't know where this variable is defined. But it is a good question.
To print your ABI:

```
pkg -vv | grep ABI
```


----------



## SirDice (May 18, 2021)

balanga said:


> Presumably your /usr/local/etc/pkg/repos/my-repo.conf contains a URL pointing at /usr/local/poudriere/data/packages/ where you have the directories you mentioned such as /130-release-desktop/ and in that you have /All/ where you put your pkgs. So do you run pkg-repo ()on every subdirectory? And when a client runs `pkg install` how does the server know which directory to get its pkgs from? Is that via /usr/local/etc/pkg/repos/my-repo.conf on the client?


As you can tell by the directory this is what poudriere produces. But that's not relevant in this case. I have that /usr/local/poudriere/data/packages/ directory shared with a webserver as http://ports.myserver/packages. Each of those sub directories is a different repository. So on my 13.0 desktops I point my-repo.conf to http://ports.myserver/packages/130-server-desktop/, my 12.2 servers point to http://ports.myserver/packages/122-release-server/, etc.



Alain De Vos said:


> There is a variable ${ABI}. I don't know where this variable is defined. But it is a good question.


It's generated in the pkg(8) code itself. It's based on the information of the machine it's running on. So on a 13.0-RELEASE AMD64 you get `FreeBSD:13:amd64`, on a 12.2-RELEASE i386 you get `FreeBSD:12:i386`, etc. If you use that `${ABI}` variable in the repository URL it will automatically get translated by pkg(8). That makes it easy to have the "same" URL for different versions and architectures in the config. I've added those symlinks in my http://ports.myserver/package directory so I can use it for my own repositories too and have it "automagically" switch to the correct repository for that version after I upgraded a machine from 12.2 to 13.0 for example. If I had used the "hardcoded" http://ports.myserver/packages/122-release-server/ I would have needed to modify it or else it's going to try to install 12.2 packages on a 13.0 machine after I upgraded it.


----------



## Alain De Vos (May 18, 2021)

Indeed. By using this ABI variable you can easily set up your own different repositories while having the same config file on the different machines.


----------



## balanga (May 28, 2021)

Another attempt to get this working... I now have a bare-bones system, plain FreeBSD (13.0) no pkgs and would like to see if I can get anywhere with my pkg repo.

Now that I think I've set it up, I understand I need to run`pkg -vv` to see if the repo is recognised... but... after running that I get:-



> The package management tool is not yet installed on your system.
> Do you want to fetch and install it now? [y/N]: y
> Bootstrapping pkg from file:///mnt/pkg-repo, please wait...
> pkg: Error fetching file:///mnt/pkg-repo/Latest/pkg/txz: No such file or directory
> ...



Now if I have already used pkg somewhere, can I expect to find a copy of pkg.txz somewhere on my system, ie in /var/cache/pkg for example?


----------



## jmos (May 29, 2021)

balanga said:


> Now if I have already used pkg somewhere, can I expect to find a copy of pkg.txz somewhere on my system, ie in /var/cache/pkg for example?


Depends. When you're using packages there's a package file by default, but not named as "pkg.txz":

```
root@freya ~>  pkg clean -a
[…]
root@freya ~>  pkg install -f pkg
[…]
root@freya ~>  ls /var/cache/pkg/
total 8576
lrwxr-xr-x  1 root  wheel    25B 29.05.2021 17:55 pkg-1.16.3.txz -> pkg-1.16.3~035fc218ad.txz
-rw-r--r--  1 root  wheel   8.3M 01.01.1970 01:00 pkg-1.16.3~035fc218ad.txz
```


----------



## balanga (May 29, 2021)

I haven't yet fathomed how `pkg install pkg` manages to retrieve pkg-1.16.3.txz, but I believe that`packagsite.yaml` provides some sort of index for want you want and what is available.

As far as building a local pkg repository I think I maybe need to have separate repos for FreeBSD versions 11, 12 and 13 as well as corresponding ones for i386 and amd64. Originally I thought I could lump them all into one repository but that doesn't seem to work.


----------



## balanga (Jun 13, 2021)

I finally managed to make some progress, at least I've managed to install a couple of pkgs. These are not pkgs built by me but copied from various /var/cache/pkg/ directories. 

In one instance I'm trying to install misc/pdmenu. One of the dependencies is png: 1.6.37 and I do have a png-1.6.37~c6ad91dc43.txz in my repo, and there is also a linked file in /var/cache/pkg.

Is there a way to create these linked files in the repo? Does pkg-repo()do that?

I'm coming back to a directory I created over a year ago before getting fully familiarised with pkg repositories.


----------



## jmos (Jun 13, 2021)

balanga said:


> Is there a way to create these linked files in the repo?


As already wrote: `pkg create png` creates a package for you. You can also create all packages with a simple `package create -a`. An alternative (not) to handle things by your own could be a test with my own tool: https://jmos.net/software/maaspare.rvt


----------



## balanga (Jun 14, 2021)

I think I have managed to create the backend of my pkg-repo now, ie I can run `pkg install -y xyz` and xyz will be installled, but I only have a couple of pkgs in my repo at the moment. What I would like to do is populate it with the contents of various /var/cache/pkg/ directories on various systems I have, but not sure how straightforward a process that is. Can I just copy the contents of those directories to my pkg-repo and run pkg-repo() or is there any way to check the ABI of pkgs in cache. At the moment my repo is setup as FreeBSD:12:amd64.


----------



## Jose (Jun 15, 2021)

The man page for pkg-repository(5) explains the layout in detail. Poudriere places all the packages it builds in the correct filesystem organization and creates the metadata files for you. All you have to do is serve up the the Poudriere directory using Nginx or similar.


----------



## jmos (Jun 15, 2021)

balanga said:


> What I would like to do is populate it with the contents of various /var/cache/pkg/ directories on various systems I have, but not sure how straightforward a process that is. Can I just copy the contents of those directories to my pkg-repo and run pkg-repo() or is there any way to check the ABI of pkgs in cache.


Of course you can copy & use the package files from there (the symlink name is the one you need in your package repository), but: Why? Again: You can simply create all of them in one go (instead of relying on a maybe incomplete cache). Also you won't get a lolly if you still want to mix ABIs into the same repository (you already know that the pkg command complains about that).


----------

