# Pkg package repository using ports-mgmt/poudriere. With or without ZFS.



## kpa (Apr 3, 2013)

This is a HOWTO on how to build your own ports-mgmt/pkg packages in a jail(8) using ports-mgmt/poudriere.

Few notes about notation:

First few mentions of a ported program will be a "port link", like net/mtr. I will not repeat the link too many times however.
I will use a link to a manual page when introducing a new command, like mtr(8). Unfortunately the official manual page are lagging badly behind what's actually available on a recent FreeBSD and many of the links below don't work. You can however take the name of the command from the link and check the manual page on your own system that will have up to date manual pages.
I will use a couple of variables to avoid repeating paths like /usr/local/poudriere. I will use ${LOCALBASE} for the path of the default location for installed ports, /usr/local. I will use ${BASEFS} for the root folder where poudriere stores ports trees, jails, packages and few other things, default ${LOCALBASE}/poudriere (but see below for the BASEFS setting in the configuration file). I will use ${jailname} in few places where file names vary depending on the name of jail used.

What is  ports-mgmt/poudriere? The port description reads as follows:


```
poudriere is a tool primarily designed to test package production on
FreeBSD. However, most people will find it useful to bulk build ports
for FreeBSD.

WWW: https://github.com/freebsd/poudriere/wiki
```

This description is now quite out of date since the official FreeBSD binary packages are now built with it. Poudriere is a package building system intended for anyone who wants to build their own binary pkg packages for FreeBSD. Its design is as follows:

Mostly sh(1) scripts for portability.
Uses jail(8)s for creating clean build environments for ports building.
Makes heavy use of nullfs(5) to avoid unnecessary copying.
Makes heavy use of ZFS features like snapshots and cloning if ZFS is used.
Has built-in utilities for setting up and updating ports(7) trees and jail(8)s.
Customizable trough a configuration file (${LOCALBASE}/etc/poudriere.conf, ${LOCALBSE} defaults to /usr/local) and a dedicated settings directory (${LOCALBASE}/etc/poudriere.d).
Can be run as a daemon that serves build requests.

Why is poudriere needed in the first place? The ports infrastructure suffers from numerous design faults that are not easy to overcome without a package building tool like poudriere. These faults include:

Building of ports as outlined in ports(7) is performed on the live system. Many build failures stem from this when an already installed port or leftover files from an old version causes conflicts with the to be built new version of the port. Poudriere avoids this problem  by starting every build from a clean state as if the ports were installed for the first time on the system.
Building ports on the live system pulls in a large number of build time dependencies that are not required to be installed after the builds have finished. These build time dependencies can cause configuration issues and trigger build failures when port dependencies and options change. Again, poudriere avoids this by starting every build from a clean state.
Building ports on the live system may cause service interruptions while the ports are being built and the situation may not be corrected until all of the ports have been rebuilt. Poudriere uses jails that are separated from the live system for building the packages and therefor can not disrupt the running services on the host.
There might not always be an easy update path when an important port gets updated because of complicated dependencies between the updated port and its dependents. Usually in such cases there are instructions in the /usr/ports/UPDATING file to remedy the situation but to apply the instructions manual intervention is needed. Poudriere nullifies the need for manual intervention and the need to follow the instructions in first place because the repository gets rebuilt starting from the lowest level of dependencies and all dependents of the changed port/ports that need rebuilding are properly rebuilt resulting in a consistent repository that will result in a consistent system when the updated packages get installed.
When building individual ports one by one with `# make install` or for example ports-mgmt/portmaster there are no opportunities to make use of the conflict resolution feature of the new pkg system. It is only available when installing sets of packages from a remote repository with pkg-install(8) or pkg-upgrade(8).
Building for a different target system that uses different defaults and options can be next to impossible because of conflicts between already installed ports and the build time dependencies the differing options would pull in to be installed at the same time
Build once, distribute to many client hosts is cumbersome to implement with only the basic ports system.

What poudriere is not:

It's not a replacement for any of the existing tools in the FreeBSD base or the ports(7) system. It doesn't modify the standard ports infrastructure in any way to achieve what it does.
It's not in competition with the few other ports building tools like ports-mgmt/portmaster or ports-mgmt/portupgrade because its goals are different. The two mentioned tools are nice frontends to the ports system but are limited to building the ports on the live system just as the basic ports system is.
It can't be used as a generic jail(8) management system like sysutils/ezjail. The jails created by poudriere are only suitable for building ports into binary packages.

Requirements:

Any supported version of FreeBSD installed on the build host system, amd64 or i386. Amd64 very much recommended over i386. The so called stable versions stable/10 and stable/11 are supported. The head branch a.k.a 12-CURRENT should also work also but see here first. Operating system versions installed in the jails used to build packages should not be more recent than the operating system installed on the build host system, jails that have older operating system versions work as long as the kernel on the build host has the proper compatibility options set.
64- and 32-bit versions can be mixed but only in a way where the build host system is a 64-bit version and the jail has a 32-bit version of FreeBSD. The kernel on the build host must include the 32-bit compatibility layer for this to work.
An up to date version of ports-mgmt/poudriere, latest version is 3.2.7 at the time of the last edit of this HOWTO. The development version ports-mgmt/poudriere-devel can be also used if bleeding edge features are needed.
An up to date version of ports-mgmt/pkg, latest version is 1.10.5 at the time of the last edit.
Enough memory to build the largest packages. This depends on what you're building, things like www/firefox may take lots of memory. 4GBs or more to be safe (amd64 version of FreeBSD needed if you intend to utilize more that 4GBs of memory).
Enough free disk space. As above, depends on what you're building. 10GBs or more to be safe.
Support for the ZFS filesystem on the build host. ZFS makes jail cloning very fast and efficient and in general should be used if possible. I will note below the settings that need to be changed in case you don't want to use ZFS for some reason.
The kernel on the build host must have support for jail(8) and nullfs(5).

Before proceeding take note:

All of this assumes that you have a working internet connection right from the start.
Commands listed as `# command` should be run with root privileges. Other commands work without the root privileges.
*1. Install ports-mgmt/pkg.*

This step is only needed on a freshly installed system that has no ports-mgmt/pkg installed yet.

On any supported version of FreeBSD there is bootstrap utility for pkg installed as /usr/sbin/pkg that can be used to install ports-mgmt/pkg. Simply run this on a freshly installed system and you'll be prompted if you want to install pkg(8):

`# pkg`


*2. Install ports-mgmt/poudriere.*

Install poudriere from the official binary packages:

`# pkg install -y poudriere`

If you need any of the so called bleeding edge features of poudriere, install the ports-mgmt/poudriere-devel port instead.

*3. Configure ports-mgmt/poudriere.*

First copy the sample configuration file ${LOCALBASE}/etc/poudriere.conf.sample to ${LOCALBASE}/etc/poudriere.conf and make sure the copy is writable:

`# cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf`
`# chmod 640 /usr/local/etc/poudriere.conf`

Edit the following entries in ${LOCALBASE}/etc/poudriere.conf (some already have the desired values, leave them untouched):


```
#### ZFS
# The pool where poudriere will create all the filesystems it needs
# poudriere will use ${ZPOOL}/${ZROOTFS} as its root
#
# You need at least 7GB of free space in this pool to have a working
# poudriere.
#
ZPOOL=zroot

### NO ZFS
# To not use ZFS, define NO_ZFS=yes
#NO_ZFS=yes

# root of the poudriere zfs filesystem, by default /poudriere
ZROOTFS=/poudriere

# the host where to download sets for the jails setup
# You can specify here a host or an IP
# replace _PROTO_ by http or ftp
# replace _CHANGE_THIS_ by the hostname of the mirrors where you want to fetch
# by default: ftp://ftp.freebsd.org
#
# Also note that every protocols supported by fetch(1) are supported here, even
# file:///
# Suggested: https://download.FreeBSD.org
FREEBSD_HOST=https://download.FreeBSD.org

# The directory where poudriere will store jails and ports
BASEFS=/usr/local/poudriere

# If set the given directory will be used for the distfiles
# This allows to share the distfiles between jails and ports tree
# If this is "no", poudriere must be supplied a ports tree that already has
# the required distfiles.
DISTFILES_CACHE=/var/cache/distfiles

# If set the ports tree or source tree marked to use svn will use the defined
# mirror (default: svn.FreeBSD.org)
# The SSL fingerprints are published here:
# https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html#svn-mirrors
SVN_HOST=svn.FreeBSD.org

# Disable linux support
NOLINUX=yes
```

Create the /var/cache/distfiles directory, you can put this somewhere else as well but this path is the most compliant one with hier(7):

`# mkdir /var/cache/distiles`

The comments in the above are verbatim from the sample configuration file, I have chosen to disable Linux support because I don't build anything that requires the Linux emulation layer. There are a number of different configuration options that I'm leaving uncovered here because the defaults are fine for most cases, refer to the configuration file and the documentation for more information.

If you want to disable the use of ZFS leave the ZPOOL and ZROOTFS lines commented out and uncomment the NO_ZFS=yes line. None of the steps below need to be modified if ZFS is not used, poudriere will use alternate methods for jail creation/cloning and handling of ports trees and package repositories that are somewhat slower than with ZFS but the end results are the same.


*4. Set up the ports tree for poudriere.*

Setting up and managing of ports(7) trees is done with the poudriere-ports(8) command.

This will create a new ports tree located at ${BASEFS}/ports/default using the SVN method of checking out and updating of the ports tree:

`# poudriere ports -c -m svn`

After this running `poudriere ports -l` should output:


```
PORTSTREE METHOD TIMESTAMP           PATH
default   svn    2018-08-16 14:29:52 /usr/local/poudriere/ports/default
```

It's now possible to use poudriere-ports(8) to update the ports tree:

`# poudriere ports -u`

It's also possible to register an existing ports tree into poudriere, however this will leave the updating of this ports tree for you to handle:

`#  poudriere ports -c -f none -m null -M /usr/ports -p test`


*5. Set up the jail for building packages.*

The command to set up and manage jails in poudriere is poudriere-jail(8). The simplest way to create a jail is to use the http method (note: the manual page will claim that http is the default but the default is in fact ftp) to fetch a FreeBSD distribution file set to be extracted in the new build jail. The http method will use the FREEBSD_HOST setting from the configuration file as the mirror site for downloading the necessary files. HTTPS is supported.

This is how an amd64 11.2-RELEASE jail would be created on an amd64 system, substitute amd64 with i386 if creating an i386 jail on an i386 build host:

`# poudriere jail -c -m http -v 11.2-RELEASE -j releng112amd64`

The jail identifier for the -j option is for you to decide, I used "releng112amd64" to signify that it's 11.2-RELEASE but can be updated with release engineering updates and security fixes. In other words the jail follows the official 11.2-RELEASE branch with updates.

After creation the jail should be in the `poudriere jail -l` listing:

```
JAILNAME       VERSION                 ARCH  METHOD       TIMESTAMP           PATH
releng112amd64 11.2-RELEASE-p1        amd64 http         2018-08-16 18:21:38 /usr/local/poudriere/jails/releng112amd64
```

Creating an i386 jail on an amd64 build host is done as follows:

`# poudriere jail -c -m http -a i386 -v 11.2-RELEASE -j releng112i386`

If you're tracking a stable or current branch of FreeBSD the easiest way to keep your poudriere jails up to date is to use the -m src=/usr/src method for creating and updating the jails. This is how a stable/11 amd64 jail would be created, it is assumed that `make buildworld buildkernel` has been done in /usr/src and the build host has been updated to use the new version of the OS with `make installkernel installworld` followed by a reboot.

`# poudriere jail -c -m src=/usr/src -v stable/11 -j stable11amd64`

It is also possible to use the -m svn method that fetches the sources for the requested version of FreeBSD, builds a new world using those sources and installs the results of the build as the new jail.

Jails that were installed using the -m ftp or -m http method are updated using freebsd-update(8). Jails that were installed with the -m src=/usr/src method are updated using the sources and objects in /usr/src and /usr/obj of the build host. Jails that were installed using the -m svn method that involves building a world, a new world build is performed with the updated sources.
Updating of a jail regardless of the method of their creation is done with:

`# poudriere jail -u -j releng112amd64`


*6. Create a list of ports to be built into packages.*

Create a new file, for example /usr/local/etc/ports.txt (can be also located at your home directory just as well) and put all the ports you want to build in there.

Example:


```
ports-mgmt/pkg
devel/git
devel/subversion
editors/vim
www/apache22
```

There is no need to put the dependencies of the built ports in the list, they are built automatically.

Note that you should include the ports-mgmt/pkg port in the list because there are no ports that depend on it. If you don't include it in your list of ports to be built it won't be updated when there's a new version of it.

IMPORTANT: In order to be sure that your repository stays in a consistent state, always try to run poudriere(8) with this full list of ports to build. If you build only a port here, port there, only parts of the repository are updated and many ports that should have been recompiled are not recompiled and that can lead to shared library mismatches.


----------



## kpa (Apr 3, 2013)

*7. Customize the build settings.*

Poudriere can use a global make.conf or a jail specific ${jailname}-make.conf to customize build settings. These files go into ${LOCALBASE}/etc/poudriere.d/ directory. Note that the build host's own /etc/make.conf is not used at all because it's assumed to be "tainted" by build host specific settings.

Setting options for ports can be done with the global or jail specific make.conf and can be easier than using the saved options that are shown in the next section.

Here's an example what could go into the global ${LOCALBASE}/etc/poudriere.d/make.conf:


```
# No CUPS DBUS or X11. These are unset globally for every port.
OPTIONS_UNSET= CUPS DBUS X11

# Force default postgresql version
WITH_PGSQL_VER=92

# Use the port version OpenSSL where possible, note the previous way of
# using WITH_OPENSSL_PORT=yes is no longer recommended.
DEFAULT_VERSIONS+= ssl=openssl


# Examples of port specific options. Note that the prefix to be used is the
# $OPTIONS_NAME for a port. In the example below for instance 'security_pinentry' is the
# $OPTIONS_NAME for security/pinentry.

# I want ZSH completions for ports-mgmt/poudriere-devel
ports-mgmt_poudriere-devel_SET= ZSH

# security/pinentry. Sets NCURSES option, unsets the other options.
security_pinentry_SET= NCURSES
security_pinentry_UNSET= GTK GTK2 QT3 QT4

# ftp/curl. Avoid linking to base system OpenSSL libraries.
ftp_curl_SET= GSSAPI_NONE
ftp_curl_UNSET= GSSAPI_BASE GSSAPI_HEIMDAL GSSAPI_MIT
```

Note that options set in this way are not saved under /var/db/ports or anywhere else. Refer to the next section for saved options for ports.

An example for a jail specific ${LOCALBASE}/etc/poudriere.d/release93amd64-make.conf file if the jail is a 9.3-RELEASE amd64 jail. This sets clang(1) as the default compiler and CPUTYPE for Intel's core duo CPUs.


```
CPUTYPE?= core2
CC=clang
CXX=clang++
CPP=clang-cpp
```

Note that the above example is for an outdated and unsupported release of FreeBSD that didn't yet default to clang(1) as the C compiler. None of these settings are needed on currently supported releases.


*8. Setting saved options for the ports to be built*

In addition to specifying the options in the make.conf or in the jail specific ${jailname}-make.conf the options can be set interactively like you would with `make config` for a port. Setting options for the ports is done with the poudriere-options(8) command and the options are saved at ${LOCALBASE}/etc/poudriere.d/options/${OPTIONS_NAME}/options. You can make use of an existing /var/db/ports hierarchy by copying the contents of it to the ${LOCALBASE}/etc/poudriere.d/options directory. Note that poudriere-options(8) uses the config-conditional target by default that only sets options for ports that have no options set yet. Use `# poudriere options -c category/port` to force overwriting of saved options. Examples:

`# poudriere options www/apache22`

`# poudriere options -c lang/python27`

It is also possible to use a jail specific options directory. If you use the `-j jailname` option as part of the `poudriere options` command line the options will be set in ${LOCALBASE}/etc/poudriere.d/${jailname}-options directory and those options will override the default options for packages built using that jail.


*9. Building packages*

Package building is done with the poudriere-bulk(8) command. Putting together everything from the previous steps, building is done as follows:

`# poudriere bulk -f /usr/local/etc/ports.txt -j releng112amd64`

Built packages are stored at the ${BASEFS}/data/packages/releng12amd64-default directory. The -default ending signifies that the packages were built using the "default" ports tree.

Build logs are stored at the ${BASEFS}/data/logs directory in HTML format and can be viewed with a locally running web browser or published with a web server such as www/nginx.

It is possible to force rebuilding of every package in the repository by using the -c option with poudriere-bulk(8):

`# poudriere bulk -c -f /usr/local/etc/ports.txt -j releng112amd64`.

Using the -C option instead of -c rebuilds only the ports listed in the file specified by the -f option or given on the command line. For example this would forcibly rebuild the www/apache22 package and no other package:

`# poudriere bulk -C -j releng112amd64 www/apache22`


*10. Using the package repository with pkg(8)*

The packages can be installed from the local ${BASEFS}/data/packages directory with pkg-add(8).

`# pkg add /usr/local/poudriere/data/packages/releng112amd64-default/All/apache22.txz`

A better way to use the packages is to use the pkg remote repository feature (see pkg-repository(5)) by creating a repository configuration file /usr/local/etc/pkg/repos/myrepo.conf and specifying the repository address as a file:// URL (to use the local file system) in this configuration file.


```
myrepo: {
    url             : "file:///usr/local/poudriere/data/packages/releng112amd64-default",
    enabled         : yes,
    mirror_type     : NONE
}
```

Details of the syntax of this file can found in the pkg.conf(5) manual page.

Note that the repository URL does not end in /Latest like it used to be with the old pkg_* tools.

Packages can be now installed with pkg-install(8):

`# pkg install apache22`

Updating the system when updated packages have been built is done with pkg-upgrade(8):

`# pkg upgrade`

Note that `# pkg-upgrade` command updates the repository metadata automatically before any other operations, there is no need to run pkg-update(8) first (unless the automatic repository update has been turned off).

Note that you probably want to disable the use of the official package repository by creating a /usr/local/etc/pkg/repos/FreeBSD.conf file with these contents:


```
FreeBSD: {
  enabled: no
}
```


Sharing the repository to other machines can be done with a web server, FTP server, NFS etc. The only requirement is that the repository files and directories are fetch(1)able.

For example this is how the repository would be shared using www/apache22. Rough guideline what to put in the httpd.conf file:


```
...
<Directory "/usr/local/poudriere/data/packages">
    Options Indexes FollowSymLinks

    AllowOverride None

    Order allow,deny
    Allow from all
</Directory>
...

<IfModule alias_module>
Alias /packages /usr/local/poudriere/data/packages
</IfModule>
```

With those settings a client machine can access the package repository with this repository configuration (save this as /usr/local/etc/pkg/repos/myrepo.conf):


```
myrepo: {
  url: "http://webserver.mydomain.tld/packages/releng112amd64-default"
  enabled : yes,
  mirror_type : NONE
}
```


*11. RSA signing key for cryptographic verification of the repository. Optional.*

This information is largely from Glen Barber's page https://glenbarber.us/2012/06/11/Maintaining-Your-Own-pkgng-Repository.html

`# mkdir -p /usr/local/etc/ssl/keys`
`# chmod 600 /usr/local/etc/ssl/keys`
`# mkdir -p /usr/local/etc/ssl/certs`
`# openssl genrsa -out /usr/local/etc/ssl/keys/pkg.key 4096`
`# openssl rsa -in /usr/local/etc/ssl/keys/pkg.key -pubout > /usr/local/etc/ssl/certs/pkg.cert`

Add this line to /usr/local/etc/poudriere.conf:


```
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/pkg.key
```

On any system that uses the packages you'll have to copy the /usr/local/etc/ssl/certs/pkg.cert file over and put it in the same place (DO NOT COPY THE pkg.key FILE ANYWHERE!) and add these lines to the /usr/local/etc/pkg/repos/myrepo.conf:


```
signature_type  : "PUBKEY",
pubkey: "/usr/local/etc/ssl/certs/pkg.cert",
```

Now the downloaded packages are verified that they have the correct cryptographic signatures before installation.

There is also a different method for signing the packages that uses an external signing command that can be used to keep the private key separate from the package build host. Refer to pkg-repo(8) for details.


----------



## rusty (Apr 7, 2013)

Set this up today. I used http://www.allbsd.org for a specific version, ie
`# poudriere jails -c -j 9-STABLE -f none -M /export/jails/9-STABLE -m allbsd -v r248915 -a amd64`
I set things up with the 
	
	



```
USE_TMPFS=all
```
 option in /usr/local/etc/poudriere.conf. It really sped things up :e


----------



## kpa (Apr 8, 2013)

Yes, it does, except the jail setup doesn't get much faster unfortunately, that's the penalty you pay for not using ZFS. I tested 
	
	



```
USE_TMPFS=all
```
 on a machine with 4 GB of RAM and it didn't seem to choke on editors/vim.


----------



## kpa (Apr 18, 2013)

The zfs.ko problem has been fixed in ports-mgmt/poudriere-devel version 3.0.99.20130417. No need to manually edit anything anymore.


----------



## kpa (Apr 22, 2013)

Version 3.0.99.20130421 of ports-mgmt/poudriere-devel provides major speed ups to the jail cloning at the beginning of a `poudriere bulk` run when NO_ZFS is used.

http://www.freshports.org/ports-mgmt/poudriere-devel/


----------



## kpa (May 2, 2013)

I have downgraded my only remaining FreeBSD system (a firewall) to an i386 machine with only 1 GB or RAM. I can report that I can still use ports-mgmt/poudriere-devel for building packages as I did on the amd64 system that had four times the memory and somewhat faster CPU.


----------



## SirDice (Jun 28, 2013)

Works like a charm. Thank you very much. 

I did hit one little snag when using PKGNG, it doesn't seem to create the needed repo.txz and you need to run `# pkg repo /usr/local/poudriere/data/packages/release90amd64-default` by hand. Can't this be automatically generated after the build run? Or did I miss something?


----------



## kpa (Jun 28, 2013)

It should be generated automatically but here are some very recent changes to the repository format and there might be bugs in the intergration with poudriere. I think the repo.txz file is now deprecated and the repository metadata is now split into multiple files. I don't have my poudriere build system set up atm so I can't check.

Edit: I found this bit in the ports-mgmt/pkg commit log:

http://www.freshports.org/commit.php?message_id=201306240602.r5O62L91054147@svn.freebsd.org


```
- pkg repo produce a repo.txz in pkg 1.0 (legacy) fromat
```

So it looks like ports-mgmt/poudriere-devel  has been changed to produce the newer type of repository metadata that does not include the repo.txz file anymore.


----------



## SirDice (Jun 28, 2013)

kpa said:
			
		

> It should be generated automatically but here are some very recent changes to the repository format and there might be bugs in the intergration with poudriere.


Ah, it's not really an issue to do it by hand but it would be nice to automate.



> I think the repo.txz file is now deprecated and the repository metadata is now split into multiple files. I don't have my poudriere build system set up atm so I can't check.


Right! I noticed that too, but it seems to be a bit wonky at the moment. Sometimes it creates the 'old' repo.txz and at other times I get the new files. Maybe it's PKGNG that's not working correctly. I'm sure it'll be fixed soon, it's updated quite frequently.


----------



## kpa (Jul 6, 2013)

If you have a ports-mgmt/pkg client that is version 1.1 or greater the repo.txz file is not used at all, it's only for version 1.0.x clients.


----------



## spork (Jul 21, 2013)

Is anyone using the "sets" feature with poudriere, and if so, how are you dealing with maintaining repos across boxes that have different requirements?  Let's take the example of three poudriere buildlists:


packages common to ALL hosts (common)
rsync
sshguard
vim
bash
apache22
php5

host group A (group-a)
perl 5.16
mysql51
postfix

host group B (group-b)
mysql50
perl 5.14
php53


Using the "sets" feature (the -z option to the bulk and options subcommands) would allow for each group specified above to have a custom set of options.  The sets however also leave you with a package repo for each set, and a duplication of the common ports amongst all sets.  

For example, building the three sets above:


```
poudriere -j 84amd64 -z common -f /usr/local/etc/poudriere.d/buildlists/common
poudriere -j 84amd64 -z common -f /usr/local/etc/poudriere.d/buildlists/common -f /usr/local/etc/poudriere.d/buildlists/group-a
poudriere -j 84amd64 -z common -f /usr/local/etc/poudriere.d/buildlists/common -f /usr/local/etc/poudriere.d/buildlists/group-b
```

And then you would have three repos:

/poudriere_data/packages/common
/poudriere_data/packages/group-a
/poudriere_data/packages/group-b

And all packages in the "common" buildlist would be built three times and the resulting packages would also exist in all three repos.  You would also need to define a different repo in each group of hosts in pkg.conf.  I'm fine with that, but I wonder if there's a more clever way to do this without using sets to avoid the duplicate builds (both saving the time it takes to build a package multiple times and saving disk space).

Perhaps sets aren't needed and this can all be done with only a supplemental buildlist for each group of hosts?  I'm not sure what poudriere does if you try to build packages that conflict with each other in the same `poudriere bulk` run (in the example above, MySQL, PHP, and Perl have conflicting versions).

Anyone doing something similar?

TL;DR:  What's the intended use-case for poudriere sets?


----------



## kpa (Jul 24, 2013)

I haven't had time to look into sets yet and I don't really need them because at the moment I'm building the packages for just one system. Feel free to post your findings on this thread.


----------



## SirDice (Jul 24, 2013)

I'd simplify things and just build two different repositories. Ports that are common are in both trees. On the one group you set PACKAGESITE to http://myserver/packages-set1 and on the other you set it to http://myserver/packages-set2.

That's the easiest way to deal with it I think.


----------



## hedgehog (Aug 8, 2013)

@kpa, thank you for the great post!

I'm currently switching to ports-mgmt/poudriere for building packages for desktop, but I have a question. Can I use it for incremental updates of my packages without rebuilding them all every time? If yes, what should I do in case if there is a port that have libraries version bumped, so everything dependent on it should be rebuilt?


----------



## SirDice (Aug 8, 2013)

hedgehog said:
			
		

> Can I use it for incremental updates of my packages without rebuilding them all? If yes, what should I do in case if there is a port that have libraries version bumped, so everything dependent on it should be rebuilt?


Yes, that's possible. And you don't really need to do anything, poudriere already does it by default. If you want to make 100% sure your dependencies are correctly built you can always add the -c switch to the bulk command, that removes all previously built packages and starts fresh. If you don't use -c it will do an 'incremental', updating only the packages that are newer. It will also take care of any dependent packages.


----------



## hedgehog (Aug 8, 2013)

SirDice said:
			
		

> Yes, that's possible. And you don't really need to do anything, poudriere already does it by default. If you want to make 100% sure your dependencies are correctly built you can always add the -c switch to the bulk command, that removes all previously built packages and starts fresh. If you don't use -c it will do an 'incremental', updating only the packages that are newer. It will also take care of any dependent packages.



Does that mean that, for instance, if the devel/gettext has it's version bumped, all the ports depending on devel/gettext will get rebuilt automatically, even if their version remained the same? Do you happen to know how is that achieved?


----------



## kpa (Aug 8, 2013)

The dependency solver takes care of that and my experience with poudriere is that it works correctly in almost all cases. I yet have to see a case where a version bump failed to rebuild a dependency correctly. A package is also considered to be changed if it's options have changed and you have enabled the options checking in poudriere.conf.

The PKGNG pkg(8) client also tracks the shared library versions and considers a package changed if there are changes in the shared library dependencies. This should catch ports that didn't get their port versions bumped but have a dependency to a shared library that got a version bump.


----------



## hedgehog (Aug 8, 2013)

Thank you for answers.

Sorry for bombarding you with questions. Do I still need to manually request of reinstalling all packages that depends on devel/gettext in that case on the target system? Like that:
`# pkg install -R gettext`

Will the ports-mgmt/poudriere take care of dependencies that have they origins changed as well? For instance:

```
20121002:
  AFFECTS: users of devel/clanlib
  AUTHOR: jhale@FreeBSD.org

  devel/clanlib has been updated to 2.3.6 and the legacy version has been
  moved to devel/clanlib1 and updated to 1.0.0.  Although both versions
  may be installed at the same time, you should update your port origins
  first since many ports that depend on clanlib will not work with the
  new version.

  Update your port origins as follows:

  # portmaster -o devel/clanlib1 devel/clanlib
  or
  # portupgrade -fo devel/clanlib1 devel/clanlib
  or
  # pkg set -o devel/clanlib:devel/clanlib1
```

I use Kubuntu at work and the only repository advantage I see there is that you don't have to care about dependencies at all. You just issue `apt-get upgrade` command and it takes care of everything. I think it would be great if you could add some metadata for packages in repository that will, for example, ask a user if he wants to automatically reinstall all the packages that depends on the specific package while updating it to current version. Same with changing ports origins.


----------



## kpa (Aug 8, 2013)

Changed port origins are not a problem as long as the ports that you want get actually built by a build run. If the changed port is a dependency of a port in your list of ports that pass to poudriere then there's no problem, the whole chain of dependencies gets rebuilt if needed. If the port is a "leaf port" (that no other port depends on) that is directly listed in your list then you have to rename the port in the list. (I have to check if MOVED is consulted in the latter case, it would be really great if it was).


----------



## SirDice (Aug 8, 2013)

hedgehog said:
			
		

> Do I still need to manually request of reinstalling all packages that depends on devel/gettext in that case on the target system? Like that:
> `# pkg install -R gettext`


Just run `# pkg upgrade`. It will correctly figure out which packages to update and/or replace. I haven't had any major issues doing this.


----------



## hedgehog (Aug 15, 2013)

Great, thank you guys. Looks like it always rebuilds all dependencies automatically and updates the repository, so `# pkg upgrade` will know that the packages were rebuilt.

Quite a smart solution to avoid conflicts and make it even more automated. The only drawback is the longer time required for incremental builds, but that's the cost of stability.


----------



## kpa (Oct 6, 2013)

The non-development version of ports-mgmt/poudriere seems have support for the NO_ZFS option starting from version 3.0.1 or 3.0.2. I have tested building a few packages with ports-mgmt/poudriere-devel replaced with ports-mgmt/poudriere without changing anything in my configuration files and everything seems to work. I'll add a notice to his HOWTO as soon as I'm satisfied that the non-development version can be used without issues.

Edit: Changes committed. I have built enough packages now to be confident that the non-development version works as well as the development version.


----------



## hedgehog (Oct 10, 2013)

rusty said:
			
		

> I set things up with the
> 
> 
> 
> ...



It gives _insane_ speed boost because ports-mgmt/poudriere uses disk a lot when installing dependencies. Before having everything in tmpfs, I saw build-depends status all the time and it required more time than actually building ports.

Now I build everything in tmpfs and it's really fast. Also, I noticed that I usually have 7-10GB RAM free, out of 16GB, when using two builders, so probably it would be a good idea to increase builders number to 3.


----------



## kpa (Oct 10, 2013)

A new configuration option was added a while ago to automatically detect dependency changes.

This is the full description from poudriere.conf.sample:


```
# Automatic Dependency change detection
# When bulk building packages, compare the dependencies from kept packages to
# the current dependencies for every port. If they differ, the existing package
# will be deleted and the port will be rebuilt. This helps catch changes such
# as DEFAULT_RUBY_VERSION, PERL_VERSION, WITHOUT_X11 that change dependencies
# for many ports.
# Valid options: yes, no
#CHECK_CHANGED_DEPS=yes
```


----------



## kpa (Oct 14, 2013)

A note on the -m url=file:///some/path method for creating a jail. This method is only supported by ports-mgmt/poudriere-devel at the moment (non-development versions is at version 3.0.10 at the time of writing this). I'd add this information to the HOWTO itself but the message is on the character limit and there's no more room.


----------



## kpa (Oct 15, 2013)

I switched to stable/10 aka 10-STABLE on my system yesterday and I'm now rebuilding everything using a stable/10 jail, very slowly though on a VIA C7 based system :OOO Everything has worked so far. I think we can call 10-STABLE a supported version of FreeBSD now.


----------



## hedgehog (Oct 15, 2013)

Is it possible to configure automounting when a jail starts using /etc/jail.conf and /etc/fstab.jailname? I am mounting system's ports options as nullfs while using ZFS, so they don't mount automatically using rc.conf.

By the way, can I just symlink /var/db/ports to poudriere data directory instead of nullfs mounting it?

(You can use *late* option to achieve this)


----------



## kpa (Oct 30, 2013)

I'd also add this to the HOWTO itself but there's no room. 

It's also possible to use SSH for fetching repository catalogues and packages. This is done by using an ssh:// URL as PACKAGESITE. For example:

```
PACKAGESITE: ssh://user@packagehost.mydomain.tld/usr/local/poudriere/data/packages/stable10i386-default
```

This saves you from setting up a web server on the package host if there's no other reason to have one.

This is more convenient when you have public key authentication with agent forwarding set up for user@packagehost so you don't have to type passwords on every `pkg update` operation.

Do not use the root user unnecessarily for logging into the package host, unprivileged user is fine as long as the user has the necessary access to the package repository directories.


----------



## kpa (Nov 16, 2013)

A little issue caught me off guard. It seems that the default options directory was changed at some point. It is now /usr/local/etc/poudriere.d/${jailname}-options. I still had a /usr/local/etc/poudriere.d/options directory and I was quite confused why my changed options were being ignored by the build runs. I deleted the /usr/local/etc/poudriere.d/options directory and re-did all the options. Everything is fine now.


----------



## wszczep (Nov 18, 2013)

What version of ports-mgmt/poudriere do you have installes at the moment? I didn't notice such behaviour with version 3.0.11 - I am building packages for four different jails with one package list file and one options set. 

Before starting each build batch I am checking options with `poudriere options -f /home/admin/pkg_list.txt` without specifying the jail name. Might be that you did run it for a named jail?

BTW. I have noticed that though the 'daemon' option is no longer specified in the man page, it still works.


----------



## kpa (Nov 18, 2013)

Yes you're right, I am indeed running the options with the -j jail option. I started doing that for some reason that I can no longer remember. I'll have to review my HOWTO if it's incorrect somehow in this regard.


----------



## kpa (Nov 18, 2013)

I shortened the poudriere.conf file part of the HOWTO. The DISTFILES_CACHE setting defaults to /usr/ports/distfiles and the default setting is completely fine now. Few of the settings I showed are the defaults anyway and don't have to be changed. I also added note about using the -j jail option in `poudriere options`.


----------



## kpa (Nov 30, 2013)

*Re: PKGNG package repository using ports-mgmt/poudriere. No *

Updated to be up to date with changes introduced in ports-mgmt/pkg version 1.2.1.


----------



## nanotek (Feb 9, 2014)

*Re: PKGNG package repository using ports-mgmt/poudriere. No *



			
				kpa said:
			
		

> The packages can be installed from the local packages directory with `pkg add`.
> 
> `pkg add /usr/local/poudriere/data/packages/release91amd64-default/Latest/apache22.txz`



Thanks for the comprehensive guide, @kpa. It was a great help.

My $JAILNAME-default/Latest directory doesn't contain any packages:

`ls /zbank/poudriere/data/packages/10x64-default/Latest/`

```
pkg.txz
```

However, my $JAILNAME-default/All directory does:

`ls /zbank/poudriere/data/packages/10x64-default/All/`

```
GeoIP-1.4.8_3.txz               php5-hash-5.4.24.txz
apache24-2.4.6_1.txz            php5-iconv-5.4.24_1.txz
apr-1.4.8.1.5.3.txz             php5-json-5.4.24.txz
aspell-0.60.6.1_4.txz           php5-ldap-5.4.24.txz
autoconf-2.69.txz               php5-mbstring-5.4.24.txz
autoconf-wrapper-20131203.txz   php5-mcrypt-5.4.24.txz
bigreqsproto-1.1.2.txz          php5-mysql-5.4.24.txz
bison-2.7.1,1.txz               php5-mysqli-5.4.24.txz
ca_root_nss-3.15.4.txz          php5-openssl-5.4.24.txz
cclient-2007f,1.txz             php5-pdo-5.4.24.txz
cmake-2.8.12.1.txz              php5-pdo_mysql-5.4.24.txz
cmake-modules-2.8.12.1_1.txz    php5-pdo_pgsql-5.4.24.txz
curl-7.34.0.txz                 php5-pdo_sqlite-5.4.24.txz
cyrus-sasl-2.1.26_4.txz         php5-pgsql-5.4.24.txz
damageproto-1.2.1.txz           php5-pspell-5.4.24.txz
db5-5.3.28.txz                  php5-session-5.4.24.txz
dovecot2-2.2.10.txz             php5-simplexml-5.4.24.txz
dri2proto-2.8.txz               php5-sqlite3-5.4.24.txz
expat-2.1.0.txz                 php5-tokenizer-5.4.24.txz
fixesproto-5.0.txz              php5-wddx-5.4.24.txz
fontconfig-2.11.0_1,1.txz       php5-xml-5.4.24.txz
freeglut-2.8.1.txz              php5-xmlreader-5.4.24.txz
freetype2-2.5.2.txz             php5-xsl-5.4.24.txz
gdbm-1.11.txz                   php5-zip-5.4.24.txz
gettext-0.18.3.1.txz            php5-zlib-5.4.24.txz
glproto-1.4.17.txz              php55-5.5.8.txz
gmake-3.82_1.txz                php55-bz2-5.5.8.txz
google-perftools-2.1_1.txz      php55-ctype-5.5.8.txz
help2man-1.43.3_1.txz           php55-curl-5.5.8.txz
icu-52.1.txz                    php55-dom-5.5.8.txz
inputproto-2.3.txz              php55-exif-5.5.8.txz
jbigkit-1.6.txz                 php55-extensions-1.8.txz
jpeg-8_4.txz                    php55-fileinfo-5.5.8.txz
kbproto-1.0.6.txz               php55-filter-5.5.8.txz
libGL-7.6.1_4.txz               php55-ftp-5.5.8.txz
libGLU-9.0.0.txz                php55-gd-5.5.8.txz
libICE-1.0.8,1.txz              php55-hash-5.5.8.txz
libSM-1.2.2,1.txz               php55-iconv-5.5.8_1.txz
libX11-1.6.2,1.txz              php55-imap-5.5.8.txz
libXau-1.0.8.txz                php55-json-5.5.8.txz
libXaw-1.0.12,2.txz             php55-ldap-5.5.8.txz
libXdamage-1.1.4.txz            php55-mbstring-5.5.8.txz
libXdmcp-1.1.1.txz              php55-mcrypt-5.5.8.txz
libXext-1.3.2,1.txz             php55-mysql-5.5.8.txz
libXfixes-5.0.1.txz             php55-mysqli-5.5.8.txz
libXi-1.7.2,1.txz               php55-opcache-5.5.8.txz
libXmu-1.1.2,1.txz              php55-openssl-5.5.8.txz
libXp-1.0.2,1.txz               php55-pdo-5.5.8.txz
libXpm-3.5.11.txz               php55-pdo_mysql-5.5.8.txz
libXrandr-1.4.2.txz             php55-pdo_pgsql-5.5.8.txz
libXrender-0.9.8.txz            php55-pdo_sqlite-5.5.8.txz
libXt-1.1.4,1.txz               php55-pgsql-5.5.8.txz
libXxf86vm-1.1.3.txz            php55-phar-5.5.8.txz
libcheck-0.9.12.txz             php55-posix-5.5.8.txz
libdrm-2.4.17_1.txz             php55-pspell-5.5.8.txz
libevent2-2.0.21.txz            php55-session-5.5.8.txz
libexecinfo-1.1_3.txz           php55-simplexml-5.5.8.txz
libgcrypt-1.5.3.txz             php55-sockets-5.5.8.txz
libgd-2.1.0_1,1.txz             php55-sqlite3-5.5.8.txz
libgpg-error-1.12.txz           php55-tokenizer-5.5.8.txz
libiconv-1.14_1.txz             php55-xml-5.5.8.txz
libltdl-2.4.2_2.txz             php55-xmlreader-5.5.8.txz
libmcrypt-2.5.8.txz             php55-xmlwriter-5.5.8.txz
libpciaccess-0.13.2.txz         php55-zip-5.5.8.txz
libpthread-stubs-0.3_4.txz      php55-zlib-5.5.8.txz
libspf2-1.2.10_1.txz            phpMyAdmin-4.1.6.txz
libtool-2.4.2_2.txz             pkg-1.2.6.txz
libxcb-1.9.3.txz                pkgconf-0.9.4.txz
libxml2-2.8.0_3.txz             png-1.5.17.txz
libxslt-1.1.28_1.txz            postfix-2.11.0,1.txz
m4-1.4.17,1.txz                 postgresql93-client-9.3.2.txz
makedepend-1.0.5,1.txz          postgresql93-server-9.3.2.txz
mp3info-0.8.5.txz               printproto-1.0.5.txz
mysql56-client-5.6.16.txz       py27-fail2ban-0.8.11_1.txz
mysql56-server-5.6.16.txz       py27-libxml2-2.8.0.txz
nginx-1.4.4_3,1.txz             py27-setuptools-2.0.1.txz
oniguruma4-4.7.1.txz            python2-2_2.txz
openldap-client-2.4.38.txz      python27-2.7.6_2.txz
owncloud-6.0.1.txz              randrproto-1.4.0.txz
p5-Locale-gettext-1.05_3.txz    renderproto-0.11.1.txz
pciids-20131225.txz             roundcube-0.9.5,1.txz
pcre-8.34.txz                   sqlite3-3.8.2_1.txz
pdflib-7.0.5_2.txz              t1lib-5.1.2_2,1.txz
pecl-intl-3.0.0_1.txz           tiff-4.0.3.txz
pecl-pdflib-3.0.4.txz           tor-devel-0.2.4.20.txz
perl5-5.16.3_6.txz              wordpress-3.8.1,1.txz
php5-5.4.24.txz                 xcb-proto-1.9.txz
php5-bz2-5.4.24.txz             xcmiscproto-1.2.2.txz
php5-ctype-5.4.24.txz           xextproto-7.2.1.txz
php5-curl-5.4.24_1.txz          xf86bigfontproto-1.2.0.txz
php5-dom-5.4.24.txz             xf86vidmodeproto-2.3.1.txz
php5-exif-5.4.24.txz            xorg-macros-1.18.0.txz
php5-fileinfo-5.4.24.txz        xproto-7.0.25.txz
php5-filter-5.4.24.txz          xtrans-1.3.2.txz
php5-gd-5.4.24.txz
```

Did Poudriere change where packages are stored or has something gone awry in my configuration?


----------



## kpa (Feb 9, 2014)

*Re: PKGNG package repository using ports-mgmt/poudriere. No *

I believe I should re-write that part. The structure of the repository has changed a bit from the time I wrote the HOW-TO. What you're seeing is normal assuming you can install packages when the repository URL ends with packages/10x64-default.


----------



## nanotek (Feb 9, 2014)

*Re: PKGNG package repository using ports-mgmt/poudriere. No *

Thanks. That's good to know. My repo.conf:

`cat /usr/local/etc/pkg/repos/repo.conf`

```
repo: {
    url             : "file:///zbank/poudriere/data/packages/10x64-default",
    enabled         : yes,
    signature_type  : "PUBKEY",
    mirror_type     : NONE
    pubkey          : "/usr/local/etc/ssl/certs/repo.crt"
}
```

Works as expected:

`pkg install tor-devel`

```
Updating repository catalogue
The following 2 packages will be installed:

        Installing libevent2: 2.0.21 [repo]
        Installing tor-devel: 0.2.4.20 [repo]

The installation will require 6 MB more space

1 MB to be downloaded

Proceed with installing packages [y/N]: y
libevent2-2.0.21.txz                100%  269KB 268.8KB/s 268.8KB/s   00:00
tor-devel-0.2.4.20.txz              100% 1082KB   1.1MB/s   1.1MB/s   00:00
Checking integrity... done
[1/2] Installing libevent2-2.0.21... done
[2/2] Installing tor-devel-0.2.4.20...===> Creating users and/or groups.
Creating group '_tor' with gid '256'.
Creating user '_tor' with uid '256'.
 done
================================================================================
To enable the tor server, set tor_enable="YES" in your /etc/rc.conf
and edit /usr/local/etc/tor/torrc.  Note that the rc.subr script overrides
many torrc options and is tunable: see /usr/local/etc/rc.d/tor for details.
To use the torify script, install the net/torsocks port. After removing the
package, or updating to a new version, users may need to delete all files
in /var/db/tor.
================================================================================
```


----------



## kpa (Feb 13, 2014)

*Re: PKGNG package repository using ports-mgmt/poudriere. No *

Small updates here and there. I changed all references to PACKAGESITE to refer to the repository configuration files under /usr/local/etc/pkg/repos. The optional RSA signing key part moved to the end since it's optional.


----------



## mecano (Apr 1, 2014)

*Re: PKGNG package repository using ports-mgmt/poudriere. No *

Thanks for this great tutorial @kpa,
at step 3 you enable devel/ccache in poudriere.conf  :

```
# Use devel/ccache to speed up building, optional.
CCACHE_DIR=/var/cache/ccache
```
From /usr/local/share/doc/ccache/ccache-howto-freebsd.txt, I read


> To use ccache for ports, just add WITH_CCACHE_BUILD=yes to
> /etc/make.conf.


In your example make.conf for the ports-mgmt/poudriere jail I see no mention of it? Why? Is it enough to just define WITH_CCACHE_BUILD in /etc/make.conf?


----------



## kpa (Apr 1, 2014)

*Re: PKGNG package repository using ports-mgmt/poudriere. No*

I haven't used devel/ccache in a long time because I had some serious problems with it so I can't answer your question unfortunately. I left the reference to ccache in this HOWTO as an example, nothing more.


----------



## mecano (Apr 1, 2014)

*Re: PKGNG package repository using ports-mgmt/poudriere. No*

thanks will try with the WITH_CCACHE_BUILD in the jail make.conf to see.
[edit] I just realized that putting this option in the jail make.conf doesn't make sense because the devel/ccache cache folder will be out of the scope of the jail.
[edit] Also ports-mgmt/poudriere will take care of it: from poudriere.conf for CCACHE_DIR


> It will be mounted into the jail and be shared among all jails.


----------



## kpa (Jun 8, 2014)

*Re: PKGNG package repository using ports-mgmt/poudriere. No *

Something to take note of. If you upgrade your jail, all packages built with that jail will be consider out of date and rebuilt. This is because the jails now have a timestamp associated with them and the packages built using that jail are actually located in a directory that uses the timestamp of the last built package in the name of the directory.

For example:


```
$ cat /usr/local/etc/poudriere.d/jails/release_10_0_amd64/timestamp 
1402191481

$ ls -l /usr/local/poudriere/data/packages/release_10_0_amd64-default/.latest    
lrwxr-xr-x  1 root  wheel  16 Jun  8 18:16 /usr/local/poudriere/data/packages/release_10_0_amd64-default/.latest -> .real_1402240565
```

In this case the timestamp of the jail is 1402191481 (seconds after epoch) and the last package building run was done at timestamp 1402240565. All good. Now if I upgraded the build jail so that the timestamp of the jail was greater that this 1402240565 all packages would have to be rebuilt.


----------



## kpa (Jun 21, 2014)

*Re: PKGNG package repository using ports-mgmt/poudriere. No*

There's a very neat new feature in the latest (3.0.99.20140619_1) version of ports-mgmt/poudriere-devel. It now allows the use of an existing jail as the build jail. This simplifies the updating of a jail greatly if the jail is of a stable branch and you're compiling everything from source yourself. This is how the jails are now created (this assumes you have finished `make buildworld buildkernel` on the host and /usr/obj contains the built files):


```
cd /usr/src
make installworld DESTDIR=/data/jails/build-stable-10-amd64 DB_FROM_SRC=1
make distrib-dirs DESTDIR=/data/jails/build-stable-10-amd64 DB_FROM_SRC=1
make distribution DESTDIR=/data/jails/build-stable-10-amd64 DB_FROM_SRC=1

poudriere jail -c -m null -M /data/jails/build-stable-10-amd64 -v 267841 -j build-stable-10-amd64
```

I used /data/jails/* here as the paths because that's what I have on my own system. The argument for the -v option needs to be an SVN revision number for some reason.

That's it. The same works for i386 jails. The neat thing is that the jails can be now shared between poudriere and other purposes like (for example) cross-compiling world and kernel for i386 only hosts. To update the jail you would first make sure the jail is not running and then just do the normal build, install, mergemaster(8), `make delete-old delete-old-libs` procedure on the jail. The other way to update the jail would be to run `make installworld`, `mergemaster` and `make delete-old delete-old-libs` in the jail itself in case you're also using the jail for building the world and kernel.


----------



## kpa (Mar 23, 2015)

With the help from moderators (thanks wblock@!) the HOWTO has been split into two separate posts to overcome the character limit for a single post that is lower (now 12500 characters) on this new forum software than it used to be and I can edit the HOWTO again.


----------



## kpa (Apr 13, 2015)

I added some motivation at the beginning about why use ports-mgmt/poudriere explaining some of the problems in the ports(7) system and why it solves those problems in a satisfactory way.


----------



## kpa (Apr 13, 2015)

getopt said:


> Maybe you want to edit PKGNG to pkg, as pkg is now standard and PKGNG became obsolete
> ?



Yes, I probably should.


----------



## kpa (Aug 3, 2015)

Removed some outdated information. For example WITH_PKGNG has been the default for a while for all supported versions of FreeBSD.


----------



## kpa (Aug 18, 2015)

Edited to reflect the removal of $UNIQUENAME.


----------



## SirDice (Oct 28, 2016)

Apparently people are having a hard time finding this excellent howto. As it's an older thread it got snowed under with newer posts. This is just a bump to get it to float back to the top.


----------



## chrbr (Oct 28, 2016)

This forum is full of jewels. And this howto helped me a lot, too.


----------



## BSDAppentic3 (Feb 27, 2018)

Once I tried to create a jail. I don't know why, but that last time that I do that I needed to reinstall the system.
I think that was because I disconnected the device that I was using where I had the jail.
Since that, I have panic of even try to create one.


----------



## SirDice (Feb 27, 2018)

The nice thing about poudriere is that you don't need to worry about all that. It's all automated with just a few commands. I would recommend using it _with_ ZFS though. Without ZFS it is incredibly I/O intense when it spawns new build jails.


----------



## kpa (Aug 13, 2018)

I'm back after some time and I intend to update this HOWTO with the latest changes to the ports system and other stuff. It's likely that I'll change this to default to a ZFS setup but offer alternative settings if you don't want to use ZFS. Quite a lot of time has passed and ZFS is now mature enough to be the de facto filesystem in your FreeBSD system.


----------



## kpa (Aug 14, 2018)

Updated now to recommend using ZFS as the default configuration and some minor rewrites here and there.


----------



## kpa (Aug 16, 2018)

I did some heavy handed editing in certain parts. It turned out that you can no longer register an existing ports tree into poudriere and have poudriere adopt that port tree completely. You have to create a new ports tree if you want poudriere to be able to update it as well.


----------



## spork (Sep 8, 2018)

kpa said:


> Quite a lot of time has passed and ZFS is now mature enough to be the de facto filesystem in your FreeBSD system.



I have quite a little collection of FreeBSD instances in the cloud, most of them small, with 1GB of RAM or less. I also have some here and there in VMware where I'm only able to allocate from 4GB to 8GB of RAM. I only use UFS on these as my understanding is that ZFS is not happy/performant when you restrict the ARC down to some tiny amount of RAM.
Has anything changed in that area?


----------



## kpa (Sep 8, 2018)

spork said:


> I have quite a little collection of FreeBSD instances in the cloud, most of them small, with 1GB of RAM or less. I also have some here and there in VMware where I'm only able to allocate from 4GB to 8GB of RAM. I only use UFS on these as my understanding is that ZFS is not happy/performant when you restrict the ARC down to some tiny amount of RAM.
> Has anything changed in that area?



On those very small systems with 1GBs of RAM it gets a bit tricky to get ZFS working but it can be done with careful tuning and the performance won't be that bad. I would still use UFS as you have done. With 4GBs of RAM however ZFS will work absolutely fine without any tuning and the performance is as good as it gets, you might want to manually limit ARC to about 1GBs out of the 4GBs to guarantee that applications and other parts of the kernel have enough memory available.


----------



## Sevendogsbsd (Nov 10, 2018)

Only a couple of months old so hope this isn't a necro-bump... I have set up a ports-mgmt/poudriere server on a beefy build machine as kpa instructed (thanks!) and it is building a couple of test packages as we speak. In poking around in the directory tree of ports-mgmt/poudriere, I noticed what looks like a copy of the ports tree. This brings me to my first question:

Since ports-mgmt/poudriere is building ports, how to I keep the ports tree updated and which tree do I update? 
Also, since I am going to set up this server as the pkg source for my main FreeBSD workstation, what is the easiest way of building /usr/local/etc/ports.txt or must that be done manually? 

My assumption is that I will regularly update the ports tree on the build machine and it will be the source of all packages for my normal workstation. Given if that is true, when I decide I'd like to install a package that I have not built before, I need to add it to /usr/local/etc/ports.txt and build again, correct? This can a very lengthy process, even given my build hardware's capability. 

What is the recommendation for the contents of the build server then, all packages (seems excessive), or how is this situation dealt with normally? I assume the official FreeBSD package repositories have multiple build machines and this is a continual process based on code updates to ports.

Thanks for the HOWTO - was very helpful!


----------



## zirias@ (Nov 10, 2018)

Sevendogsbsd said:


> Given if that is true, when I decide I'd like to install a package that I have not built before, I need to add it to /usr/local/etc/ports.txt and build again, correct? This can a very lengthy process, even given my build hardware's capability.



`poudriere bulk` (without the `-c` option) doesn't rebuild packages if it isn't needed because they're new, updated, have updated dependencies or changed build options. So, just add what you need, and run `poudriere bulk` on it again.


----------



## Sevendogsbsd (Nov 10, 2018)

Oh that's awesome - even with 12 Xeon cores and 96gb of ram, this devel/llvm60 build is on 2 and a half hours so this is going to get old really fast...


----------



## zirias@ (Nov 11, 2018)

Sevendogsbsd what do you expect? my latest poudriere bulk run is now working on www/chromium for 9 hours and counting.

If you have some heavy-weight ports in your package list, there's (at least) two things you can do to speed up the bulk run:

list those fat ports in `PRIORITY_BOOST`, so they are started as early as possible
if you are sure parallel builds are fine, list fat ports in `ALLOW_MAKE_JOBS_PACKAGES`, so their build uses more than one core. I'm personally leaving this empty because poudriere already builds several ports in parallel, so building a single port in parallel would just increase load -- but if you're in a situation where one single fat port often ends up being the only active job, it might be worthwile to allow parallel building for that one.


----------



## Sevendogsbsd (Nov 11, 2018)

Just building chromium by configuring and doing a `make` in the port's directory takes me about an hour and 20 minutes. As an experiment, I tried ports-mgmt/synth and it built devel/llvm60 in about an hour and a half.

I know it takes time, especially with big builds, I just expected it to be faster. No worries, it all completed. Thanks for the pointers!


----------



## kpa (Nov 13, 2018)

Yes, ports are rebuilt only when there are updates. However, FreeBSD ports doesn't yet have proper tracking if the dependents of a port really require rebuilding. This is results in rather large churn when an important port is updated and possibly over half of your packages are rebuilt even if there was no need for to rebuild anything but the one changed port. For example if there's a small bugfix to security/openssl you can expect a large amount of built packages even if nothing really changed in them.

You have to update and maintain the ports.txt yourself, you're the one who knows which ports you need in your system.


----------



## Sevendogsbsd (Nov 13, 2018)

Right - got it. I figured out that documenting the ports I needed first was the key. I have a system/process down pat now so it works well. There is definitely a "cascading" of builds after a ports tree update but it's fine: I have a build server so I just fire it up every week or so and refresh the ports tree, then see what needs to be built. It doesn't affect my regular workstation so I can let it run overnight if need be.


----------



## Nyakov (Mar 18, 2019)

Do I need to do additional setup before I can use  poudriere, setting jail_enable="YES" in /etc/rc.conf for example?

Do jail specific jail_name-make.conf file in "/usr/local/etc/poudriere.d" merge with make.conf from the same dir when ports a builded for the specific jail?


----------



## rigoletto@ (Mar 18, 2019)

Nyakov said:


> Do I need to do additional setup before I can use  poudriere, setting jail_enable="YES" in /etc/rc.conf for example?
> 
> Do jail specific jail_name-make.conf file in "/usr/local/etc/poudriere.d" merge with make.conf from the same dir when ports a builded for the specific jail?



No. Poudriere take care of everything it needs about jail mounting etc.

No. Totally separated configurations.


----------



## hukadan (Mar 18, 2019)

rigoletto@ said:


> No. Totally separated configurations.


Are you sure ? From poudriere(8) :


> You can also specify a global make.conf which will be used for all the jails.  Any of the following are allowed and _will *all* be used in the order shown_:
> /usr/local/etc/poudriere.d/make.conf
> /usr/local/etc/poudriere.d/<setname>-make.conf
> /usr/local/etc/poudriere.d/<tree>-make.conf
> ...


My understanding is that poudriere(8) will read all of them and overwrite variables if/when they are redefined.


----------



## Nyakov (Mar 18, 2019)

rigoletto@ said:


> No. Totally separated configurations.


So,  if jail_name-make.conf present then make.conf ignored?
Why I am asking.
Handbook states: _The system make.conf and this new file are combined at build time to create the make.conf used by the build jail_

And this howto states that:


> Note that the build host's own /etc/make.conf is not used at all because it's assumed to be "tainted"



So I confused.  What _"system make.conf" _in handbook means?

*Question is answered:*


hukadan said:


> My understanding is that poudriere(8) will read all of them and overwrite variables if/when they are redefined.


----------



## rigoletto@ (Mar 18, 2019)

hukadan said:


> Are you sure ? From poudriere(8) :
> 
> My understanding is that poudriere(8) will read all of them and overwrite variables if/when they are redefined.



Good point. What I got from the message was something related with /etc/make.conf.


----------



## SirDice (Mar 18, 2019)

Nyakov said:


> So, if jail_name-make.conf present then make.conf ignored?


No, they are stacked. Poudriere will always ignore /etc/make.conf. 


```
Create optional make.conf
     You can also specify a global make.conf which will be used for all the
     jails.  Any of the following are allowed and will all be used in the
     order shown:

           /usr/local/etc/poudriere.d/make.conf
           /usr/local/etc/poudriere.d/<setname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/hooks/plugins/<plugin>/make.conf
```
See poudriere(8).


----------



## rigoletto@ (Mar 18, 2019)

Nyakov

If you are asking about /etc/make.conf, yes it is completely ignored. Think the ports-mgmt/poudriere jailname-make.conf is equivalent to the /etc/make.conf inside a jail.


----------



## Mjölnir (Jun 26, 2020)

Since your howto is so long, IMHO an index at the top would be appropiate.  It helps the reader to build up some mental structure in advance of what is going to come, and with an index you give a reasonable preset.
Missing: 1. Integration of ccache.  This is considered common practice. 2. Using svn instead of portsnap to update the portstree, for those who want to stick to the quarterly port branches.
Besides that, the text is structured well and easy to read and understand, which is hard to get right.  Thanks.


----------



## Snurg (Jan 27, 2021)

Thank you kpa for your great writeup!

Question:
There are only a number of ports that I need other configurations than the default ones.

I, the user, just want to say a command "update my system!" and poudriere builds the packages with modified configs that I have listed in the poudriere configuration file, and then pkg update/upgrade will somehow take packages from poudriere, if there are (modified) ones, and all the other packages (with the default build options) will be taken for the FreeBSD pkg server?

So, if I read correctly, it is possible to configure/use poudriere in this way?
So that it only manages a few selected applications, and I don't need a server that constantly fetches and builds thousands of packages I never use?

Ideally I would like only need to press a button "Update my system!" on my personal web admin tool and let the scripts do the work.

Another issue is, I read portsnap is deprecated and will be removed soon, so mjollnir 's point is very important, as the howto needs to be updated, too...


----------



## SirDice (Jan 27, 2021)

Snurg said:


> I, the user, just want to say a command "update my system!" and poudriere builds the packages with modified configs that I have listed in the poudriere configuration file, and then pkg update/upgrade will somehow take packages from poudriere, if there are (modified) ones, and all the other packages (with the default build options) will be taken for the FreeBSD pkg server?


Poudriere just builds everything you tell it to build. This results in a directory full of packages and the required files to make it a repository (pkg-repo(8)). You set up a simple webserver that shares this directory. Then configure your local pkg(8) to fetch its packages from that webserver.

Build everything you need from your own repositories. Mixing your custom settings with the official packages that use default settings is bound to cause dependency issues or other conflicts. Your packages for example might depend on MariaDB 10.4 while the official packages depend on MySQL 5.7. So to prevent this potential nightmare build everything yourself and install only from your own repository.


----------



## Snurg (Jan 27, 2021)

Well this is a good explanation why it's not good to mix "homemade" packages with "official" ones.

The big problem now is, many people don't have the time and resources to set up a dedicated poudriere server with immense disk space and CPU power, which builds tens of thousands of packages that will mostly never be used.

So I think there should then exist a solution about this:

_client pkg sends request ->
web server CGI checks whether the package file is already present ->
if it is not present yet, add it to poudrieres list of packages to maintain (for updates),
let poudriere build package (and its dependencies),
update pkg-repo
-> send repo info to client pkg_

This would have poudriere to _build only what the user actually requests_, given pkg is patient enough to wait for the response.
In turn, this would result in a repo small enough to keep locally on the computer.

Does such a solution already exist?


----------



## SirDice (Jan 27, 2021)

Snurg said:


> set up a dedicated poudriere server with immense disk space and CPU power


You don't need a "massive" server for this. I do all my package building at home on an old Core i5 with 16GB RAM. It just takes a little longer to finish compared to the 24 core, 96GB RAM beast I can use at a client. 



Snurg said:


> which builds tens of thousands of packages that will mostly never be used.


You don't build _everything_ (as in all 40.000+ packages), you feed it a list of things you need.

```
-f file  Build ports listed in the file.

              The path to the file has to be absolute.  Ports must be
              specified in the form of "category/port" and sh(1)-style
              comments are allowed.  Multiple -f file arguments may be
              specified at once.
```
See poudriere-bulk(8).

I have a poudriere set up for a client, where I need to maintain about 30 FreeBSD servers. Client has no need for desktop or Xorg applications, only Apache, MySQL and an assortment of other web based tools (like Ruby-on-Rails, PHP, etc). I've set this up many years ago and we're still using it right now. The list needs some occasional updates (changes in requirements of the client) but the list is pretty much the same now as it was many years ago. I just let it update its ports tree weekly and run the builds at night. So next morning I have a freshly built and updated package repository. The nice thing about this setup is that _I_ get to decide when to change the default Ruby or PHP version for example, so we can migrate at our own pace. I can also easily revert any updates or temporarily fix ports if needed. I'm also sure _all_ servers have the same versions and settings for everything.


----------



## Snurg (Jan 27, 2021)

That's awesome!

Question 1:
If I understand correctly, if the package <mynewprogram> is not already in the custom repo, then the process of adding some new port would be (in case the dependencies in our repo are all up-to-date):

1. build the port and its dependencies using `poudriere-bulk -f <file_containing_mynewprograms_name>`,
2. copy the generated repo files to the repo storage directory of our repo webserver,
3. run pkg-repo to update the package index,

and then the client can just run `pkg install <mynewprogram>`?

Question 2:
And, when I use `pkg install <mynewprogram>`, does pkg send some information that I like <mynewprogram> and want my cgi to add it to poudriere's build list, too?
(I ask, because, if possible, I would like to automate the steps described above using my cgi)


----------



## SirDice (Jan 27, 2021)

2) don't need to copy it. I have a basic nginx running on the build server itself. You can find configuration examples for nginx and Apache in /usr/local/share/examples/poudriere. 
3) No need, poudriere already does this at the end of its run. 



Snurg said:


> nd then the client can just run `pkg install <mynewprogram>`?


Add the custom repo in /usr/local/etc/pkg/repos/myrepo.conf for example:

```
MyRepo: {
  enabled: yes
  url: http://some.internal.host/packages/
}
```
Also add a /usr/local/etc/pkg/repos/FreeBSD.conf to disable the FreeBSD repositories:

```
FreeBSD: {
  enabled: no
}
```



Snurg said:


> And, when I use `pkg install <mynewprogram>`, does pkg send some information that I like <mynewprogram>


pkg(8) just uses fetch(3) to grab the index files (meta.txz and packagesite.txz; these are used to update the locally cached information; /var/db/pkg/repo-*.sqlite) and then fetches the right file directly. There's no CGI or anything like that involved here.


```
root@molly:~ # ll /usr/local/poudriere/data/packages/122-release-server/
total 118
lrwxr-xr-x  1 root  wheel  18 Oct 28 19:22 .buildname@ -> .latest/.buildname
lrwxr-xr-x  1 root  wheel  20 Oct 28 19:22 .jailversion@ -> .latest/.jailversion
lrwxr-xr-x  1 root  wheel  16 Jan 25 13:34 .latest@ -> .real_1611578069
drwxr-xr-x  4 root  wheel   9 Jan 18 10:19 .real_1610961573/
drwxr-xr-x  4 root  wheel   9 Jan 20 04:54 .real_1611114841/
drwxr-xr-x  4 root  wheel   9 Jan 22 11:45 .real_1611312342/
drwxr-xr-x  4 root  wheel   9 Jan 22 22:14 .real_1611350051/
drwxr-xr-x  4 root  wheel   9 Jan 25 13:34 .real_1611578069/
lrwxr-xr-x  1 root  wheel  11 Oct 28 19:22 All@ -> .latest/All
lrwxr-xr-x  1 root  wheel  14 Oct 28 19:22 Latest@ -> .latest/Latest
lrwxr-xr-x  1 root  wheel  17 Oct 28 19:22 meta.conf@ -> .latest/meta.conf
lrwxr-xr-x  1 root  wheel  16 Oct 28 19:22 meta.txz@ -> .latest/meta.txz
lrwxr-xr-x  1 root  wheel  23 Oct 28 19:22 packagesite.txz@ -> .latest/packagesite.txz
```


----------



## Snurg (Jan 27, 2021)

Damn, is this gosu 
I want local repos 
And, I think now I know the most important points I need to start with trying to integrate poudriere into my postinstaller script soon.
Just extract the options information from the package, edit them in a web form, and store in poudriere configuration for later use (install replay, repo cloning,...), let poudriere and after that pkg run 

Thanks kpa and SirDice


----------

