# Building in one jail, installing in another



## Shew (Oct 7, 2010)

Hello.

I am trying to use a chroot jail to compile ports such that I can install them in other jails. I have multiple jails and previously I had compiled programs in each jail, which meant that a huge amount of compile time had to be duplicated for each jail. I am trying to discover a better method, but it seems that everything I have tried has be very unreliable (such that I can only do it for one or two programs before failing).

I am currently trying to do something like this:
`chroot /usr/shew/chroots/compile make -C /usr/ports/"$category"/"$program" package-recursive clean`

Previously I had tried the following:
Compiling without using `make clean` and then mounting the ports folders in other jails (via nullfs) and using `make install`
Both package and install the ports when installing to and individual jail, then using a combination of pkg_add and make to install new ports in other jails. Installing some ports common to all of the jails, and then copying the entire jail directory or using unionfs to use that jail as a template for others.

Nothing seems to work as well as duplicating the work for all the jails. With what I am currently trying, for example, I get an error with Python:


```
tar: lib/python2.6/lib-dynload/_multiprocessing.so: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
pkg_create: make_dist: tar command failed with code 256
```
Now, the packing lists seem to be fairly static, even though one might change the package options (though, I have not in this case), so I'm not sure this is going to work once I start trying to customize my options.

Is there some recommended general method for installing ports across jails? I sure would appreciate a few tips.

Thanks.


----------



## SirDice (Oct 7, 2010)

Use a jail for building. That's what I've been doing for years. Build packages, make use of portmaster. Then mount_nullfs() your ports tree and install the built packages inside each other jail.


----------



## Shew (Oct 10, 2010)

I just tried `make package-recursive` in a jail and I am getting the same errors (different ports reporting that files are missing when tarring a package). I haven't tried portmaster yet, so I guess I will try that next. There seems to be something buggy going on with package-recursive though.


----------



## anomie (Oct 13, 2010)

@Shew: what about building packages in a full-on FreeBSD jail - not a "chroot jail"? (Did I misunderstand you?)


----------



## Shew (Oct 16, 2010)

@anomie: Yes, I am building in a FreeBSD jail now, but before I was trying to do it using a chroot jail.

It seems my problem was that I was making a mistake with the port configuration such that the port was not set to the default configuration. I have that fixed now, but it still leaves my that problem when I start to customize the build options: If the customized builds make an extra file that needs to be packaged, or don't make a file that is packaged in the default build, it will fail. I looked in the Porter's Handbook and it suggests that you can get the port to build a packing list on the fly, but it looks like this would have to be baked into the Makefile. Is there a different way I can package customized ports? Some ports seem to handle it OK and some ports don't (like Python and threading).


----------



## acheron (Oct 16, 2010)

You should have a look at ports-mgmt/tinderbox.


----------



## SirDice (Oct 19, 2010)

Shew said:
			
		

> @anomie: Yes, I am building in a FreeBSD jail now, but before I was trying to do it using a chroot jail.


To stop further confusion, chroot != jail. Do not refer to a chroot'ed environment as a jail. They are completely different.


----------



## mecano (Nov 25, 2011)

SirDice said:
			
		

> Use a jail for building. That's what I've been doing for years. Build packages, make use of portmaster. Then mount_nullfs your ports tree and install the built packages inside each other jail.



Is that to say you are maintaining a "building jail" ports tree separate from main host one?
For now I'm using the main host mount_nullfs'ed ports tree in a "building jail" and some awk commands that check dependencies on packages, generate a list of packages, then in target jail, copy them, add them, replace them (just in case), quiet.. awkward.
Your solution looks far more elegant, any caveats to mention ? I thought it was a very bad idea to maintain more than one ports tree per box, not the case?


----------



## SirDice (Nov 25, 2011)

mecano said:
			
		

> Is that to say you are maintaining a "building jail" ports tree separate from main host one?


No. It's the same tree.
`# mount -t nullfs /usr/ports /jail/build/usr/ports/`

Because I do have seperate filesystems for packages and distfiles I also need to do
`# mount -t nullfs /usr/ports/distfiles /jail/build/usr/ports/distfiles/`
`# mount -t nullfs /usr/ports/packages /jail/build/usr/ports/packages/`

And I'm lazy so I added those to the jail's fstab. In the host's rc.conf:


```
jail_enable="YES"
jail_list="build"
jail_build_rootdir="/jails/build/"
jail_build_hostname="build.dicelan.home"
jail_build_flags="-l -U root -n build"
jail_build_ip="192.168.100.200"
jail_build_interface="re0"
jail_build_mount_enable="YES"
jail_build_devfs_enable="YES"
```
For jail_build_mount_enable to work you need to create a separate /etc/fstab.build for the jail:

```
/usr/src/             /jail/build/usr/src/             nullfs  ro,noatime  0  0
/usr/ports            /jail/build/usr/ports            nullfs  rw,noatime  0  0
/usr/ports/distfiles  /jail/build/usr/ports/distfiles  nullfs  rw,noatime  0  0
/usr/ports/packages   /jail/build/usr/ports/packages   nullfs  rw,noatime  0  0
```
This will make sure the filesystems are mounted/unmounted when the jail is started/stopped.
I've mounted /usr/src/ read-only as some ports need access to the sources.

When the jail is running just:
`# jexec build /usr/bin/su -`
The su(1) will make sure the environment is setup correctly. Once inside the jail just build packages as you would normally do.


----------



## mecano (Dec 6, 2011)

Dear Sir, thanks for such a detailed answer! Are you deploying ports packages by mounting nullfs as well? For example temporary mounting /var/ports/packages (or another location where you built packages are stored) in the target jail as read only and install from there?


----------



## SirDice (Dec 6, 2011)

mecano said:
			
		

> Are you deploying ports packages by mounting nullfs as well? For example temporary mounting /var/ports/packages (or another location where you built packages are stored) in the target jail as read only and install from there?


On the same machine, yes. For other machines I just have my /usr/ports/packages/ NFS exported read-only.


----------



## fbsd1 (Dec 7, 2011)

You are trying to RE-Invent the wheel again.
Install the "qjail" port and use it to build all your jails. Read the documentation for simple way to populate your jails with a pre-installed group of ports.


----------



## mecano (Dec 8, 2011)

So does ezjail with flavours.

When installing/updating ports in target jail, is it better to
*pkg_add -r* then *pkg_replace -Bf* or to *pkg_replace -Bf* then *pkg_add -r*?


----------



## dougb@ (Dec 31, 2011)

Shew said:
			
		

> @anomie: Yes, I am building in a FreeBSD jail now, but before I was trying to do it using a chroot jail.
> 
> It seems my problem was that I was making a mistake with the port configuration such that the port was not set to the default configuration. I have that fixed now, but it still leaves my that problem when I start to customize the build options: If the customized builds make an extra file that needs to be packaged, or don't make a file that is packaged in the default build, it will fail.



Someone else already suggested portmaster as your tool to build packages, I will ever-so-humbly state that I agree.   As long as the infrastructure in the ports tree is correct, using the -g option for portmaster will produce correct packages.

The issue of customization is more complex however. There are 3 ways to accomplish this. If all of the jails can mount /usr/ports and /var/db/ports then you can use the nullfs trick that others have mentioned so that they all see the same options. 

If they are physically separate, then you have 2 choices. The simplest is to copy the package and all of its dependencies to the remote jails and install them by hand. Obviously that doesn't scale well. 

The other option is to create a custom INDEX that reflects your customizations and install on the jails using *portmaster --packages-only --index-only*. This has advantages of not needing to mount anything in the remote jails, but requires that you set up INDEX creation/distribution, and package distribution on your package-building system. It's not particularly difficult, but it is a bit of a project. There is some information in the portmaster man page, and the pkg_add man page should also be helpful. For custom INDEX creation I highly recommend ports-mgmt/p5-FreeBSD-Portindex. It takes some time to set up, but subsequent INDEX creation is then very fast. 

Hope this helps,

Doug


----------



## Anonymous (Jun 5, 2012)

mecano said:
			
		

> So does ezjail with flavours.
> 
> When installing/updating ports in target jail, is it better to
> *pkg_add -r* then *pkg_replace -Bf* or to *pkg_replace -Bf* then *pkg_add -r*?



Been wondering the same thing.
I'm using ezjails with one "build" jail in which I build ports. *portmaster -g <port>* and then install on destination jail with *pkg_add <package>*.

It's the update part I'm worried about, can't get the workflow straight in my head.
Hope this is relevant to the thread.


----------



## SirDice (Jun 5, 2012)

Just wipe the jail and start fresh. Then you'll never have any updating issues. I takes a little more effort but it pays off in the end.


----------



## Anonymous (Jun 5, 2012)

Hm, so you mean *pkg_delete -a*? Or create a whole new jail?


----------



## SirDice (Jun 5, 2012)

Norrland said:
			
		

> Hm, so you mean *pkg_delete -a*? Or create a whole new jail?



Creating a whole new jail is faster  You don't want to keep any settings anyway as nothing is configured inside the build jail. You only use it to build things.


----------



## Anonymous (Jun 5, 2012)

Hehe. Don't think that'll work for some of my jails (IRC shell for some friends ).

When creating a jail, with ezjail and flavours I've noticed that it won*'*t install the packages I put in $FLAVOUR/pkg. Maybe I've missed something there.


----------



## bbzz (Jun 5, 2012)

There's no need to blow up a whole build jail every time unless there are major changes (like recent) and especially if you are building so much different things (such as KDE and GNOME) for different machines.

Simply use *portmaster* in build jail to rebuild what you need, make sure everything is working as far as dependencies go (*pkg_libchk*) and then use *portmaster* *-PP* on install machine to update packets.

One thing you want to make sure is that build jail and install machine point to same ports tree (or at least INDEX file), and options, /var/db/ports. That way there's no issues.


----------



## Anonymous (Jun 6, 2012)

bbzz said:
			
		

> There's no need to blow up a whole build jail every time unless there are major changes (like recent) and especially if you are building so much different things (such as KDE and GNOME) for different machines.
> 
> Simply use *portmaster* in build jail to rebuild what you need, make sure everything is working as far as dependencies go (*pkg_libchk*) and then use *portmaster* *-PP* on install machine to update packets.
> 
> One thing you want to make sure is that build jail and install machine point to same ports tree (or at least INDEX file), and options, /var/db/ports. That way there's no issues.



How do I make sure that the jails use the same ports tree? Is it described in some config file, or do I have too mount the build-jail ports into application jails /usr/ports?


----------



## SirDice (Jun 6, 2012)

Mount your own ports tree in the jail using nullfs(5):

`mount -t nullfs /usr/ports /jails/j1/usr/ports`

I have, in /etc/rc.conf:

```
jail_build_mount_enable="YES"
```

And created an /etc/fstab.build:

```
tmpfs                           /jails/j-build-amd64/tmp                        tmpfs   rw,mode=1777    0       0
/usr/src                        /jails/j-build-amd64/usr/src                    nullfs  ro,noatime      0       0
/usr/ports                      /jails/j-build-amd64/usr/ports                  nullfs  rw,noatime      0       0
/usr/ports/distfiles            /jails/j-build-amd64/usr/ports/distfiles        nullfs  rw,noatime      0       0
/usr/ports/packages             /jails/j-build-amd64/usr/ports/packages         nullfs  rw,noatime      0       0
```

That makes sure the filesystems are automatically mounted/unmounted when the jail starts/stops.

This same ports tree is also NFS exported (read-only) and used on all my other machines.


----------



## Anonymous (Jun 6, 2012)

Okay, easey peasey then.

Any special flag in portmaster to build packages for ports which a new version is available? Or should I build with `# portmaster -g type/port` for the packages that need an update, and then install with `# pkg_add /usr/ports/packages/package.tar.gz`?


----------



## bbzz (Jun 6, 2012)

It's better to update with *portmaster* since it will also update dependencies, whereas *pkg_add* will add missing packages and give you warning about dependencies which are not in check (e.g cairo-1.10.2 needed but cairo_1.10.1 installed, etc).

`# portmaster -PP -D -B --packages-local --local-packagedir=/usr/ports/packages -y`


----------



## Anonymous (Jun 26, 2012)

I've been trying out the things mentioned in previous posts.
And the thing I found working somewhat OK is this.
`host # ezjail-admin update -P` Update ports tree.
`build-jail # portmaster -D -a` To update stuff in the jails.
`app-jail # portmaster -D -B -a` Update ports on specific jail.

My setup looks something like this.
`host # mount | grep ports`

```
/usr/local/jails/build/var/ports on /usr/local/jails/ns1/var/ports (nullfs, local, noatime)
/usr/local/jails/build/var/db/ports on /usr/local/jails/ns1/var/db/ports (nullfs, local, noatime)
```

When using this I had to have /var/ports writable on the app-jail. While this is working as far as I can see, I'm not sure if I'm creating more work for myself .


----------



## hedgehog (Sep 24, 2012)

dougb@ said:
			
		

> Someone else already suggested portmaster as your tool to build packages, I will ever-so-humbly state that I agree.   As long as the infrastructure in the ports tree is correct, using the -g option for portmaster will produce correct packages.



Let's assume I'm trying to install several ports that has dependencies using portmaster. Will the -*g* option let the portmaster create packages for ALL built ports, including dependecies?

PS: I'm sorry for posting into old topic.


----------



## SirDice (Sep 24, 2012)

hedgehog said:
			
		

> Let's assume I'm trying to install several ports that has dependencies using portmaster. Will the -*g* option let the portmaster create packages for ALL built ports, including dependecies?


Yes.


----------



## zeissoctopus (Sep 24, 2012)

I use ports-mgmt/poudriere to update ports tree and build three pkgng repos in jail

each jail just install packages by pkg


----------



## hedgehog (Sep 24, 2012)

zeissoctopus said:
			
		

> I use ports-mgmt/poudriere to update ports tree and build three pkgng repos in jail
> 
> each jail just install packages by pkg


I was thinking of it. Most likely I'll try it later when I have more information about ports-mgmt/poudriere, zfs/jail and so on 

At this time I'm just trying to update the whole old desktop to 9.x with minimal GUI downtime period.


----------



## hedgehog (Sep 27, 2012)

bbzz said:
			
		

> It's better to update with *portmaster* since it will also update dependencies, whereas *pkg_add* will add missing packages and give you warning about dependencies which are not in check (e.g cairo-1.10.2 needed but cairo_1.10.1 installed, etc).
> 
> `# portmaster -PP -D -B --packages-local --local-packagedir=/usr/ports/packages -y`



For some reason it doesn't work... The portmaster build software from sources instead of installing it from local packages directory. For example:

```
portmaster -PP -D -B --packages-local --local-packagedir=/usr/ports/packages/All devel/gettext
```
Portmaster compiles cmake, libtool, libiconv and gettext.

Trying brand new pkg tool:

```
# pkg add /usr/ports/packages/All/gettext-0.18.1.1.txz
Installing gettext-0.18.1.1...Installing libiconv-1.14... done
 done
```
I.e. the pkg picked up packages from /usr/ports/packages/All/ and installed it. I tried to install more complex thing like kde4, worked well. However, I didn't test if the kde runs, because I installed it in the temporary jail.

Am I doing something wrong when I use portmaster?


----------



## hedgehog (Sep 28, 2012)

Fortunately I compiled packages using pkgng compatible portmaster. In this case I can easily create a repo: 
`# pkg repo .`
Then configure host's pkg to use that directory as repository in /usr/local/etc/pkg.conf by specifying path like this: file://localhost/usr/ports/packages

Now the host or another jail can use jailed repo. Of course you can share repo over web server instead


----------



## SirDice (Sep 28, 2012)

Keep in mind portmaster(1) needs a patch to be able to work with pkgng.

https://github.com/pkgng/pkgng/blob/master/ports/patch-portmaster-pkgng

Save this file in the files/ directory of the ports-mgmt/portmaster port.


----------



## hedgehog (Sep 28, 2012)

SirDice said:
			
		

> Keep in mind portmaster(1) needs a patch to be able to work with pkgng.
> 
> https://github.com/pkgng/pkgng/blob/master/ports/patch-portmaster-pkgng
> 
> Save this file in the files/ directory of the ports-mgmt/portmaster port.


It was done before building packages. Looks like portmaster is not ready to install pkgng packages yet:

```
portmaster -PP -D -B --packages-local --local-packagedir=/usr/ports/packages sysutils/bsdadminscripts
===>>> Package installation support cannot be used with pkgng yet,
       it will be disabled
```

Well, I guess that I could use portmaster() to build packages in jail and then install/upgrade the host with pkgng.

By the way, it took like 7 minutes to install KDE from packages that I built in the jail:

```
# pkg search kde
kdegames-4.8.4                 Games for KDE 4
kde4-style-qtcurve-1.8.13      QtCurve KDE4 widget style
kdemultimedia-4.8.4_1          KDE Multimedia applications
kdepim-4.4.11.1_3              Libraries for KDE-PIM applications
kdenetwork-4.8.4_2             KDE Network applications
kdeutils-4.8.4                 Collection of utilities for KDE 4
kdeedu-4.8.4                   Set of entertaining, educational programs for KDE 4 (meta-port)
libkdeedu-4.8.4                Libraries used by KDE educational applications
kdegraphics-svgpart-4.8.4      SVG KPart
kde4-shared-mime-info-1.2      Handles shared MIME database under ${KDE4_PREFIX}
pam_kde-1.0                    PAM modules for KDE Display Manager
kdetoys-4.8.4                  Collection of entertaining programs for KDE
kdegraphics-strigi-analyzer-4.8.4 Strigi analyzers for various graphics file formats
kdepimlibs-4.8.4               Libraries for KDE-PIM applications
kdesdk-4.8.4                   KDE Software Development Kit
kde-4.8.4                      The meta-port for KDE Software Compilation 4
kdeadmin-4.8.4                 KDE Admin applications
kdeplasma-addons-4.8.4         Extra plasmoids for KDE4
kdegraphics-mobipocket-4.8.4   Mobipocket plugins for Strigi indexing, Okular, and thumbnails
ru-kde-l10n-4.8.4              Russian messages and documentation for KDE4
kde-baseapps-4.8.4             Basic applications for the KDE system
kdelibs-4.8.4_1                Base set of libraries needed by KDE programs
polkit-kde-0.99.0_3            KDE4 frontend to PolKit-1
kde-workspace-4.8.4            Basic applications for the KDE system
kde4-xdg-env-1.0.1             Script which hooks into startkde and helps KDE pick up XDG apps/menus
kde4-icons-oxygen-4.8.4        The Oxygen icon theme for KDE
kdeartwork-4.8.4               KDE Artworks Themes
kdepim-runtime-4.4.11.1_2      Libraries for KDE-PIM applications
kdeaccessibility-4.8.4         Accessibility applications for KDE4
kde-wallpapers-4.8.4           Set of wallpapers for the KDE Plasma workspace
kdewebdev-4.8.4                Comprehensive html/website development environment
kde-runtime-4.8.4_2            Basic applications for the KDE system
kdegraphics-thumbnailers-4.8.4 Thumbnailers for various graphics file formats
kdegraphics-4.8.4              Graphics utilities for KDE 4 (meta-port)
kdehier4-1.0.10                Utility port that creates hierarchy of shared KDE4 directories
```


```
pkg install kde-4.8.4
```
At this moment I use another jail to test if I can install these packages.


----------



## SirDice (Sep 28, 2012)

hedgehog said:
			
		

> Well, I guess that I could use portmaster() to build packages in jail and then install/upgrade the host with pkgng.


That's what I'm currently doing. I'm basically only using portmaster(8) to build everything. On my other systems I simply set PACKAGESITE to my server and use pkg(8). Works like a charm. It seems to work slightly faster than my 'old' way of doing pkg_delete(1)/pkg_add(1).


----------



## hedgehog (Sep 28, 2012)

SirDice said:
			
		

> That's what I'm currently doing. I'm basically only using portmaster(8) to build everything. On my other systems I simply set PACKAGESITE to my server and use pkg(8). Works like a charm. It seems to work slightly faster than my 'old' way of doing pkg_delete(1)/pkg_add(1).


Nice, I'm on the right way then. May I please ask you one more question before I start upgrading my desktop?

I upgraded 8.3-release on desktop to 9.1-RC1 without updating third-party software. I got all orphaned packages list from my system, did some cleanup and compiled it using portmaster (pkgng support) within 9.1-RC1 jail.

Now I would like to upgrade ports on the host using the repository I created. In my view I should perform the following steps:

 Make zfs snapshots of /usr/local and /var/db/pkg (just in case)
 Completely wipe all packages:

```
# pkg_delete -af
# rm -rf /usr/local/*
# rm -rf /var/db/pkg/*
```

 Install pkgng, configure PACKAGESITE to use my repository and define WITH_PKGNG=yes in /etc/make.conf
 Install packages from repository using orphanes list
 Delete old libraries from the system: `# freebsd-update install`
 Deal with zfs snapshot
Am I correct?

PS: My current ports on the desktop is really outdated.


----------



## SirDice (Sep 28, 2012)

I'd delete the old libraries right after removing all ports. There's no need for them anymore and it'll prevent accidental use. But besides that it all looks good.

I did notice I had to set PACKAGESITE slightly different for pkgng. Normally you'd point it to the /Latest directory but for pkgng it should be pointing to the root of the package dir.

Old:

```
PACKAGESITE=http://j-internetz/FreeBSD/packages/Latest
```

pkgng style:

```
PACKAGESITE=http://j-internetz/FreeBSD/packages/
```


----------



## hedgehog (Sep 28, 2012)

SirDice said:
			
		

> I'd delete the old libraries right after removing all ports. There's no need for them anymore and it'll prevent accidental use. But besides that it all looks good.


I think you're right. Anyway there is compat8x libraries in ports.



			
				SirDice said:
			
		

> I did notice I had to set PACKAGESITE slightly different for pkgng. Normally you'd point it to the /Latest directory but for pkgng it should be pointing to the root of the package dir.
> [/code]


I noticed it too. I pointed repository to the packages dir that has directories like All, Latest and ports-like structure.

PS: Just installed kde/xorg on freebsd in virtualbox. Worked like a charm with this repo!


----------



## kpa (Sep 28, 2012)

It depends on where the repo.txz file has been created, with PKGNG it's recommended to create it at the top level directory and not in the All -directory.

For example I would create the repo manually like this (ports-mgmt/poudriere does this automatically):

`# pkg repo /usr/local/poudriere/data/packages/release91amd64-default [optional_signing_key]`

The /usr/local/poudriere/data/packages/release91amd64-default directory would have the All, Latest etc. subdirectories.


----------



## hedgehog (Sep 29, 2012)

Thanks everyone, upgrade ran smoothly and quickly. Now it's time to consider writing an article regarding full 8.3-9.1 upgrade and switching to pkgng.


----------



## UNIXgod (Sep 29, 2012)

hedgehog said:
			
		

> Thanks everyone, upgrade ran smoothly and quickly. Now it's time to consider writing an article regarding full 8.3-9.1 upgrade and switching to pkgng.



Do it!


----------

