# Call for discussion: creating a FreeBSD software repository



## fraenki (Jan 23, 2009)

Hi,

I want to know your opinion!

Updating packages from source with a local copy of the FreeBSD
ports tree offers great flexibility. You can choose from almost
20.000 ports and different versions of the same software. Besides
that you can optionally configure the software and environment 
with just the options that fit your needs.

But when it comes to larger environments this flexibility sometimes
lead to annoying surprises. Especially when trying to upgrade a
large farm of servers with different sets of ports installed and
even different port-options for the same software.

There is a good chance that updates will behave differently from 
server to server. Thus automating the maintenance becomes tricky.

I decided to stop updating from the local ports tree. Instead I set 
up a server with Tinderbox. I've defined some Tinderbox BUILDs
to represent the different package sets:

7.1_freebsd_i386_build1 (apache22, php5, ...)
7.1_freebsd_i386_build2 (database ports)
7.1_freebsd_i386_build3 (apache13, php4, ...)

Additionally I configured the port options and build environment
to match my needs. This makes every of the above build a unique set
with different dependencies.

Now I have ready-to-use package sets that just needs to be installed.
I found the remote fetching feature of _pkg_add_ somewhat useful.
Also the _-PP_ option found in _portupgrade_ helps installing
the packages and keeping them in sync with the server that builds the
package sets.

But this is not enough. pkg_add does it's job quite well, but you have
to do each update manually. Portupgrade automates the update process,
but it depends on a local copy of the portstree... What if the local
ports tree does not exactly match the version I used for building the 
packages? How do I tell all my servers to use a specific version of the
ports tree? [...]

Well, I think for this purpose I'd need somethink like yum or
the smart package manager. Both use a repository that provides
information about the packages it holds, and the client basically just 
takes care for installing, deleting, updating and resolving conflicts.

So the basic idea would be to take the tinderbox approach and build a
toolset around it that collects the meta-data from the +CONTENS file 
of every package. This data could be saved as an XML file. The Origin
of a package (shells/zsh) is the unique identifier and the package name
(zsh-4.3.9_5) is the base for version comparison.

The client would just do the same: generate a XML file from the all
files found in /var/db/pkg/*/+CONTENTS. Afterwards it downloads the XML
file from the repository. So this is the base for comparing versions,
beeing aware of conflicts, and so on... 
Of course the repository XML file could also contain information from
/usr/ports/MOVED so we know about origin changes or removed ports...

Basically it's just meant as a addition to the ports system, not as
a replacement, because the whole repository idea really depends on the
ports system.

I think this is not only interesting for server farms, but also for
other distributions of FreeBSD: desktop versions of FreeBSD, ...


fraenki


----------



## Oko (Jan 23, 2009)

fraenki said:
			
		

> Hi,
> Well, I think for this purpose I'd need somethink like yum or
> the smart package manager. Both use a repository that provides
> information about the packages it holds, and the client basically just
> takes care for installing, deleting, updating and resolving conflicts.



Are you aware of the pkg_mgr tool from OpenBSD? Have a look before you decide to use Yum, apt-get, and similar Linux tools.
http://dawn.rhaalovely.net/pkg_mgr/


----------



## trev (Jan 24, 2009)

fraenki said:
			
		

> Portupgrade automates the update process,
> but it depends on a local copy of the portstree... What if the local ports tree does not exactly match the version I used for building the packages? How do I tell all my servers to use a specific version of the ports tree? [...]



This issue is easily solved by using the same ports tree mounted via NFS - that what I do and it works well for my servers. It also means only one set of ports files needs to be retrieved and so is port server friendly.


----------



## fraenki (Jan 24, 2009)

Oko said:
			
		

> Are you aware of the pkg_mgr tool from OpenBSD? Have a look before you decide to use Yum, apt-get, and similar Linux tools.
> http://dawn.rhaalovely.net/pkg_mgr/



Thanks for the hint, I'll have a look at pkg_mgr. Please don't get me wrong: I don't want to use RPM packages or similar Linux stuff. I just want a package manager for FreeBSD packages that can handle a software repository, not just the ports tree. 

I would be glad if any tool would support the FreeBSD package format, but Yum is hardwired to RPM. Maybe Smart could be extended to handle the FreeBSD package format, but I suppose this will greatly outrun my programming skills.

So currently I try to get the big picture of the current situation: what is available that could help, what are people thinking about this idea, and why doesn't the repository-thingy exist yet. Seriously.

I really want all the pros and cons.

fraenki


----------



## fraenki (Jan 24, 2009)

trev said:
			
		

> This issue is easily solved by using the same ports tree mounted via NFS



Pragmatic approach, nice  However it does not solve it for me, because NFS can't be used everywhere.


----------



## mk (Jan 24, 2009)

1. make a building machine - take duo or quad cpu, nice load of ram
2. setup multiple jails 
   each jail have his own /etc/make.conf
   set a proper env var for TARGET, TARGET_ARCH etc..
3. share /usr/ports btw all ( read only nfs export )
   chek /usr/ports/Mk/bsd.port.mk for WRKDIRPREFIX,WRKDIR etc to 
   change where are port_name/work dirs are made (remeber is nfs      -ro and shared btw multiple hosts)
4. tune kernel,world,ports at will
5. setup test machine for kernel/world to see does it gone blow in your face or not 
6. use pkg_create to build packages from already tuned and   installed ports
7. redistribute over nfs 
8. change PACKAGESITE env var on target install machine to her specific jail builder
9. pkg_add -r xyz
happy building


----------



## danger@ (Jan 24, 2009)

why?


----------



## mk (Jan 24, 2009)

danger@ said:
			
		

> why?



i don't get that?


----------



## danger@ (Jan 24, 2009)

I mean why the poster cannot use NFS on each of his systems?


----------



## fraenki (Jan 24, 2009)

danger@ said:
			
		

> I mean why the poster cannot use NFS on each of his systems?



Several situations can occur that make it impossible (political, not technical) to use NFS. Multiple datacenters, different network administrators or people involved that argue against NFS.

I think NFS serves good for several tasks (on a local network), but I wouldn't use NFS as base for a software repository. HTTP(S) and FTP would fit much better.


fraenki


----------



## danger@ (Jan 24, 2009)

I thought you have been talking about a package repository for a server farm under your administration (or your team).

Multiple datacenters shouldn't be a problem, you can always create VPN...


----------

