# The urge to maintain binary packages



## piggy (Jul 26, 2011)

I think someone at FreeBSD should consider to maintain binary packages updated. FreeBSD is often used in production environment and the need to compile all the packages, especially when recompiling is needed, it is really time consuming and a pain in the ass.

In the process of rolling out updates, someone at FreeBSD should consider to offer updates also in the form of binary packages.

I think this could be needed by a lot of professional users and not always.

I personally schedule my updates in one month time, then sometimes in one months there are too much packages to be updated and this create downtimes for important machines, like I heavily experimented this days with ICU update.


----------



## SirDice (Jul 26, 2011)

piggy said:
			
		

> In the process of rolling out updates, someone at Freebsd should consider to offer updates also in the form of binary packages.


They already are.



> I personally schedule my updates in one month time, then sometimes in one months there are too much packages to be updated and this create downtimes for important machines, like I heavily experimented this days with ICU update.


Use a build server and test before putting things in production.


----------



## piggy (Jul 26, 2011)

SirDice said:
			
		

> Quote: Originally Posted by piggy
> In the process of rolling out updates, someone at Freebsd should consider to offer updates also in the form of binary packages.
> 
> They already are.


Well, NOT. Binary are updated just sometimes, not in real time. I talk about to have source and binary updated IN THE SAME TIME.

I think not all the users need optimization and so to compile them packages from source, many just need to be up to date on generic systems and binary could be perfect for this.


----------



## SirDice (Jul 26, 2011)

piggy said:
			
		

> Well, NOT. Binary are updated just sometimes, not in real time. I talk about to have source and binary updated IN THE SAME TIME.



ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/
http://pointyhat.freebsd.org/errorlogs/
http://pointyhat.freebsd.org/errorlogs/packagestats.html (note the dates)

I'm sure they could use some extra hardware you have lying around. With enough hardware and people maintaining it they might do a package build everyday. But with the limited resources they have they simply can't.

If you are serious about maintaining a large amount of machines, setup your own build server. The effort in setting it up will pay for itself in the long run. You might want to check out ports-mgmt/tinderbox.


----------



## tingo (Jul 27, 2011)

piggy said:
			
		

> I think someone at FreeBSD should consider to maintain binary packages updated. FreeBSD is often used in production environment and the need to compile all the packages, especially when recompiling is needed, it is really time consuming and a pain in the ass.



Cool idea! Why don't you do it?


----------



## piggy (Jul 27, 2011)

SirDice said:
			
		

> They already are.
> 
> 
> Use a build server and test before putting things in production.





			
				tingo said:
			
		

> Cool idea! Why don't you do it?


Useless post.


----------



## fadolf (Jul 27, 2011)

I would recommend ports-mgmt/tinderbox-devel since it supports many more advanced features, like mass-queuing and parallel builds.


----------



## mix_room (Jul 27, 2011)

piggy said:
			
		

> Useless post.


No. Since they explain the problem. 

The binary packages are updated, which might lead you to believe that there is someone out there, the pointyhat cluster, that builds them. Since they are not always updated they are obviously not able to keep up. 

So you have two solutions:
a) build your own binary packages before rolling them out.
or
b) helping speed up the building. Either by donating time, knowledge, money or hardware. Which ever you have most to spare. 

According to http://www.freebsd.org/ports/ there are over 22'000 ports. How long do you think it takes to build these for the 6 different architectures which FreeBSD supports. I think it would take a long time.


----------



## jrm@ (Jul 27, 2011)

I agree, it would be nice to have more frequent updates to the binary packages, but for the reasons already mentioned it's not currently feasible.  Your tone suggests a sense of entitlement (somebody should fix my problem for me).  FreeBSD exists mostly because of a volunteer community.  Imagine if you volunteered to help someone and that person used a tone like you did.  Constructive feedback would likely come across better.  Better yet, like others have suggested, you could take a stronger role within the community.


----------



## piggy (Jul 27, 2011)

fadolf said:
			
		

> I would recommend ports-mgmt/tinderbox-devel since it supports many more advanced features, like mass-queuing and parallel builds.


Cool. Seems really nice and it seems it could be usefull for my needs. I have to maintain like 10 machines, three are production machines, they serve web pages, streams and files to a not so little number of clients , so I need to organize all this updates runs better.

The point is: should I do like many and just update the system when security updates are out or should I do like I try to do to maintain - as possible as I can - all up to date?

Yep, and frequent binary updates can help to spread Freebsd in the world too, becouse, u know, with Red Hat or SuSE server software (just to name the kings of the bill), you don't need to compile for hours just to update your servers. You got fresh binary ready to use.

Then if for Freebsd resources are limited, ok, don't do it, obviously I don't pretend nothing, I can live without it like I always did. Many people in this business will continue to call me a crazy person becouse I try to maintain an infrastructure of Freebsd servers compiling software all the time.

And honest, many comments of some people here (others' been helpful and I want to thank you all) are just frustrated geeky rants: no need of this understatement attitude, no need to specify everything is on volunteer basis and so on, I know it, and I do appreciate it, this doesn't mean I can't propose something I think usefull and yes, maybe I can help too.

For me honest, personally, I like to learn, then - if I could choose (AKA If we had the money to buy the very expensive licenses) - I will run problem less Windows Servers software: just a click to get everything updated and now I consider Microsoft software the most complete and professional around (apart some PCBsd/Freebsd and Archlinux boxes everything I do administer on client sides are Windows boxes).

As I said I have to stay very low with the costs and so I pay with my time and efforts and I run Freebsd becouse I think it runs better compared to Linux, especially on tight hardware. I also like to learn and thats all. No religion, no pity, no cry, nothing at all related with it, but just a big thank you to developers.

If I think I become sick of it, having sever working 50 per cent becouse I'm compiling and compiling and compiling updates, I do like what I did ten years ago (probably more), when I started with Freebsd at version 4, I will run Linux as a server again on our infrastructure and I come back to Freebsd in like 10 other years and five releases after if I feel the need. For me don't change nothing: I look for the less expensive and powerfull platform I can have, regardless of it's name and architecture.

Thats all.


----------



## gkontos (Jul 27, 2011)

piggy said:
			
		

> For me honest, personally, I like to learn, then - if I could choose (AKA If we had the money to buy the very expensive licenses) - I will run problem less Windows Servers software: just a click to get everything updated and now I consider Microsoft software the most complete and professional around (apart some PCBsd/Freebsd and Archlinux boxes everything I do administer on client sides are Windows boxes).


Just a click and pray that everything will work again after. If you ever work in an environment that availability is a must you will see that most windows sys admins are scared to update their machines because of this.

I personally would like to see a binary package manager for FreeBSD but mainly for desktop use. Servers usually don't need so much software installed so, compiling time isn't that time consuming. Besides, when you upgrade your software from ports you know exactly what and when to do and it is very rare that you find yourself in a situation where some other update prevents a service from coming up.


----------



## sossego (Jul 28, 2011)

pkg_create()

Build your packages on one system and distribute it to the other machines by network.
You could also dedicate a desktop system with a similar CPU to FreeBSD and also make it the package repository. If all machines are within the same LAN, then just point it to the desktop repo. One can make a DVD, CD, or add packages to a USB device and mount that after the initial install.

Maintaining packages for one architecture is work enough for a single person. You could upload packages to googlecode and then use 
	
	



```
ftp /path/to/googlecode/repo
```
 with options for recursive copying of files. Or, you could set PKG_PATH to the repo there.


----------



## SirDice (Jul 28, 2011)

For a very long time I've used a simple jail to build all my own packages. Just create a jail and start it. Then jexec into it and start building. Make sure /usr/ports/packages/ exists and use *make package* or *make package-recursive*. You can also use the *-g* switch with ports-mgmt/portmaster. 

NFS export the resulting /usr/ports/ read-only or just share /usr/ports/packages/ on a web server. With a web server you can set PACKAGESITE to your own distribution server and use pkg_upgrade(1) from sysutils/bsdadminscripts to do a binary upgrade.


----------



## SirDice (Jul 29, 2011)

gkontos said:
			
		

> Just a click and pray that everything will work again after. If you ever work in an environment that availability is a must you will see that most windows sys admins are scared to update their machines because of this.


Fire those admins because they are clearly not doing their job properly. If availability is that much of an issue you build a HA solution, not single machines.


----------



## expl (Jul 29, 2011)

gkontos said:
			
		

> I personally would like to see a binary package manager for FreeBSD but mainly for desktop use.



Check my signature.


----------



## gkontos (Jul 29, 2011)

expl said:
			
		

> Check my signature.



Looks very promising. I will definitely give it a shot for my desktop.


----------



## jrm@ (Jul 29, 2011)

What prevents users from submitting packages?  I'm guessing this must have been suggested before, but I don't see it here: http://www.freebsd.org/doc/en/articles/contributing-ports/article.html.  I'm guessing security issues, but if they could be minimized, it would be cool to tap the resources of individual users this way.


----------



## sossego (Jul 30, 2011)

Tier2 architectures may be more of a candidate for your idea.


----------



## namor (Jul 30, 2011)

mingrone said:
			
		

> What prevents users from submitting packages?  I'm guessing this must have been suggested before, but I don't see it here: http://www.freebsd.org/doc/en/articles/contributing-ports/article.html.  I'm guessing security issues, but if they could be minimized, it would be cool to tap the resources of individual users this way.



How do you think, the issues of users submitting malware could be minimized? The only think that comes to mind is having at least _n_ users commit the exactly same package and compare the checksums. But that idea seems useless, someone with access to enough boxes will probably be able to work around measures to ensure _n_ *different* users commit the packages.


----------



## gkontos (Aug 1, 2011)

@expl

Probably a silly question but:


```
> bxpkg 

(process:26247): Gtk-WARNING **: This process is currently running setuid or setgid.
This is not a supported use of GTK+. You must create a helper
program instead. For further details, see:

    http://www.gtk.org/setuid.html

Refusing to initialize GTK+.
```

Any ideas ?


----------



## expl (Aug 2, 2011)

gkontos said:
			
		

> @expl
> 
> Probably a silly question but:
> 
> ...



Is the user you running it with part of "wheel" group? It has to be or the inbuilt authentication will not work. A work around would be to run it as root directly. Or run it with "-r" switch, that will force it to read-only mode and you will not be able to delete or install anything, just to inspect.


----------



## gkontos (Aug 2, 2011)

expl said:
			
		

> Is the user you running it with part of "wheel" group? It has to be or the inbuilt authentication will not work. A work around would be to run it as root directly. Or run it with "-r" switch, that will force it to read-only mode and you will not be able to delete or install anything, just to inspect.



Thanks for your reply. My account is in the wheel group and if I try to run it as root I get:

```
core2duo# bxpkg 
No protocol specified

(bxpkg:3333): Gtk-WARNING **: cannot open display: :0.0
```
So, I suspect that the problem is somewhere else.


----------



## fonz (Aug 2, 2011)

gkontos said:
			
		

> ```
> core2duo# bxpkg
> No protocol specified
> 
> ...


That can be solved with xhost(1). As the user running the X sesion, type either
`% xhost +`
or
`% xhost +[i]canopy[/i]` where for _canopy_ you need to fill in your hostname.

This may not solve your original problem, but it should solve the "cannot open display" thing.

Fonz


----------



## gkontos (Aug 2, 2011)

@fonz,

My desktop is running the X-Server so this is not the issue.


----------



## expl (Aug 2, 2011)

gkontos said:
			
		

> @fonz,
> 
> My desktop is running the X-Server so this is not the issue.



Then the hostname is "localhost", clients need to know that.


----------



## DutchDaemon (Aug 3, 2011)

This has now gone way off-topic. Either wrap this up or take it to a new topic.


----------



## piggy (Aug 4, 2011)

DutchDaemon said:
			
		

> This has now gone way off-topic. Either wrap this up or take it to a new topic.


I do agree this thread is going OT, then I'm not agree it is solved. It will be solved when Freebsd foundation do have a binary manager. So it is not solved.


----------



## vermaden (Aug 4, 2011)

There is also other 'temporary sollution' for that problem (binary updates).

Use pkg_upgrade from sysutils/bsdadminscripts to update single packages (pkg_upgrade -r -R mplayer) or whole collection of packages (pkg_upgrade -a -C), it will grab latest available packages built for the STABLE tree.

Other sollution is to use portmaster but only when recompiling is needed (audio/lame for example) or when portaudit will show some vulnerabilities and binary package is still not built.

Other useful command is to use portmaster --check-depends after pkg_upgrade work is done.

I generally check for new packages like that:
# portsnap fetch update (update the _Ports _tree)
# portmaster -L --index-only | grep -i ' new ' (check what new versions are available)
# portaudit -Fda (check vulnerabilities)
# pkg_updating -d $( ls -ltr -D '%Y%m%d' /var/db/pkg | awk 'END{print $6}' ) (check notes from /usr/ports/UPDATING)

... then update everything that really needed to be updated (not only because the are newer versions) with

# pkg_upgrade -a -C (update everything binary)
# portmaster --check-depends (fix +REQUIRED_BY dependencies)

... or by source way if there is no other way, for all newer packages:

# portmaster -y --no-confirm -a --packages-if-newer -m 'BATCH=yes' -d
# portmaster --check-depends"

... or just a single port/package:

# portmaster -y --no-confirm --packages-if-newer -m 'BATCH=yes' -d ${NAME}
# portmaster --check-depends"


----------



## jrm@ (Aug 4, 2011)

namor said:
			
		

> How do you think, the issues of users submitting malware could be minimized?



I don't know, that's why I asked.

Right now we are trusting someone associated with the project who builds packages and puts them on FreeBSD servers.  Yes, that's a heck of a lot better than trusting a package submitted by some random user.  But, I just (naively) thought that with all those idle CPUs out there and the need for packages there might be some solution.  Our group has a cluster with over a 100 CPUs and there are days when they are all idle.  I would love to be able to donate packages, if I could somehow get credentials/trust to do so.  

The problem reminds me off some chatter a few years back about distributed compilation over networks.

In the meantime, like others have suggested, build servers are pretty helpful.  I just do:

On the build server:
`% sudo portsnap fetch update; pkg_version -Ivl'<' # check for new ports`
`% sudo portmaster -g <some_big_port>`,
make /usr/ports/packages web accessible

On the slow boxes:
`% sudo portsnap fetch update`
`% sudo env PACKAGESITE="http://<buildserver>/pkg/" portmaster -PP <some_big_port>`.


----------



## SirDice (Aug 5, 2011)

mingrone said:
			
		

> But, I just (naively) thought that with all those idle CPUs out there and the need for packages there might be some solution.  Our group has a cluster with over a 100 CPUs and there are days when they are all idle.  I would love to be able to donate packages, if I could somehow get credentials/trust to do so.


Hmmm... Instead of Folding@Home we could have Packaging@Home :e


----------



## ahavatar (Aug 5, 2011)

It's a logistics problem. Some ports are stand-alone and it's doable almost in real time to offer binary packages, but a port like perl - all the ports depending on it need to be updated, and there are hundreds of ports depending on perl. Doing such many ports real time is a serious business(tm).


----------

