# poudriere seems to like to rebuild everything



## Alain De Vos (Jun 27, 2020)

I would like to rebuild only what is necessary.
In a file pkglist.txt i have 400 ports listed
This is how i run poudriere.

```
pkglist="./pkglist.txt"
jail="poudbjail"
port="local"
time nice -n 15 poudriere bulk    -j $jail -p $port -f $pkglist
```

This command currently want to build 700 packages. I want poudriere to stop rebuilding everything when it is not needed. Maybe something in poudriere.conf ?

When i put only one port in pkglist.txt poudriere seems to be clever enough to know a rebuild is not needed. With 400 ports it just rebuilds everything.


----------



## zirias@ (Jun 27, 2020)

poudriere never rebuilds "everything". But it's "playing it safe" with dependencies -- whenever a port needs to be rebuilt, all dependent packages are deleted as well, so they will be rebuilt, too. Often, this wouldn't be really necessary (e.g. with RUN_DEPENDS, or LIB_DEPENDS if the library ABI doesn't change), but poudriere currently has no way to figure that out. The only thing you can do is to use ccache; at least with C and C++ software, this will avoid a lot of unnecessary recompiles.


----------



## garry (Jun 27, 2020)

Zirias said:


> poudriere ... is "playing it safe" with dependencies -- whenever a port needs to be rebuilt, all dependent packages are deleted as well, so they will be rebuilt, too. Often, this wouldn't be really necessary ... but poudriere currently has no way to figure that out.


Humans don't necessarily have a way to figure that out either.  For several years I used Archlinux and, at least at that time and probably still, the developers did not "play it safe" and rebuild dependencies when a package was updated.  The new package was rebuilt and the binary tossed into the pile for everyone to use.  Usually they caught the need to rebuild depending packages but sometimes not, and then there would be bugs, messages in the forum, and a dev would realize that "oops, we need to rebuild X for the new Y".  Mysterious crashes, strange bugs, frequent "fixes" were the price of avoiding "massive rebuilds".  I could live with that, but I know that it's not the FreeBSD way.  The only way to guarantee that packages that depend on one another play nice together is to build them together.  FreeBSD has the infrastructure for that and keeps "perfect" binary repositories.

The poudriere and synth approach guarantees a "clean" _repository_.  `pkg` does not necessarily keep the _installation_ that clean.  To keep a perfectly consistent installation you either use an even more advanced package manager such as Guix or do a complete re-install for each upgrade (remove all packages, then re-install from the clean repository).  When I do build my own repository with poudriere or synth I do a *complete* re-install for each upgrade.  Since the packages are already on the local disk a complete re-install of all packages does not take long, just a couple of minutes.

If you aren't willing to build a clean repository then don't bother with poudriere.


----------



## Jose (Jun 28, 2020)

Same story on Gentoo





						Gentoolkit - Gentoo Wiki
					






					wiki.gentoo.org
				








						Project:Perl/perl-cleaner - Gentoo Wiki
					






					wiki.gentoo.org
				




I'm convinced the live system and the build system should be separate beasts now.


----------



## Alain De Vos (Jun 28, 2020)

Rebuilding 700 packages takes around two days, not minutes
In the winter I can use the CPU-heat, but in the summer it is lost energy.
I have 8G or ram and wired + active memory is 4G leaving at max 4G for builds.
What ccache size would you advice to speed up ?
How do i make sure the cache is used by all building jails.


----------



## zirias@ (Jun 28, 2020)

Use ccache. If you still think there's too much time wasted, abandon poudriere, use something simpler (or just manually use the ports tree) and conciously opt for the occassional breakage you will experience by *not* having rebuilt dependent ports


----------



## zirias@ (Jun 28, 2020)

Just noticed your edit ...


Alain De Vos said:


> What ccache size would you advice to speed up ?


ccache is on disk, so "ideal" size doesn't depend on your available RAM. IMHO, the ideal size is such that the results of a complete build of your whole package set fits. My package list contains a few huge desktop ports, so I use `max_size = 20G`


Alain De Vos said:


> How do i make sure the cache is used by all building jails.


You don't have to worry about that, poudriere integrates ccache pretty well. Just set a `CCACHE_DIR` (containing a ccache.conf) in /usr/local/etc/poudriere.conf and poudriere will use it. Of course, there are ports that can't make use of ccache, either because they use something other than C or C++, or because their build-system is "broken".


----------



## Lamia (Jun 28, 2020)

You can also use swap file given that you don't have so much RAM. It plays well with Poudrière. Ccache is important given your situation. Your buildsystem can take advantage of already built dependencies in cache. A Big swap file md99 will help with swap error owing to lack of ram.


----------



## Alain De Vos (Jun 28, 2020)

The problem with "swapping"' even on SSD, is that at that moment real-time stuff like playing a simple youtube music video stops functioning.
On the danger of nagging, I do not mind compiling "bash" each day, but do not want to compile "rust" or "llvm" each day.


----------



## Lamia (Jun 29, 2020)

It is invaluable for such a small RAM even if you need prevent building rust, llvm, gcc, chromium, firefox, etc.  You can also make some reasonable changes to sysctl.conf e.g. hw.page.size.


----------



## astyle (Oct 19, 2022)

garry said:


> The only way to guarantee that packages that depend on one another play nice together is to build them together. FreeBSD has the infrastructure for that and keeps "perfect" binary repositories.





Alain De Vos said:


> On the danger of nagging, I do not mind compiling "bash" each day, but do not want to compile "rust" or "llvm" each day.


There are a couple possibilities in poudriere.conf... `CHECK_CHANGED_DEPS` and `BAD_PKGNAME_DEPS_ARE_FATAL`... in my case, I want to update KDE ports only, and build them against what's already installed. I have ~400 such ports for my whole tree, and need to build ~200 of those for a functional desktop. So I give Poudriere the list with 200 ports, it wants to rebuild 100 more - even though they're already installed, they're the stuff I want to build against!


----------



## facedebouc (Oct 19, 2022)

Alain De Vos said:


> Rebuilding 700 packages takes around two days, not minutes
> In the winter I can use the CPU-heat, but in the summer it is lost energy.


That's why I decided to rent a VPS for this use. This VPS will also host my HTTP, Mail, and backup servers for my personal use. I found a bigger VPS than the one which currently host my HTTP and Mail servers for a great price ;-)
The annual cost of this VPS will be lower than the electricity bill of my own poudriere build machine at home.


----------



## Alain De Vos (Oct 19, 2022)

Please elaborate , can you specify exactly which VPS at which price.
And then after a repository is build we should be able to transfer it easily to the desktop PC you are running ...


----------



## facedebouc (Oct 19, 2022)

Contabo Cloud VPS S
My poudriere repository is served by a HTTP server in jail on this VPS and my local network machines get their packages from this VPS HTTP server.


----------



## facedebouc (Oct 19, 2022)

I took the offer with double SSD storage size for limited time


----------

