# Preserving Successful Builds with Poudriere and git, rsync, or similar



## gladiola (Jun 29, 2020)

Got my poudriere server up and running fairly steady.  However, sometimes I see a package that I want built will succeed in one run but fail in another.  I would like to know how others preserve their successful builds in these situations.  How do you do it?




For example, with a build in February, I was able to get Chromium 79 to build.  Later, in June, Chromium 83 would fail.  Before I began the build, I didn't know Chromium would fail.  It seems to me that when we initiate the poudriere build, we are always at risk that a build will fail.  Further, we are creating a situation where we know we are overwriting our past jails.  A backup scheme might save us. 

When building with poudriere, what kind of backups do you use to preserve the successful builds?  Do you snapshot each jail and zfs send and receive somewhere?  Do you put the successful binaries in a git repo?  rsync?  What do you do? 

Is there a way to script a poudriere command that will report the successful from failed builds?  Somehow the web server tied to poudriere knows.  Where and how does it find that information?  Can this be readily discovered to identify a successful build held in a jail?  If one is identified, is it possible to also identify the matching dependency ports and their builds, which were also successful?

Has someone already built such a program or features of a program?  Where should we look for those?  It seems like, to be practical, a backup scheme would want to filter for the last successful constructions; maybe keep the last "n" number of binaries that worked; this way, the storage would not be clogged with more unsorted examples of package construction failure.

I am about to try to devise such a plan for myself, and any advice you might offer could be helpful.  Thanks.


----------



## zirias@ (Jun 29, 2020)

I'm not sure I really understand your question. It doesn't make sense to mix a package from a previous build into a repository of a more recent build, as chances are they will not be compatible (linking to the wrong versions of libraries etc...)

So what I do is, if a build has failed packages, fix the problems and retrigger the build.

On a side note, poudriere doesn't have a webserver. It has a few static assets and a javascript that parses information out of the logs directory to display. So, you find all information in the logs. If there's a subdir errors in the log dir of your current bulk run, you have packages that failed.


----------



## SirDice (Jun 29, 2020)

Zirias said:


> On a side note, poudriere doesn't have a webserver.


Actually, it does. Well, sort of anyway. Look in /usr/local/share/examples/poudriere/.


----------



## zirias@ (Jun 29, 2020)

And how is this a "webserver"? It's example configs for other webservers to serve the static assets that come with poudriere. poudriere.js will then work in the browser, which requires the logs to be served as well.


----------



## SirDice (Jun 29, 2020)

Zirias said:


> It's example configs for other webservers to serve the static assets that come with poudriere.


It's not exactly static though. It uses the data to create a dynamic page. But yes, it's not an "interactive" interface to poudriere, I can agree to that. All it does is format the available information in a nice way.


----------



## zirias@ (Jun 29, 2020)

SirDice said:


> It's not exactly static though.


For the webserver that serves it (which is external to poudriere, you need e.g. nginx), it is. All interactivity is implemented in the javascript running in the browser. So that's all I said above, asking how "the webserver knows something" is technically the wrong question, it doesn't know anything


----------



## gladiola (Jun 29, 2020)

The goal is to retain successful builds of a program.  In the jail pictured above, we've tasked poudriere with building some browsers:  Chromium, Firefox, and Lynx.  Well, on a previous run Chromium built just fine.  On a recent run it failed.  Imagine if we could get a client to call packages not from the latest build, but from the latest successful build.  Or maybe from the latest "n" successful builds back.

Is there a good pattern to retain those successful builds in a repo as an archive?

Maybe that archive should be the target for client pkg calls?

Is there a way to specify which build pkg calls from, per program?  When I configure pkg on the client, I generally point it to the poudriere server's jail.  Since I am building programs in multiple jails, I just line up the jails to be used one behind the other in the config.  But, within each jail, there are varying degrees of success.  The clients receiving the packages would similarly operate the programs from jails which would pull their pkg from X poudriere server jails.

It seems to me that, rather than having clients filter a server's files based on scattered criteria, it would be better to point clients to a repo that is filled only with successful builds.  And maybe from there, allow them to choose older successful builds as needed.

_On the web server part:_  My poudriere server is hooked up to nginx; but, I did not have to draft a program to instruct the web server what to show.  I merely pointed nginx to the poudriere installation, and somehow all of that was figured out.  I configured the server to look in two directories: /usr/local/share/poudriere/html and /usr/local/poudriere/data/logs/bulk.  I followed this reference as a guide for implementing poudriere:  https://www.digitalocean.com/commun...m-to-create-packages-for-your-freebsd-servers .



Zirias said:


> So what I do is, if a build has failed packages, fix the problems and retrigger the build.


OK, sometimes I see runs where merely retriggering the build will lead to a successful build.  I have other runs where a program build may succeed in one jail but fail in another (looking at you, rust, python37, gcc9, llvm80, and llvm90).  It can be that, as an operator, we might not need to do anything at all to fix the build besides try again.  In some other instances, the build might bust because there's some other problem beyond our control.  Either way, we would want a steady access to "n" past successful builds and maybe the successful builds of their associated dependencies.

If something came up, and we knew that X months ago we had a successful build of a program that we'd like to rollback to today:  what would be the best way to do that?  It seems to me that hooking up poudriere to a strong backup plan would pave the way for success.  Without it, then an operator might be lost or risking loss with each run of build attempts.  So, there's the question:


gladiola said:


> sometimes I see a package that I want built will succeed in one run but fail in another. I would like to know how others preserve their successful builds in these situations. How do you do it?


----------



## zirias@ (Jun 29, 2020)

I still don't get what's the actual problem behind that question. Say your recent build had n failing packages -- then these packages (and, if applicable, packages depending on them) will miss from the repository. This is exactly what happens with official FreeBSD repositories as well. So, what's the consequence? Someone using this repository can't install this particular package for a while. If an older version is already installed, it will not be lost. Is your concern that someone could accidentally remove a package for not reading the `pkg upgrade` output before hitting `Y`?

Having multiple versions of a repository doesn't sound like a useful solution to me, should a client just switch the repository and reinstall everything? Normally, you just move forwards, if there's something to fix to make the build succeed again, you do it or wait for upstream to do it and after the next build, the package is available again.

Don't get me wrong, maybe I still misunderstood the actual problem or your idea...


----------



## gladiola (Jun 29, 2020)

We're OK.  I'm just asking and you're just answering.  It's fine.  

I will not only want to automate the builds, but automate the client calls to get recent builds.  



Zirias said:


> If an older version is already installed, it will not be lost.


... on the client, yet.  If I automate a client's calls to check for packages which can be updated, then how will I prevent the client from loading a broken package in a simple way?  Also, suppose I would want a client to use not the latest version of a package, but version X from some builds ago.  It seems without filtering the packages built by poudriere and holding them in a repo that it would not be likely to have an easy way for a client to automatically call only a working package.  

Maybe, in the case of the dependencies, part of the answer might be to tighten up what's in a client's jail.  That is, when a client uses a program in a jail, everything in that jail must work together.  If we could assume our way to be blind to that point for now, then perhaps we might see that having a repository filled with successful builds could be important.


----------



## Lamia (Jun 30, 2020)

gladiola said:


> I will not only want to automate the builds, but automate the client calls to get recent builds.


You can write a script to automate Poudrière Builds. And you can take a look at the buildbot package in the freshports repo.

For those difficult packages - Firefox, chromium, etc - you can use 'poudriere testports' command. Firefox would fail if rust fails to build so, you can use a script to check the rate of success and prevent the last package from being deleted before the build starts.


----------



## gladiola (Jul 5, 2020)

I have noticed that there is a setting in poudriere.conf for preserving old packages.  Defaults to 5.  I was unaware of this, and I have not experimented with it yet.  Thanks to everyone who replied.  I am focusing on getting poudriere to run better; perhaps this will alleviate some of my anxiety about preserving builds.  Also, maybe I will find a way to use this preserving old packages option.


----------

