# synth > poudriere?



## toprank (Jan 25, 2018)

Is synth(1) not just a newer but a better alternative to poudriere (as well as portmaster)?

It appears that it has all the functionality of poudriere plus more. For example, does setting the "Fetch prebuilt packages" to active when running `synth configure` act the same as `portmaster --packages-build` (i.e., does it "use packages for all build dependencies")? synth(1) suggests as much:



> When active, this option    will scan the external
> repository for suitable prebuilt    packages to
> satisfy build requirements.  *To be considered
> suitable, the ABI, options, and dependencies of
> ...



However, what would need to be set to maximise suitability (cf. synth(1) quote emphasis). I assume pkg(8) and the ports tree both need to be configured for the _Latest_ packages. But what else? Using packages for all build dependencies is a massive optimisation, and I'd like to make use of it as often as possible.


----------



## swegen (Jan 25, 2018)

You can also follow the latest quarterly ports branch with `svnlite` which matches to the quarterly packages.


----------



## Datapanic (Jan 25, 2018)

I use ports-mgmt/poudriere to build packages for FreeBSD 10.4 i386, amd64 and FreeBSD 11.1 i386 and amd64 all on one machine.  I'm not sure if ports-mgmt/synth can build packages for different FreeBSD versions and architectures.  Poudriere uses Jails while Synth uses chroot.


----------



## SirDice (Jan 25, 2018)

toprank said:


> It appears that it has all the functionality of poudriere plus more.


Likewise, Poudriere has features Synth doesn't have. Like building packages for different versions of FreeBSD and the ability to build ARM/ARM64 packages on AMD64/i386. Also note that Poudriere is used on the official FreeBSD package build clusters.

One tool isn't meant to replace another, they're different tools with different use cases. Use the tools which best suits _your_ situation.


----------



## marino (Jan 25, 2018)

SirDice said:


> Like building packages for different versions of FreeBSD and the ability to build ARM/ARM64 packages on AMD64/i386.



For the record, Synth can build ARM/ARM64 packages on x86 and also ARM64 packages natively, so that's not a feature that poudriere has over Synth.


----------



## toprank (Jan 26, 2018)

Datapanic said:


> I use ports-mgmt/poudriere to build packages for FreeBSD 10.4 i386, amd64 and FreeBSD 11.1 i386 and amd64 all on one machine.  I'm not sure if ports-mgmt/synth can build packages for different FreeBSD versions and architectures.  Poudriere uses Jails while Synth uses chroot.



As far as I can see, this (building packages for different versions) is the only feature poudriere provides that as yet isn't available in synth(1). Nevertheless, as SirDice alludes to below, in my case, it isn't as important a feature as the resource optimisation from using packages for build dependencies.



SirDice said:


> Likewise, Poudriere has features Synth doesn't have. Like building packages for different versions of FreeBSD and the ability to build ARM/ARM64 packages on AMD64/i386. Also note that Poudriere is used on the official FreeBSD package build clusters.
> 
> One tool isn't meant to replace another, they're different tools with different use cases. Use the tools which best suits _your_ situation.



In this case, I think the developer has suggested that it is an intended replacement for portmaster(8) at least, if not poudriere. Which makes sense given that portmaster(8) doesn't have any feature not found in synth(1). Further, freshports.org states that, _"It is intended to replace Portmaster, portupgrade, and poudriere for the average user."_ It certainly appears to have all the requirements to succeed poudriere.


----------



## Datapanic (Jan 26, 2018)

Well, yes, what Sir Dice said: "Use the tools which best suits your situation."

I'm not going to get into wars over OS, package compilers or politics!


----------



## toprank (Jan 26, 2018)

I don't see any political relevance to the topic, nor do I care to engage in any philosophical debate over operating systems or compilers. But think there's something to be gained from discussing the merits of different software applications.


----------



## SirDice (Jan 26, 2018)

marino said:


> For the record, Synth can build ARM/ARM64 packages on x86 and also ARM64 packages natively, so that's not a feature that poudriere has over Synth.


Nice. Did you add that recently? If I recall correctly it wasn't possible with the first versions.


----------



## marino (Jan 26, 2018)

It was always capable of it.  Jonathon Chen added a FAQ entry on how to do it last march, so 10 months ago.


----------



## marino (Jan 26, 2018)

toprank said:


> As far as I can see, this (building packages for different versions) is the only feature poudriere provides that as yet isn't available in synth(1). Nevertheless, as SirDice alludes to below, in my case, it isn't as important a feature as the resource optimisation from using packages for build dependencies.



Again: synth can do that.  it's just a matter of setting up the environment (same as poudriere).



> In this case, I think the developer has suggested that it is an intended replacement for portmaster(8) at least, if not poudriere. Which makes sense given that portmaster(8) doesn't have any feature not found in synth(1). Further, freshports.org states that, "It is intended to replace Portmaster, portupgrade, and poudriere for the average user."



I never believed somebody would take the time to upgrade portmaster for flavors.  I also never believed people would refuse to see how awful building on a system (not using a controlled build environment).  I was wrong on both counts.  IMO portmaster and portupgrade should have both been left to die, but I'm not waging that war anymore.  The opposing army of postmaster loyalists "beat me with experience" if you catch my drift.  Bottom line: use synth or don't.  Doesn't affect me either way.


----------



## toprank (Jan 27, 2018)

I can't see any utility in using postmaster, and certainly not port upgrade, over synth. As it stands at the moment, I'm more familiar with poudriere as I've been using that since 2014 and only became aware of synth; but it looks like a replacement for poudriere too. I can't understand why FreeBSD wouldn't focus its resources on the development of synth as it is a less resource intensive, more user friendly, and yet more feature full program.


----------



## marino (Jan 27, 2018)

well, poudriere is better suited for the purpose it was built for: official ports builder for poudriere.  At least slightly.  By using "real" jails, poudriere has the ability to turn off networking after post-fetch, which catches bad acting ports that try to fetch during the build.  Synth with it's null mounts can't do that.  I've never offered up Synth as a replacement for poudriere (even though synth outperforms poudriere by 33% in terms of build time for long builds).  Basically if somebody is set up with poudriere and knows it, there's not that much a reason to switch.  

Another huge difference is log handling.  With poudriere, if you don't clear out logs regularly, you'll eat the disk up.  I think it's about 20G per run, and poudriere is set by default to save every single long with unique filenames.  Synth just keeps the last log so hard disk space is never an issue.  That decision is fine for the individual user, but for the beefy instructure, they want the poudriere behavior.  And it's got a web interface designed for beefy.  And obviously, poudriere works on all of freebsd's platforms whereas synth just works on the popular ones.  So the right choice for freebsd leaders is poudriere and I never disputed that.


----------



## Datapanic (Jan 27, 2018)

yeah, you need to add something like:

```
# delete old poudriere log files
30      1       *       *       *       root    /usr/bin/find -s /usr/local/poudriere/data/logs/bulk/ -ctime +10d -delete
```
to clean up those poudriere log files.

I think the best feature of synth is that it fetches pre-built packages if the config options are defaults and that definitely speeds up the build time for ports that depend on things like, well devel/llvm50!


----------



## toprank (Jan 27, 2018)

marino said:


> well, poudriere is better suited for the purpose it was built for: official ports builder for poudriere.  At least slightly.  By using "real" jails, poudriere has the ability to turn off networking after post-fetch, which catches bad acting ports that try to fetch during the build.  Synth with it's null mounts can't do that.  I've never offered up Synth as a replacement for poudriere (even though synth outperforms poudriere by 33% in terms of build time for long builds).  Basically if somebody is set up with poudriere and knows it, there's not that much a reason to switch.
> 
> Another huge difference is log handling.  With poudriere, if you don't clear out logs regularly, you'll eat the disk up.  I think it's about 20G per run, and poudriere is set by default to save every single long with unique filenames.  Synth just keeps the last log so hard disk space is never an issue.  That decision is fine for the individual user, but for the beefy instructure, they want the poudriere behavior.  And it's got a web interface designed for beefy.  And obviously, poudriere works on all of freebsd's platforms whereas synth just works on the popular ones.  So the right choice for freebsd leaders is poudriere and I never disputed that.



A very reasonable and rational explanation. Thank you. I'm discussing the merits and operation of these applications from an enduser perspective--not from that of FreeBSD (the collective) in its provision of binary packages to FreeBSD users--as that's where the majority of the use occurs.



Datapanic said:


> yeah, you need to add something like:
> 
> ```
> # delete old poudriere log files
> ...



I think that's the best, too, and almost a necessity. Its utility can't be overstated, and certainly shouldn't be overlooked by developers.


----------



## ShelLuser (Jan 27, 2018)

toprank said:


> In this case, I think the developer has suggested that it is an intended replacement for portmaster(8) at least, if not poudriere. Which makes sense given that portmaster(8) doesn't have any feature not found in synth(1).


Have to disagree. Although not a real feature I really value the fact that I can get ports-mgmt/portmaster installed without having to leave my server building for quite a while. And there's also the thing that Portmaster easily allows itself to be used in scripts to set up bigger (customized) things whereas the others (Poudriere and Synth) are more aimed at providing "all in one" kind of solutions.

I personally prefer the customability so that I can use the building tools in the way which I want to instead of having to comply to whatever scenario it is that they're setting up for you (for example: Poudriere using Jails, to my knowledge that's a mandatory feat).


----------



## xtaz (Jan 27, 2018)

Datapanic said:


> yeah, you need to add something like:
> 
> ```
> # delete old poudriere log files
> ...



There is now a more official way to do this on the latest version....


```
/usr/local/bin/poudriere logclean -y 7
```


----------



## xtaz (Jan 30, 2018)

marino said:


> I've never offered up Synth as a replacement for poudriere (even though synth outperforms poudriere by 33% in terms of build time for long builds).



Out of curiosity I'm wondering if you can elaborate on this. I've used both Synth and Poudriere and can't really see any major difference in build times between them. Is the 33% difference based on the default settings for both? As Synth will do multiple workers and jobs per worker by default whereas Poudriere only does multiple workers with 1 job each. But you can easily configure it to do multiple jobs per worker as well which in my eyes should make them identical. The other reason I guess is the prefetching of binary packages?


----------



## marino (Jan 31, 2018)

no, the difference is in how fast one can build up and take down jails (although poudriere may use persistent jails now).
anecdotes quoted 24-hour bulk runs on poudriere being done in 18 hours in synth.
My own experiences with poudriere on DF supports these numbers.
Not to mention synth forks several orders or magnitude less than than poudriere (which is just a shell script that all it does is fork).  That forking has a huge taxation, especially on FreeBSD (DragonFly is more performant there).

You don't see the times on individual packages.  You need runs of several hours to measure it, especially runs with hundreds or thousands of diverse packages.


----------



## abishai (Jan 31, 2018)

poudriere can manage port options interactively, synth is not, it's just a package building tool.


----------



## marino (Jan 31, 2018)

abishai said:


> poudriere can manage port options interactively, synth is not, it's just a package building tool.


I don't know what you mean by that.  Poudriere was not designed for interactive use.  In fact, all interactive ports have been eliminated.  Anything that would require interaction (e.g. option settings, license settings, knobs) is supposed to handled through make.conf files.

plus poudriere doesn't "manage" ports.  It updates a ports tree in its entirety for the typical user.  I suspect you're playing fast and loose with the definition of "interactive" here.  Finally, as has been demonstrated multiple times, people keep claiming synth can't do something like poudriere only to have to be corrected.   So you might want to be a lot more specific here about what you mean.

I'm going to take a guess and think you mean that poudriere will show option menus if options are not set.  Synth was intentionally designed NOT to do this as it's considered a nuisance.  It's also simple to run an external tool to set all the port options recursively.  So if the "interactive" comment is about that, then yes, it's true, and it was designed intentionally like that.  In most cases, port options aren't even tested beyond their default settings.  It is extremely common to change default options only to find A) the port doesn't build or B) it causes a circular dependency making it impossible to build.  That's due to inadequate testing by the port maintainer and like I said, extremely common.  Even power users only want to customize a few ports -- a few as possible if they don't want to encounter issue.

P.S. Unlike synth, poudriere cannot detect when saved options are obsolete so packages aren't rebuilt.  It also continues to use bad saved options forever (they have to be manually fixed) so packages are misbuilt.  The poudriere approach to options is actually inferior in this common case.


----------



## abishai (Jan 31, 2018)

marino said:


> I suspect you're playing fast and loose with the definition of "interactive" here


I can imagine only one meaning of 'manage port options interactivly' - invoke dialog4ports and ask. Surely, we can invite native english speakers and ask them, if you insist.



marino said:


> In most cases, port options aren't even tested beyond their default settings.


In most cases, ports delegate options directly to cmake recipes or similar stuff. In my practice of localhost administrator, I change a lot of options without any issues. My experience with synth was a real pain, I even returned back to portmaster for some time, the stupidity of running `make options` manually is astonishing.



marino said:


> Synth was intentionally designed NOT to do this as it's considered a nuisance.


Understandable, I'm not using it for that reason. Gladly, we always have an alternative.



marino said:


> Unlike synth, poudriere cannot detect when saved options are obsolete so packages aren't rebuilt


This is obviously wrong, poudriere invokes dialog4ports if options have changed after tree update.


----------



## marino (Feb 1, 2018)

abishai said:


> I can imagine only one meaning of 'manage port options interactivly' - invoke dialog4ports and ask. Surely, we can invite native english speakers and ask them, if you insist.



BY DEFINITION, an interactive port is one that requires input from the user to continue to build.  All such ports were eliminated a couple of years ago.  There is no port that REQUIRES manual input.  The definition of interactive is not in question -- you're using a word that has a specific meaning to apply to options.



> In most cases, ports delegate options directly to cmake recipes or similar stuff. In my practice of localhost administrator, I change a lot of options without any issues. My experience with synth was a real pain, I even returned back to portmaster for some time, the stupidity of running `make options` manually is astonishing.



And if somebody audited those decisions, you might have trouble justifying moving away from defaults on some ports.  There could be a lot of "oh, shiny!" going on there.  And you ignored that fact that you can set all options recursively against a build list with a simple script (or individually per single command), so you're creating a false scenario to fit your narrative.   Basically you're saying you don't want to set the groundwork (a one-time thing).



> Understandable, I'm not using it for that reason. Gladly, we always have an alternative.


And I'm glad too.



> This is obviously wrong, poudriere invokes dialog4ports if options have changed after tree update.


Only if you change the defaults to make it do that.
Unless things have changed, it uses the defaults or last saved options unless you specifically say "bring up the nuisance dialogs".

Are you really challenging me on this?  I've seen many times poudriere fail out because previously saved options were outdated.  And I know its possible for options to change without poudriere detecting the package needs to be rebuilt for that reason alone.  I had commit access to poudriere; I obviously know how it works at the code level.  Can you say the same?


----------



## xtaz (Feb 1, 2018)

marino said:


> no, the difference is in how fast one can build up and take down jails (although poudriere may use persistent jails now).
> anecdotes quoted 24-hour bulk runs on poudriere being done in 18 hours in synth.
> My own experiences with poudriere on DF supports these numbers.
> Not to mention synth forks several orders or magnitude less than than poudriere (which is just a shell script that all it does is fork).  That forking has a huge taxation, especially on FreeBSD (DragonFly is more performant there).
> ...



In my case poudriere creates a reference jail and then clones that reference jail once for each build worker. This takes about 30 seconds because of UFS, would be quicker on ZFS. But once those jails are created it never tears them down again. It reuses the same jails. It only removes them at the very end when it's complete. So maybe this behavior is relatively new and it didn't used to do this then.

I'm just going by the fact on my server I build around 270 packages. Both synth and poudriere take around 6.5 hours to complete this (it's a slow Intel Atom CPU). Both of them are set to use 4 workers and 4 jobs.


----------

