# Synth dependencies rebuild overhead ?



## ASX (Nov 19, 2016)

I'm have observed a a strange behavior for a few times now, and I need to ask about:

just yesterday I completed an update (`synth prepare-system` ; `pkg upgrade -r Synth`), the process built approximatedly 150 pkgs, and installed only very few (4 or 5).

Something similar happened the previous time too. However, once built/installed I have no data to display to check it the whole process is correct or not.

Today, I just updated the port tree (`svnlite upgrade /usr/ports` and before doing anything else I checked:


```
# pkg info | wc
     704    4812   50498
```


```
# ls /var/synth/live_packages/All/ | wc
     844     844   18858
```

The difference (844 - 704) easily account for build dependencies, that's fine.

portupgrade report 1 single port needs to be rebuild:


```
# portupgrade -a -n
...
   + security/trousers (trousers-0.3.13_1 -> trousers-0.3.14)
--->  Packages processed: 1 done, 10 ignored, 0 skipped and 0 failed
```
 (The 10 ignored are out of port tree pkgs)

`synth status` report 113 pkgs need to be rebuilt:


```
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scan of x11-themes/ghostbsd-wallpapers failed (port deleted), it will not be considered.
Scan of x11/xfce-installed-settings failed (port deleted), it will not be considered.
Scan of sysutils/inxi failed (port deleted), it will not be considered.
Scan of net-mgmt/networkmgr failed (port deleted), it will not be considered.
Scan of x11/ghostbsd-slim-theme failed (port deleted), it will not be considered.
Scan of x11-themes/ghostbsd-icons failed (port deleted), it will not be considered.
Scan of x11/ghostbsd-installed-common-settings failed (port deleted), it will not be considered.
Scan of devel/ssft failed (port deleted), it will not be considered.
Scan of x11-themes/ghostbsd-mate-themes failed (port deleted), it will not be considered.
Scan of sysutils/ghostbsd-grub2-settings failed (port deleted), it will not be considered.
Scanning existing packages.
gnutls-3.4.16.txz failed dependency check.
cups-2.2.1.txz failed dependency check.
gtk2-2.24.29_2.txz failed dependency check.
gtk3-3.18.8_3.txz failed dependency check.
xfce4-conf-4.12.1.txz failed dependency check.
libxfce4menu-4.12.1_1.txz failed dependency check.
libwnck-2.30.7.txz failed dependency check.
garcon-0.4.0_1.txz failed dependency check.
libexo-0.10.7.txz failed dependency check.
xfce4-panel-4.12.1.txz failed dependency check.
ghostscript9-agpl-base-9.16_5.txz failed dependency check.
glib-networking-2.46.1_1.txz failed dependency check.
libsoup-2.52.2.txz failed dependency check.
texlive-base-20150521_13.txz failed dependency check.
texlive-texmf-20150523_3.txz failed dependency check.
geoclue-2.3.0.txz failed dependency check.
gconf2-3.2.6_4.txz failed dependency check.
libsoup-gnome-2.52.2.txz failed dependency check.
tex-formats-20150521_2.txz failed dependency check.
rest-0.7.93.txz failed dependency check.
webkit2-gtk3-2.8.5_6.txz failed dependency check.
gcr-3.18.0.txz failed dependency check.
libglade2-2.6.4_8.txz failed dependency check.
qt5-printsupport-5.6.2.txz failed dependency check.
tex-xmltex-1.9_2.txz failed dependency check.
policykit-gnome-0.9.2_7.txz failed dependency check.
gnome-online-accounts-3.18.4_1.txz failed dependency check.
uhttpmock-0.5.0.txz failed dependency check.
libcanberra-0.30_3.txz failed dependency check.
libgdata-0.17.4.txz failed dependency check.
tex-jadetex-3.13_3.txz failed dependency check.
qt5-webkit-5.6.2.txz failed dependency check.
gnome-mount-0.8_12.txz failed dependency check.
ffmpeg-2.8.8_5,1.txz failed dependency check.
gvfs-1.26.3_2.txz failed dependency check.
qt5-assistant-5.6.2.txz failed dependency check.
tex-dvipsk-5.995_1.txz failed dependency check.
py27-gtk2-2.24.0_4.txz failed dependency check.
docbook-utils-0.6.14_13.txz failed dependency check.
Thunar-1.6.10_2.txz failed dependency check.
doxygen-1.8.12,2.txz failed dependency check.
qt5-designer-5.6.2.txz failed dependency check.
webkit-gtk2-2.4.11_4.txz failed dependency check.
claws-mail-3.14.1.txz failed dependency check.
gtkmm24-2.24.4_2.txz failed dependency check.
gtksourceview2-2.10.5_4.txz failed dependency check.
unique-1.1.6_6.txz failed dependency check.
vte3-0.42.4_1.txz failed dependency check.
gnupg-2.1.15.txz failed dependency check.
orage-4.12.1_1.txz failed dependency check.
xfce4-notifyd-0.3.4.txz failed dependency check.
qt5-linguist-5.6.2.txz failed dependency check.
cups-pk-helper-0.2.5_1.txz failed dependency check.
libspectre-0.2.8.txz failed dependency check.
py27-pycups-1.9.73_1.txz failed dependency check.
firefox-50.0_2,1.txz failed dependency check.
thunderbird-45.4.0_2.txz failed dependency check.
xfce4-settings-4.12.1.txz failed dependency check.
samba36-3.6.25_3.txz failed dependency check.
libxfce4gui-4.10.0_5.txz failed dependency check.
xfce4-appfinder-4.12.0.txz failed dependency check.
xfce4-desktop-4.12.3.txz failed dependency check.
xfce4-session-4.12.1_3.txz failed dependency check.
xfce4-wm-4.12.3.txz failed dependency check.
gtk-xfce-engine-3.2.0.txz failed dependency check.
mousepad-0.4.0_2.txz failed dependency check.
pavucontrol-3.0.txz failed dependency check.
pinentry-gnome3-0.9.7.txz failed dependency check.
keybinder-0.3.1.txz failed dependency check.
keybinder-gtk3-0.3.1.txz failed dependency check.
xfce4-terminal-0.8.1.txz failed dependency check.
xfce4-volumed-pulse-0.2.2.txz failed dependency check.
xfce4-xkb-plugin-0.7.1.txz failed dependency check.
virtualbox-ose-5.1.8.txz failed dependency check.
cups-filters-1.11.4.txz failed dependency check.
cups-smb-backend-1.0_7.txz failed dependency check.
ghostscript9-agpl-x11-9.16_2.txz failed dependency check.
system-config-printer-1.4.7_3.txz failed dependency check.
firefox-i18n-50.0.txz failed dependency check.
py27-webkitgtk-1.1.8_6.txz failed dependency check.
xfce4-smartbookmark-plugin-0.4.6_1.txz failed dependency check.
claws-mail-fancy-3.14.1.txz failed dependency check.
claws-mail-pdf_viewer-3.14.1.txz failed dependency check.
thunderbird-i18n-45.4.0.txz failed dependency check.
xfburn-0.5.4_4.txz failed dependency check.
xfce4-battery-plugin-1.0.5_4.txz failed dependency check.
xfce4-genmon-plugin-3.4.0_5.txz failed dependency check.
xfce4-power-manager-1.6.0.txz failed dependency check.
xfce4-wavelan-plugin-0.5.12_1.txz failed dependency check.
xfce-4.12_1.txz failed dependency check.
mplayer-1.3.0.20160912_1.txz failed dependency check.
evince-lite-3.18.2_1.txz failed dependency check.
gpicview-0.2.5.txz failed dependency check.
ristretto-0.8.1.txz failed dependency check.
shotwell-0.24.2.txz failed dependency check.
hexchat-2.12.3.txz failed dependency check.
xarchiver-0.5.4.7.txz failed dependency check.
xfce4-dict-plugin-0.7.2.txz failed dependency check.
libreoffice-5.2.3_2.txz failed dependency check.
vim-8.0.0082.txz failed dependency check.
xfce4-datetime-plugin-0.6.2_4.txz failed dependency check.
xfce4-timer-plugin-1.6.0_1.txz failed dependency check.
xfce4-mixer-4.11.0_3.txz failed dependency check.
xfce4-pulseaudio-plugin-0.2.4.txz failed dependency check.
gnome-keyring-3.18.3_1.txz failed dependency check.
nvidia-settings-355.11_3.txz failed dependency check.
trayer-1.1.6.txz failed dependency check.
xfce4-quicklauncher-plugin-1.9.4_17.txz failed dependency check.
xfce4-screenshooter-plugin-1.8.2_2.txz failed dependency check.
xfce4-taskmanager-1.1.0_1.txz failed dependency check.
xfce4-verve-plugin-1.1.0_1.txz failed dependency check.
zenity-3.18.0.txz failed dependency check.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  U => security/trousers (0.3.13_1 => 0.3.14)
  R => security/gnutls
  R => print/cups
  R => x11-toolkits/gtk20
  R => x11-toolkits/gtk30
  R => x11/xfce4-conf
  R => x11/libxfce4menu
  R => x11-toolkits/libwnck
  R => sysutils/garcon
  R => x11/libexo
  R => x11-wm/xfce4-panel
  R => print/ghostscript9-agpl-base
  R => net/glib-networking
  R => devel/libsoup
  R => print/texlive-base
  R => print/texlive-texmf
  R => net/geoclue
  R => devel/gconf2
  R => devel/libsoup-gnome
  R => print/tex-formats
  R => devel/librest
  R => www/webkit2-gtk3
  R => security/gcr
  R => devel/libglade2
  R => print/qt5-printsupport
  R => print/tex-xmltex
  R => sysutils/policykit-gnome
  R => net/gnome-online-accounts
  R => net/uhttpmock
  R => audio/libcanberra
  R => devel/libgdata
  R => print/tex-jadetex
  R => www/webkit-qt5
  R => sysutils/gnome-mount
  R => multimedia/ffmpeg
  R => devel/gvfs
  R => devel/qt5-assistant
  R => print/tex-dvipsk
  R => x11-toolkits/py-gtk2
  R => textproc/docbook-utils
  R => x11-fm/thunar
  R => devel/doxygen
  R => devel/qt5-designer
  R => www/webkit-gtk2
  R => mail/claws-mail
  R => x11-toolkits/gtkmm24
  R => x11-toolkits/gtksourceview2
  R => x11-toolkits/unique
  R => x11-toolkits/vte3
  R => security/gnupg
  R => deskutils/orage
  R => deskutils/xfce4-notifyd
  R => devel/qt5-linguist
  R => print/cups-pk-helper
  R => print/libspectre
  R => print/py-pycups
  R => www/firefox
  R => mail/thunderbird
  R => sysutils/xfce4-settings
  R => net/samba36
  R => x11-toolkits/libxfce4gui
  R => misc/xfce4-appfinder
  R => x11-wm/xfce4-desktop
  R => x11-wm/xfce4-session
  R => x11-wm/xfce4-wm
  R => x11-themes/gtk-xfce-engine
  R => editors/mousepad
  R => audio/pavucontrol
  R => security/pinentry-gnome3
  R => x11/keybinder
  R => x11/keybinder-gtk3
  R => x11/xfce4-terminal
  R => deskutils/xfce4-volumed-pulse
  R => deskutils/xfce4-xkb-plugin
  R => emulators/virtualbox-ose
  R => print/cups-filters
  R => print/cups-smb-backend
  R => print/ghostscript9-agpl-x11
  R => print/system-config-printer
  R => www/firefox-i18n
  R => www/py-webkitgtk
  R => www/xfce4-smartbookmark-plugin
  R => mail/claws-mail-fancy
  R => mail/claws-mail-pdf_viewer
  R => mail/thunderbird-i18n
  R => sysutils/xfburn
  R => sysutils/xfce4-battery-plugin
  R => sysutils/xfce4-genmon-plugin
  R => sysutils/xfce4-power-manager
  R => sysutils/xfce4-wavelan-plugin
  R => x11-wm/xfce4
  R => multimedia/mplayer
  R => graphics/evince-lite
  R => graphics/gpicview
  R => graphics/ristretto
  R => graphics/shotwell
  R => irc/hexchat
  R => archivers/xarchiver
  R => textproc/xfce4-dict-plugin
  R => editors/libreoffice
  R => editors/vim
  R => x11-clocks/xfce4-datetime-plugin
  R => x11-clocks/xfce4-timer-plugin
  R => audio/xfce4-mixer
  R => audio/xfce4-pulseaudio-plugin
  R => security/gnome-keyring
  R => x11/nvidia-settings
  R => x11/trayer
  R => x11/xfce4-quicklauncher-plugin
  R => x11/xfce4-screenshooter-plugin
  R => x11/xfce4-taskmanager
  R => x11/xfce4-verve-plugin
  R => x11/zenity
Total packages that would be built: 113
The complete build list can also be found at:
/tmp/synth_status_results.txt
[/U]
```


The question is: what's is triggering all those "failed dependency check" on an otherwise updated system ? Is something wrong in my setup or am I doing something wrong ?

Must be noted that I have no issue, other than some (apparently) wasted time rebuilding those packages.

I have not performed further steps right now, if needed I can provide info about the current state of the things.


----------



## PacketMan (Nov 19, 2016)

I get that often on my GNOME3 machine. In fact it was all up to date a couple days ago. But I decided to run an upgrade again to install a new port, and blam a ton of fail dependency checks. Through other discussion I have been told that is normal expected behavior.  Now I am trying to figure out with synth aborting on www/webkit2-gtk3. (See other topic I posted earlier today.) Fun times!


----------



## marino (Nov 19, 2016)

it is normal behavior.
I've been thinking about an option that would use multiple passes to lessen this effect, but as you might imagine, it increases the complexity greatly.


----------



## ASX (Nov 19, 2016)

marino@ said:


> it is normal behavior.
> I've been thinking about an option that would use multiple passes to lessen this effect, but as you might imagine, it increases the complexity greatly.


OK, I'm fine with that, just needed to be sure I was not missing something.


----------



## marino (Nov 19, 2016)

pkg(8) just checks for version changes.  poudriere and synth aggressively rebuild (or most conservatively) even when there was no version change because the dependency chain changed somewhere.


----------



## kpa (Nov 19, 2016)

There have been earlier threads on the same issue but I couldn't find them now.

Basically what happens is that it's impossible to know in advance if a rebuild of a port would trigger a need to rebuild one or more of the dependent ports of the changed port. That knowledge if it was available would save your from rebuilding all of the dependents (and their dependents recursively) if it was available but we don't have anything resembling a universal ABI compatibility checker that could take two ports and be able to decide if the two will operate together correctly if installed at the same time. If you try to be cheap and not rebuild the dependents by force you're going to come across a dependent port that should have been rebuilt and the only way you can find that out is running the software and find out that something went wrong. This is not acceptable on a build cluster that is supposed to build ports to packages without manual intervention. This is how the early versions of ports-mgmt/poudriere behaved and it produced broken builds where the cause of the breakage was really hard to figure out and poudriere had to be changed to rebuild all of the dependents recursively like it does now.

This is not a FreeBSD specific problem. Any system that uses dynamic linking in the same way as FreeBSD does is going face the same difficulty in determining if a change in one binary or dynamic link library is going to trigger the same rebuild of the dependents.


----------



## ASX (Nov 19, 2016)

kpa,
thanks for the additional explanation.
If that is the case I would expect portupgrade producing the same behavior, which is not what I see.

Now, I'm possibly more confused, there is a post where marino@ state it is possible to lessen the build but complex/difficult, there is yours where you state it is near to "impossible", and there is the FreeBSD handbook which happily suggest to rely on portmaster or portupgrade.
handbook: https://www.freebsd.org/doc/handbook/ports-using.html

There is still a portion of your explanation that is difficult to accept:


kpa said:


> If you try to be cheap and not rebuild the dependents by force you're going to come across a dependent port that should have been rebuilt and the only way you can find that out is running the software and find out that something went wrong.


... (unless you meant to write "building the software" instead of "running the software")

The fact is that, even if those dependent packages are rebuilt, they are not going to be installed and I have not noticed issues. (however, I'm using ports/synth from just few weeks ... may be just I'm lucky ?)

If I have to trust your assertion above, I must conclude that using (in my case)  `pkg upgrade -r Synth` is not enough to have all the required pkgs updated, instead I should force some reinstall of *all the rebuilt packages*.

However  I look at the whole, something is not right, one or more of these:

- misleading handbook: because it suggest to use portupgrade / portmaster knowing their job is not enough.
- build overhead, because of difficult in tracking dependency relations
- missing reinstallation, because pkgs are built but later not installed
- me, not forcing the pkgs reinstall, just upgrading from local rebuilt repository.

which one(s) ?


----------



## Remington (Nov 19, 2016)

You might want to look into adding devel/ccache to cut down the time on rebuilding packages that hasn't been changed.


----------



## kpa (Nov 19, 2016)

I absolutely meant "running the software". ABI compatibility issues may not manifest at compile time at all but instead they will show up at run time as crashes and other problems *). Build time problems are much nicer because you have at least a clue based on the error messages where to start looking at. In case of runtime crashes you have to first load the program into a debugger and then start looking where the problem might be.

*) In fact FreeBSD has had some problems maintaining it's promised stable ABI in the base system where a unintended change has caused base system programs start to crash even if they compile fine.


----------



## ASX (Nov 19, 2016)

Remington thanks, I'm already using ccache.
kpa, fine ... then should I force the re-installation of those rebuilt packages ?


----------



## marino (Nov 19, 2016)

ASX said:


> kpa,
> thanks for the additional explanation.
> If that is the case I would expect portupgrade producing the same behavior, which is not what I see.



Why would you expect that?  It's not logical to do so.

portupgrade uses the same garbage foundation as portmaster. 
You should *never* compare portupgrade to poudriere or synth, they are apples to oranges.  If you must compare it to something, limit those comparisons to portmaster.


----------



## kpa (Nov 19, 2016)

ASX said:


> kpa, fine ... then should I force the re-installation of those rebuilt packages ?



No, they are still the same packages for pkg(8) because in its view the packages haven't changed even if they have been recompiled. You can trust the there is no need to reinstall anything that pkg doesn't see as an update because this forced recompilation of the dependent ports has already taken care of figuring out if something had to be rebuilt after all. This of course assuming that you're building your ports using Poudriere or Synth, the other tools are not to be trusted in this manner at all.


----------



## ASX (Nov 19, 2016)

marino@ said:


> Why would you expect that? It's not logical to do so.
> 
> portupgrade uses the same *garbage* foundation as portmaster.


That's what is officially suggested in the handbook to upgrade / install from ports.
That both are based on garbage is something I don't know, until someone like you tell me that.



kpa said:


> No, they are still the same packages for pkg(8) because in its view the packages haven't changed even if they have been recompiled. You can trust the there is no need to reinstall anything that pkg doesn't see as an update


Then I don't follow your reasoning about crashes happening after some component update. I can only think those packages are rebuilt for nothing, because in the end aren't going to be installed, and accept that like normal behavior. Like said above I'm fine with that.


----------



## kpa (Nov 19, 2016)

ASX said:


> That's what is officially suggested in the handbook to upgrade / install from ports.
> That both are based on garbage is something I don't know, until someone like you tell me that.
> 
> 
> Then I don't follow your reasoning about crashes happening after some component update. I can only think those packages are rebuilt for nothing, because in the end aren't going to be installed, and accept that like normal behavior. Like said above I'm fine with that.



You have read what I wrote as what would happen if the package builders like Poudriere and Synth were not doing the "excessive" rebuild. They are not rebuilt for nothing, there is always a possiblity that one or more of the rebuilds trigger a change in the shared library dependencies and that's something that pkg(8) will notice and you will be then presented with a changed package for that dependent port.


----------



## ASX (Nov 19, 2016)

kpa, I will make it a little more practical here:
`portupgrade -a-n` show that 1 pkg need to be rebuilt: security/trousers
`synth status` report the same need to be rebuilt, from my initial report :


ASX said:


> *U* => security/trousers (0.3.13_1 => 0.3.14)


in additions synth status report some more 100 ports to be rebuild all marked with an *R* in example:


ASX said:


> *R* => editors/libreoffice



As far as I can see, editors/libreoffice depend on security/trousers, not the inverse.
ports-mgmt/synth rebuild editors/libreoffice ... libreoffice version is not changed anyway, is renewed in synth local repository and later is *not* going to be installed.

How is libreoffice is not going to be rebuilt for nothing ? How would `pkg` notice the package is newer and need to be reinstalled if the libreoffice version has not been changed ?


----------



## marino (Nov 19, 2016)

ASX said:


> That's what is officially suggested in the handbook to upgrade / install from ports.



It's not.  These are no longer recommended.  They are mentioned as "tools that exist".  They used to be recommended but the words about recommendation should have been removed.  If you can point to explicit "recommendations" of either portmaster or portupgrade, please paste the reference here so we can get those adjusted.


----------



## ASX (Nov 19, 2016)

marino@ said:


> It's not.  These are no longer recommended.  They are mentioned as "tools that exist".  They used to be recommended but the words about recommendation should have been removed.  If you can point to explicit "recommendations" of either portmaster or portupgrade, please paste the reference here so we can get those adjusted.



I started by using pkgs only, and later went to use ports, thus I went to "upgrade ports":

https://www.freebsd.org/doc/handbook/ports-using.html


> *4.5.3. Upgrading Ports*
> *..."*
> To perform the actual upgrade, use either Portmaster or Portupgrade.


It is not "recommended" ... but that is what I read ... I have also written "officially suggested", not "recommended".


----------



## marino (Nov 19, 2016)

yes, that line should be changed or removed.


----------



## ASX (Nov 19, 2016)

well, I guess that the attribute "uses the same garbage foundation" also qualify to remove both from port tree, even if their sinth or poudriere build/test log show success.


----------



## marino (Nov 19, 2016)

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214679


----------



## marino (Nov 19, 2016)

ASX said:


> well, I guess that the attribute "uses the same garbage foundation" also qualify to remove both from port tree, even if their sinth or poudriere build/test log show success.



I tried to get portmaster marked as deprecated with no expiration date because it was unmaintained and people went ape-****.  Finally somebody put their name on it, but their next commit to it will be their first.  It's not maintained.  Even if it did perfectly what it was designed it do, it's defective by design.  It's like how Linux thinks about svn.  (There's no way to do cvs "right").  There's no way to make portmaster right.


----------



## wblock@ (Nov 20, 2016)

ASX said:


> well, I guess that the attribute "uses the same garbage foundation" also qualify to remove both from port tree, even if their sinth or poudriere build/test log show success.


Opinions vary.

ports-mgmt/portmaster and ports-mgmt/portupgrade still work.  ports-mgmt/portmaster is nice because it does not need Ruby and the defaults are somewhat more in line with preferred procedure.  I think the separate port database requirement of ports-mgmt/portupgrade has been fixed with pkg, but I used that for years and see no reason to go back to it.


----------



## ASX (Nov 20, 2016)

wblock@ 
there was an implicit cross-thread reference in my previous post, addressing new ports requirements, where I had a little discussion about requirements not being listed in handbook/documentation: It was (partially) supposed to be ironic.
https://forums.freebsd.org/threads/58578/#post-335068



wblock@ said:


> ports-mgmt/portmaster is nice because it does not need Ruby


Please correct me if I'm wrong, I tried portmaster very briefly and *if I remember correctly* it will install the requested/upgraded port* along with the build dependencies.* (in addition to run-dependencies).

Do people using portmaster really need to worry about Ruby considering the context ?

Beside that I'm really interested in the next reply from kpa, his comments about possible crashes. it is a key factor for me to take a final position.


----------



## marino (Nov 20, 2016)

> ports-mgmt/portmaster and ports-mgmt/portupgrade still work


If you play fast and loose with the definition of "work" ...


Among people that develop ports, opinions do not really vary.


----------



## wblock@ (Nov 20, 2016)

ASX said:


> Do people using portmaster really need to worry about Ruby considering the context?


portmaster does not need or use Ruby, so that does not affect its use.  portupgrade does use Ruby, so upgrades can be a problem.

I'm not sure what you mean about the other thread.

There is a section on Poudriere in the Handbook.  I did not write it, but spent several hours working with the author on it and committed it.


----------



## ASX (Nov 20, 2016)

wblock@ said:


> I'm not sure what you mean about the other thread.


When marino@ was insisting that attaching a build log will help to get a new port committed, I pointed out that there could be reason to not be so strict, portmaster and portupgrade, considering his opinion "based on garbage" they qualify for removal from port tree, even when the build logs would be positive.

Of course, I know there is a section about poudriere, I hope there will be one about Synth too.

I can clearly see that the initial design of portmaster and portupgrade are meant to install/upgrade ports on a single system, while poudriere and synth are designed to build a local repository.

Although there is a difference, I'm understanding that this is a theoretical difference, and that in fact a proper update from port tree require to build/install some port "subtree". (subtree based on dependency relations, not on filesystem hierarchy).

Now, both kpa and marino@ in different ways are suggesting that portmaster and portupgrade are unable to properly upgrade the "subtree", while synth and poudriere can manage that properly, and in fact kpa highlighted that poudriere has been improved to rebuild all the dependent packages, something that neighter portupgrade nor portmaster do on their own.

I have yet to understand the technical reasons in the deep details, but I trust their opinion until proven wrong.

If what they are saying is correct (there is the need to rebuild a whole subtree), I can not see how portupgrade or portmaster can be trusted to do the job.


----------



## xtaz (Nov 20, 2016)

Configuring synth to use ccache and to fetch prebuilt packages whilst making sure that /var/db/ports only has options configured for ports that you absolutely want to customise goes a long way to making synth work pretty quickly for you. I've been using it for 6 months or so now and I can totally see why the old portmaster way of doing things is wrong.

It's a shame that the official handbook doesn't just mention synth or poudriere and removes all mention of portmaster.


----------



## ASX (Nov 20, 2016)

xtaz said:


> It's a shame that the official handbook doesn't just mention synth or poudriere and removes all mention of portmaster.



Actually pudriere has a whole chapter in the handbook: https://www.freebsd.org/doc/handbook/ports-poudriere.html

Synth hasn't one, supposedly because is younger and because some people jugde "9 months" like being a *tiny* amount of time! https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206922#c17


----------



## Remington (Nov 20, 2016)

xtaz said:


> Configuring synth to use ccache and to fetch prebuilt packages whilst making sure that /var/db/ports only has options configured for ports that you absolutely want to customise goes a long way to making synth work pretty quickly for you. I've been using it for 6 months or so now and I can totally see why the old portmaster way of doing things is wrong.



It's better to have your own file equivalent to /etc/make.conf with all options listed in there which can be used by poudriere.  I don't use synth but I'm sure it does have one as well.



> Synth hasn't one, supposedly because is younger and because some people jugde "9 months" like being a *tiny* amount of time!



Synth is new with many features and the developer, Marino@, is very active on this forum which is good.  The only main difference is that synth build its packages based on the host system in chroot.  Poudriere build packages in jails based on different FreeBSD releases apart from its host system.  So poudriere is better if you have multiple servers with different FreeBSD releases.

Anyway I stopped using portmaster or portupgrade since pkg, poudriere and synth are beginning to replace it as they are much more reliable without littering the system with orphaned dependencies or breakages.


----------



## wblock@ (Nov 20, 2016)

ASX said:


> Synth hasn't one, supposedly because is younger and because some people jugde "9 months" like being a *tiny* amount of time!


To my knowledge, nobody has submitted such a section.


----------



## ASX (Nov 20, 2016)

wblock@ said:


> To my knowledge, nobody has submitted such a section.


My apologies, clearly I'm failing at trying to be ironic. 

The 9 months approximately are the time that synth is in port tree ... as above, is a tiny amount of time, clearly insufficient to mention about ports-mgmt/synth existence.

"ports-mgmt/synth is an alternative to ports-mgmt/poudriere build system, right now limited to i386/amd64 arch targets."

That's all is required for a start. In my view, of course, the man page already provide lot of additional info.


----------



## marino (Nov 20, 2016)

Remington said:


> The only main difference is that synth build its packages based on the host system in chroot.  Poudriere build packages in jails based on different FreeBSD releases apart from its host system.  So poudriere is better if you have multiple servers with different FreeBSD releases.



I need to emphasize again that Synth can easily support multiple FreeBSD releases.  While there are specific advantages that poudriere has over synth, this feature is definitely not one of them.  One should not use that reason as a differentiator.


----------



## ANOKNUSA (Nov 20, 2016)

ASX said:


> The 9 months approximately are the time that synth is in port tree ... as above, is a tiny amount of time, clearly insufficient to mention about ports-mgmt/synth existence.



Synth is an excellent tool, and marino@ has been diligent about addressing potential problems here in the open, while helping people learn how it works and how to use it to their advantage. That said, no, 9 months isn't necessarily enough time for a third-party party tool to be given "semi-official" status. I don't really see a problem with mentioning it in the _Handbook_, since it's been the official build tool for DragonFly for some time. But Poudriere is the official build system for the FreeBSD project, so it should be given more weight.

From a documentation standpoint, too, the entire section on ports and packages would need to be modified in such a way that newer users could easily follow the logic from installing and removing single ports or packages; to manually upgrading ports and packages; to understanding how these two tools (Poudriere and Synth) work, and what differentiates them; to using these two tools (Poudriere and Synth) to manage large numbers of ports/packages. In short, updating the _Handbook_ with the intention of promoting these tools as the primary means of managing ports would involve more than simply adding an "official seal of approval" to the Poudriere section and tacking on a page about Synth---the whole chapter would need a partial rewrite covering all those things and stringing them together.

I've been meaning to get seriously involved with DocProj for a while, and would be willing to work with marino@ and wblock@ to touch up and put together the English sections, but then next couple weeks are somewhat full and I still need to familiarize myself with the mark-up the project uses.  Sorry for the cop-out.


----------



## Remington (Nov 20, 2016)

marino@ said:


> I need to emphasize again that Synth can easily support multiple FreeBSD releases.  While there are specific advantages that poudriere has over synth, this feature is definitely not one of them.  One should not use that reason as a differentiator.



My apologies for not being aware of recent changes to synth, so what are the major differences between synth and poudriere?


----------



## ASX (Nov 20, 2016)

ANOKNUSA said:


> That said, no, 9 months isn't necessarily enough time for a third-party party tool to be given "semi-official" status



Thanks for your (present of future) contribution, however let me disagree about that: 9 months are a large amount of time in my view, especially considering the context where *it seems* that the use of poudriere or synth can solve some issues left unattended from portmaster and/or portupgrade. And that is of primary importance at my eyes.


----------



## marino (Nov 20, 2016)

Remington said:


> My apologies for not being aware of recent changes to synth, so what are the major differences between synth and poudriere?


This isn't a new feature, it's been there from the beginning (enabled by the profiles).

There aren't many things that poudriere can do that synth can't.  What comes to mind:

platform agnostic.  poudriere is set of shell scripts so it has no dependencies and works everwhere (although it may not be practical to run everywhere).  In comparison, synth is limited to i386 and amd64 right now.
"cross compiling".  While not true cross compiling, poudriere can support building ARMv6 packages on amd64.  Synth can't do it although I don't know how much is required to support.
one command to install different arch/release jails (Synth can't do this for a user but can leverage it, even poudriere jails)
Synth can do things poudriere can't.  Off the top of my head:

It seems to be 25% faster on the same hardware
It has the use-prebuilt-binaries feature which is pretty popular
the "test mode" is more useful than "poudriere testport" although the latter has more options but it's not a mainstream feature
ncurses display is pretty
web display is pretty

Another design difference is that synth overwrites logs while poudriere generates new ones indefinitely.  This is intentional because full runs consume 1.5 - 2.5 Gb and people can actually run out of diskspace if they aren't cleaned periodically.  To save logs in synth, an external command would have to snapshot them or something.

If I come up with the "aggressive multipass" build then synth would have a clear performance advantage where it would rebuild 5 packages when the other rebuilds 150 (e.g.)


----------



## marino (Nov 20, 2016)

ANOKNUSA said:


> I've been meaning to get seriously involved with DocProj for a while, and would be willing to work with marino@ and wblock@ to touch up and put together the English sections, but then next couple weeks are somewhat full and I still need to familiarize myself with the mark-up the project uses.  Sorry for the cop-out.



It's been on the "to-do" list to create a full handbook for Synth and publish it on github pages.  At that point, the freebsd handbook would mostly link it other than most basic of commands.  I haven't been pushing freebsd docs because A) there's inertia there (possibly intentional) and B) I need to do my part first.


----------

