# Using synth as a builder for many machines



## PacketMan (Nov 25, 2016)

Well, for now, just one machine running x11/gnome3 and a few other 'apps'.  I still don't really understand how it all comes together so this is what I did.

On my builder machine:

```
Synth configuration profile: GNOME3
===============================================================================
   [A] Ports directory            /usr/ports
   (B) Packages directory         /var/synth/gnome3_packages
   [C] Distfiles directory        /usr/ports/distfiles
   [D] Port options directory     /var/db/ports_synth_gnome3
   [E] Build logs directory       /var/log/synth_gnome3
   [F] Build base directory       /usr/obj/synth_gnome3
   [G] System root directory      /
   [H] Compiler cache directory   disabled
   (I) Num. concurrent builders   4
   [J] Max. jobs per builder      2
   [K] Use tmpfs for work area    true
   [L] Use tmpfs for /usr/local   true
   [M] Display using ncurses      true
   [N] Fetch prebuilt packages    true
```

I then used a syncing program to transfer files between the two machines. So:
Builder machine ----------------------------------- Old slow poke machine
/var/db/ports_synth_gnome3 = /var/db/ports
/var/synth/gnome3_packages = var/synth/live_packages

Only reason I used var/synth/live_packages because its existing on the old machine and I haven't removed synth from it yet.

On the builder machine I basically fed it a "just-build" lis which are the main ones, and a few leaf ports:

```
x11/xorg
x11/gdm
x11/gnome3
ftp/filezilla
games/minecraft-client
games/warsow
games/wesnoth
games/xonotic
www/firefox
x11/nvidia-driver-340
x11/nvidia-settings
```

Synth fetched and built a very long list of ports. Then after completion I did `pkg upgrade` on the old machine.  I got two unexpected results: (1) pkg used some of the packages made by synth, and it also fetched a fair few packages from FreeBSD. Which I thought hmmm okay.  After it was all done I decided to run `synth status` on the old machine just as a check to see. To my surprise it said it would download 145 packages and build 14 packages.

Before launching the build job on the builder machine I did a `portsnap fetch update` on both machines at the exact same time, so both machines have the same ports tree. And the ports options directory are in sync.

So my question is why? Why would synth on the old machine still want to do a whole bunch of work right after the builder machine finished up? Where is my mistake? What other key source of info is missing so that the true state of the old machine is pulled into synth on the builder machine?


----------



## marino (Nov 25, 2016)

Chances are you have CONSERVATIVE_UPGRADE set to yes on pkg.  (see `pkg -vv | grep CONS` to verify)
To exclude FreeBSD, you can always call pkg(8) directly: `pkg update -r Synth`
Or even delete all package and reinstall only from Synth repo to make the setting of CONSERVATIVE_UPGRADE moot.


----------



## PacketMan (Nov 25, 2016)

I really didn't think you would reply. Thank you. You're a good man.  I'll give that a try and fart around some more. I see now too I missed some 'root' ports.


----------



## PacketMan (Nov 26, 2016)

A `synth status` as a sanity check still spews a lot of fetching and rebuilding, immediately after a `pkg update -r Synth`, and a `pkg upgrade -r Synth`.  Tired, will pick at this again later. Long day; up at 5, now midnight.

`#synth status`

```
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scan of devel/gmake-lite failed (port deleted), it will not be considered.
Scan of textproc/clucene-qt5 failed (port deleted), it will not be considered.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
Scanning existing packages.   
These are the packages that would be fetched:
  => linux_base-c6-6.8_6.txz (emulators/linux_base-c6)
  => qt5-qmake-5.6.2.txz (devel/qmake5)
  => qt5-buildtools-5.6.2.txz (devel/qt5-buildtools)
  => qt5-core-5.6.2_1.txz (devel/qt5-core)
  => qt5-network-5.6.2.txz (net/qt5-network)
  => qt5-dbus-5.6.2.txz (devel/dbus-qt5)
  => xcb-util-image-0.4.0_1.txz (x11/xcb-util-image)
  => xcb-util-keysyms-0.4.0_1.txz (x11/xcb-util-keysyms)
  => xcb-util-wm-0.4.1_3.txz (x11/xcb-util-wm)
  => qt5-gui-5.6.2_1.txz (x11-toolkits/qt5-gui)
  => linux-c6-expat-2.0.1_2.txz (textproc/linux-c6-expat)
  => linux-c6-fontconfig-2.8.0_1.txz (x11-fonts/linux-c6-fontconfig)
  => qt5-widgets-5.6.2.txz (x11-toolkits/qt5-widgets)
  => linux-c6-xorg-libs-7.4_6.txz (x11/linux-c6-xorg-libs)
  => qt5-sql-5.6.2.txz (databases/qt5-sql)
  => qt5-testlib-5.6.2.txz (devel/qt5-testlib)
  => linux-c6-jpeg-1.2.1_1.txz (graphics/linux-c6-jpeg)
  => linux-c6-png-1.2.49_3.txz (graphics/linux-c6-png)
  => qt5-qml-5.6.2.txz (lang/qt5-qml)
  => qt5-xmlpatterns-5.6.2.txz (textproc/qt5-xmlpatterns)
  => snappy-1.1.3.txz (archivers/snappy)
  => linux-c6-pixman-0.32.8.txz (x11/linux-c6-pixman)
  => qt5-quick-5.6.2.txz (x11-toolkits/qt5-quick)
  => qt5-opengl-5.6.2.txz (graphics/qt5-opengl)
  => linux-c6-libthai-0.1.12.txz (devel/linux-c6-libthai)
  => linux-c6-nspr-4.11.0.txz (devel/linux-c6-nspr)
  => tradcpp-0.5.2.txz (devel/tradcpp)
  => qt5-printsupport-5.6.2.txz (print/qt5-printsupport)
  => linux-c6-cairo-1.8.8_6.txz (graphics/linux-c6-cairo)
  => linux-c6-jasper-1.900.1_1.txz (graphics/linux-c6-jasper)
  => linux-c6-tiff-3.9.4_2.txz (graphics/linux-c6-tiff)
  => leveldb-1.19.txz (databases/leveldb)
  => linux-c6-sqlite-3.6.20_3.txz (databases/linux-c6-sqlite3)
  => qt5-sqldrivers-sqlite3-5.6.2.txz (databases/qt5-sqldrivers-sqlite3)
  => gccmakedep-1.0.3.txz (devel/gccmakedep)
  => xorg-cf-files-1.0.6.txz (x11/xorg-cf-files)
  => linux-c6-atk-1.30.0.txz (accessibility/linux-c6-atk)
  => linux-c6-pango-1.28.1_6.txz (x11-toolkits/linux-c6-pango)
  => linux-c6-cyrus-sasl2-2.1.23_3.txz (security/linux-c6-cyrus-sasl2)
  => linux-c6-nss-3.21.3.txz (security/linux-c6-nss)
  => linux-c6-openssl-1.0.1e_11.txz (security/linux-c6-openssl)
  => linux-c6-gdk-pixbuf2-2.24.1_3.txz (graphics/linux-c6-gdk-pixbuf2)
  => lua51-5.1.5_9.txz (lang/lua51)
  => tcl86-8.6.6_1.txz (lang/tcl86)
  => qt5-webkit-5.6.2_1.txz (www/webkit-qt5)
  => imake-1.0.7,1.txz (devel/imake)
  => linux-c6-libelf-0.164.txz (devel/linux-c6-libelf)
  => linux-c6-libpciaccess-0.13.4_1.txz (devel/linux-c6-libpciaccess)
  => swig-2.0.12_1.txz (devel/swig20)
  => p5-Unicode-String-2.10.txz (converters/p5-Unicode-String)
  => linux-c6-gtk2-2.24.23_5.txz (x11-toolkits/linux-c6-gtk2)
  => pangox-compat-0.0.2_1.txz (x11-toolkits/pangox-compat)
  => qt5-uiplugin-5.6.2.txz (x11-toolkits/qt5-uiplugin)
  => tk86-8.6.6.txz (x11-toolkits/tk86)
  => linux-c6-libssh2-1.4.2_5.txz (security/linux-c6-libssh2)
  => linux-c6-openldap-2.4.40_2.txz (net/linux-c6-openldap)
  => linux-c6-libogg-1.1.4_2.txz (audio/linux-c6-libogg)
  => qt5-xml-5.6.2.txz (textproc/qt5-xml)
  => apr-1.5.2.1.5.4_2.txz (devel/apr1)
  => gamin-0.1.10_8.txz (devel/gamin)
  => linux-c6-sdl-1.2.14_3.txz (devel/linux-c6-sdl12)
  => p8-platform-2.0.1.txz (devel/p8-platform)
  => qt5-script-5.6.2.txz (devel/qt5-script)
  => swig30-3.0.10_1.txz (devel/swig30)
  => jakarta-commons-codec-1.10.txz (java/jakarta-commons-codec)
  => jakarta-commons-logging-1.2.txz (java/jakarta-commons-logging)
  => gnome-mime-data-2.18.0_5.txz (misc/gnome-mime-data)
  => c-ares-1.12.0.txz (dns/c-ares)
  => libirman-0.4.6.txz (comms/libirman)
  => p5-Unicode-Map8-0.13_1.txz (converters/p5-Unicode-Map8)
  => recode-3.6_12.txz (converters/recode)
  => Xaw3d-1.5E_8.txz (x11-toolkits/Xaw3d)
  => py27-tkinter-2.7.12_6.txz (x11-toolkits/py-tkinter)
  => linux-c6-openssl-compat-0.9.8e_4.txz (security/linux-c6-openssl-compat)
  => linux-c6-libvorbis-1.2.3_2.txz (audio/linux-c6-libvorbis)
  => linux-c6-curl-7.19.7_8.txz (ftp/linux-c6-curl)
  => linux-c6-dri-11.0.7_3.txz (graphics/linux-c6-dri)
  => python34-3.4.5.txz (lang/python34)
  => redeclipse-data-1.5.3.txz (games/redeclipse-data)
  => gsed-4.2.2.txz (textproc/gsed)
  => rasqal-0.9.33.txz (textproc/rasqal)
  => 9base-b20100604.txz (devel/9base)
  => ChipmunkPhysics-7.0.1.txz (devel/ChipmunkPhysics)
  => electricfence-2.2.2_2.txz (devel/ElectricFence)
  => dmake-4.12.20150309.txz (devel/dmake)
  => epm-4.3.txz (devel/epm)
  => gettext-0.19.8.1.txz (devel/gettext)
  => gperf-3.0.3_1.txz (devel/gperf)
  => libplist-1.12.txz (devel/libplist)
  => libsigsegv-2.10_1.txz (devel/libsigsegv)
  => llvm35-3.5.2_2.txz (devel/llvm35)
  => p5-IO-HTML-1.001_1.txz (devel/p5-IO-HTML)
  => patch-2.7.5.txz (devel/patch)
  => py27-logilab-common-1.2.2.txz (devel/py-logilab-common)
  => py34-setuptools34-28.1.0.txz (devel/py-setuptools34)
  => qt5-concurrent-5.6.2.txz (devel/qt5-concurrent)
  => upnp-1.6.20,1.txz (devel/upnp)
  => bsdadminscripts-6.1.1_8.txz (sysutils/bsdadminscripts)
  => jakarta-commons-httpclient-3.1_1.txz (java/jakarta-commons-httpclient)
  => xf86-video-fbdev-0.4.4_7.txz (x11-drivers/xf86-video-fbdev)
  => cmatrix-1.2a_1.txz (misc/cmatrix)
  => qt5-serialport-5.6.2.txz (comms/qt5-serialport)
  => enca-1.19.txz (converters/enca)
  => CoinMP-1.8.3.txz (math/coinmp)
  => dialog4ports-0.1.6.txz (ports-mgmt/dialog4ports)
  => croscorefonts-1.31.0.txz (x11-fonts/croscorefonts-fonts-ttf)
  => crosextrafonts-caladea-20130214_1.txz (x11-fonts/crosextrafonts-caladea-ttf)
  => crosextrafonts-carlito-20130920_2.txz (x11-fonts/crosextrafonts-carlito-ttf)
  => ffmpeg0-0.7.17_6,1.txz (multimedia/ffmpeg0)
  => libass-0.13.4.txz (multimedia/libass)
  => libcec-3.1.0.txz (multimedia/libcec)
  => libmpeg2-0.5.1_6.txz (multimedia/libmpeg2)
  => librtmp-2.4.20151223.txz (multimedia/librtmp)
  => qt5-multimedia-5.6.2.txz (multimedia/qt5-multimedia)
  => mDNSResponder-765.1.2.txz (net/mDNSResponder)
  => i386-wine-1.8.5,1.txz (emulators/i386-wine)
  => libsamplerate-0.1.9.txz (audio/libsamplerate)
  => libshairport-1.2.0.20310_2.txz (audio/libshairport)
  => lcms-1.19_6,1.txz (graphics/lcms)
  => py27-pillow-3.1.1_1.txz (graphics/py-pillow)
  => silgraphite-2.3.1_4.txz (graphics/silgraphite)
  => bsh-2.0.b6.txz (lang/bsh)
  => 3dc-0.8.1_3.txz (games/3dc)
  => briquolo-0.5.7_13.txz (games/briquolo)
  => bzflag-2.4.6.txz (games/bzflag)
  => linux-defcon-1.42_2.txz (games/linux-defcon)
  => asciidoc-8.6.9_5.txz (textproc/asciidoc)
  => hyphen-2.8.8.txz (textproc/hyphen)
  => libtextcat-2.2_6.txz (textproc/libtextcat)
  => mythes-1.2.4.txz (textproc/mythes)
  => redland-1.0.17_4.txz (textproc/redland)
  => tinyxml-2.6.2_1.txz (textproc/tinyxml)
  => libmicrohttpd-0.9.37_2.txz (www/libmicrohttpd)
  => serf-1.3.9_1.txz (www/serf)
Total packages that would be fetched: 134
The complete build list can also be found at:
/tmp/synth_prefetch_list.txt
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  N => devel/qt5-help
  N => devel/qt5-assistant
  N => devel/libftdi
  N => devel/qt5-designer
  N => x11-toolkits/wxgtk28-common
  N => www/libxul
  N => devel/gnome-vfs
  N => java/icedtea-web
  N => java/openjdk7-jre
  N => comms/lirc
  N => x11-toolkits/qt5-declarative
  N => x11-toolkits/wxgtk28
  N => graphics/libopenraw
  N => games/redeclipse
  N => www/linux-c6-flashplugin11
  N => www/nspluginwrapper
Total packages that would be built: 16
The complete build list can also be found at:
/tmp/synth_status_results.txt
[/U]
```


----------



## marino (Nov 26, 2016)

manually remove these two with `pkg remove`

```
Scan of devel/gmake-lite failed (port deleted), it will not be considered.
Scan of textproc/clucene-qt5 failed (port deleted), it will not be considered.
```

The downloaded files are probably build dependencies that won't get installed or even used.
Yes I know it's a waste to download something that's not used, but it's a side effect of the "fetch prebuilt packages".

After building those 16 with `synth prepare-system` followed by `synth upgrade-system` upon success, I imagine `synth status` will then be happy as long as you don't update the ports tree.


----------



## PacketMan (Nov 26, 2016)

Sweet, it works!

I left prefetching enabled on the builder machine, but I turned it off for the target recipient machine. And I configured pkg 
	
	



```
CONSERVATIVE_UPGRADE = false;
```
 on the target machine. (And I installed and got devel/ccache on the builder for performance gains.)



marino@ said:


> After building those 16 with `synth prepare-system` followed by `synth upgrade-system` upon success, I imagine `synth status` will then be happy as long as you don't update the ports tree.



I don't want to use synth to do the upgrading on the target machine, I want to use pkg. (But I might keep using synth we'll see). The goal is to remove synth from the old slow poke machines.  And I don't think you want me to `synth upgrade-system` on the builder machine because that will actually install the ports there right?  Anyway I got through it. It took a few multiple passes, but it worked. I did the 16 as you suggested, and then another pass suggested another 67; and I did these using a file with the ports listed; so like this `#synth just-build synth_temp_list.txt`.

After completed I did one more pass around and got here:

`# synth rebuild-repository`

```
Stand by, prescanning existing packages.
Stand by, recursively scanning 1026 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
```

`# synth status`

```
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/tmp/synth_status_results.txt
```

`# pkg upgrade` (note nothing is actually upgraded)

```
Updating Synth repository catalogue...
Synth repository is up-to-date.
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (252 candidates): 100%
Processing candidates (252 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
```

`# synth status`

```
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/tmp/synth_status_results.txt
```

Sweet!  I think the root cause of my issue was the initial seeding of synth on the builder machine with all the ports.  Is there a command that will list all the installed ports on a machine in the format category/port-name (without the -xx.x.x,x version info) that I can easily feed back into synth? I can't find a portmaster option to do it. And note, you need to ensure the syncing program has finished and synchronization has been achieved.

When I buy my next new machine it too will likely be GNOME3 desktop. I'll simply add it to the syncing program. I use 
net-p2p/btsync which works well.

Thanks again.


----------



## marino (Nov 26, 2016)

PacketMan said:


> Is there a command that will list all the installed ports on a machine in the format category/port-name (without the -xx.x.x,x version info) that I can easily feed back into synth?.



`pkg query -e '%a==0' '%o' > ~/my.build.list` will create it in the format you want.
I should get that recipe added to pkg(8) default aliases as "prime-origins"


----------



## PacketMan (Dec 3, 2016)

So this is all working great it seems. But one last question I think. I use the same process to build package that I want to install. Trouble is pkg won't install from Synth respository, pkg wants to use the FreeBSD respository. I tried some variations of the pkg command to no avail. I'm guessing I need to use a flag or to, and maybe manually specify the directory full path. Will try that later. Any thoughts?

`# ls -l | grep obs`

```
-rw-r--r--  1 btsync  wheel     2673944 Nov 30 19:57 obs-studio-0.16.0.txz
```

`# pkg add obs-studio`

```
pkg: obs-studio: No such file or directory
pkg: Was 'pkg install obs-studio' meant?
```

`# pkg install obs-studio`

```
Updating Synth repository catalogue...
Synth repository is up-to-date.
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
The following 7 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   obs-studio: 0.16.0 [FreeBSD]
   jackit: 0.124.1_5 [FreeBSD]
   libsysinfo: 0.0.3 [FreeBSD]
   qt5-imageformats: 5.6.2 [FreeBSD]
   libmng: 1.0.10_2 [FreeBSD]
   qt5-x11extras: 5.6.2 [FreeBSD]
   fdk-aac: 0.1.4 [FreeBSD]

Number of packages to be installed: 7
```


`pkg install -R Synth obs-studio`

```
Updating Synth repository catalogue...
Synth repository is up-to-date.
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
The following 7 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   obs-studio: 0.16.0 [FreeBSD]
   jackit: 0.124.1_5 [FreeBSD]
   libsysinfo: 0.0.3 [FreeBSD]
   qt5-imageformats: 5.6.2 [FreeBSD]
   libmng: 1.0.10_2 [FreeBSD]
   qt5-x11extras: 5.6.2 [FreeBSD]
   fdk-aac: 0.1.4 [FreeBSD]
```


`# pkg -vv`


```
Repositories:
  Synth: {
    url             : "file:///var/synth/live_packages",
    enabled         : yes,
    priority        : 0
  }
  FreeBSD: {
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }

CONSERVATIVE_UPGRADE = false;
```

Lastly, I tried to install using Synth and it works without issue:
`# synth install multimedia/obs-studio`

```
New packages to be INSTALLED:
   obs-studio: 0.16.0 [Synth]
   jackit: 0.124.1_5 [Synth]
   libsysinfo: 0.0.3 [Synth]
   qt5-imageformats: 5.6.2 [Synth]
   libmng: 1.0.10_2 [Synth]
   qt5-x11extras: 5.6.2 [Synth]
   fdk-aac: 0.1.4 [Synth]

Number of packages to be installed: 7
```


----------



## Remington (Dec 3, 2016)

This is incorrect.  It should be `pkg install -r Synth obs-studio` with lowercase 'r'.  Check `man pkg-install` and you'll see why your command didn't work since 'R' is recursive and 'r' is reponame.



PacketMan said:


> `pkg install -R Synth obs-studio`



Priority number should be higher for Synth than FreeBSD.  ie, 100.


PacketMan said:


> ```
> Repositories:
> Synth: {
> url             : "file:///var/synth/live_packages",
> ...


----------



## PacketMan (Dec 3, 2016)

Remington said:


> It should be `pkg install -r Synth obs-studio`
> Priority number should be higher for Synth than FreeBSD.  ie, 100.



I did both and neither worked. I was wondering about the priority number. But the strange thing is, I uninstalled the package and then tried both again just to view the results again.  This time was different, the -r option worked. I musta typoed.


----------

