# Synth 1.30 has been released



## marino (Mar 10, 2016)

```
ports-mgmt/synth: Upgrade version 1.22 => 1.30

This is an enhancement release that contains a bug fix.

Description of bug:
  If synth is launched from mountpoint of the ports directory
  (e.g. /usr/ports), it may malfunction with strange messages such
  as "invalid origins" and printing strange symbols to the screen
  along with parts of a directory.

  As a result, Synth now detects this launch location (as it already
  did with /usr/local) and refuses to run until the current directory
  is changed outside of the ports tree.

Enhancement 1:
  It is now possible to remove alternate profiles.  When more than
  one profile exists, a menu option "<" appears that provides the
  user with an opportunity to remove one to all inactive profiles
  from the configuration.  The man page has been updated as well.

Enhancement 2:
  Synth will automatically convert any directory inputs on the
  configure command to the true path.  For example, if somebody types
  in "/usr/xports" for the ports directory, but that's just a
  symbolic link to /vcs/freebsd-ports", the entry will automatically
  convert to its true resolved path (e.g. /vcs/freebsd-ports). This
  enables the cwd detection of the bug fix above to always work.
```


----------



## marino (Mar 10, 2016)

FYI, there was one previously unannounced bug fix before it:


```
ports-mgmt/synth: Upgrade version 1.21 => 1.22

Two minor bug fixes:
* A specific check during test mode would emit a failure to stdout when
   testing devel/py-setuptools27.  It turns out that there's a file there
   with a space in the filename.  The filename was an argument for
   /usr/bin//file and it wasn't escaped.  The file in question had
   parentheses too which the shell was trying to process.  The fix was
   to escape the filename in the /usr/bin/file command.
* The builders were mounting the source directory from "/usr/src", not
   $sysroot/usr/src as intended.  This potentially causes breakage when
   the $sysroot reflects a different versions/release than the host
   machine has (e.g. making FreeBSD 10.2 packages on FreeBSD 11-current).
   Now the source directory mount is relative to profile's $sysroot.
```

So definitely upgrade to 1.30, mainly for the ports directory check which prevents nasty output.  (Thanks Crivens)


----------



## france75006 (Mar 10, 2016)

I've just installed ports-mgmt/synth 1.30 on a FreeBSD 9.3 machine and on the first run, this is what I get:


```
$ synth

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 1.30
====================================================================
  Copyright (C) 2015-2016 John R. Marino


Usage: synth [zero-parameter-option]
-or-  synth [list-option] <list of port origins | filename>

zero-parameter-option includes 'help', 'configure', 'version' (this screen),
  'status', 'upgrade-system', 'prepare-system',
  'status-everything', 'everything', 'purge-distfiles',
  'rebuild-repository'
list-option includes  'status', 'build', 'just-build', 'install', 'force'
  'test'

# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
mount: /usr/obj/synth-live/SL09/boot/modules: No such file or directory

raised REPLICANT.SCENARIO_UNEXPECTED : /sbin/mount -t tmpfs -o size=100M tmpfs /usr/obj/synth-live/SL09/boot/modules => failed with code 64
```

I've tried again a few seconds after and got:


```
# synth status
Builder mounts detected; attempting to remove them automatically ...
piped_mute_command failure:
 => umount: 18G: statfs: No such file or directory
umount: 18G: unknown file system
umount: 9.1G: statfs: No such file or directory
umount: 9.1G: unknown file system
umount: 8.3G: statfs: No such file or directory
umount: 8.3G: unknown file system
umount: 53%: statfs: No such file or directory
umount: 53%: unknown file system
umount: /usr/obj/synth-live/SL09/construction: not a file system root directory

piped_mute_command failure:
 => umount: 18G: statfs: No such file or directory
umount: 18G: unknown file system
umount: 9.1G: statfs: No such file or directory
umount: 9.1G: unknown file system
umount: 8.3G: statfs: No such file or directory
umount: 8.3G: unknown file system
umount: 53%: statfs: No such file or directory
umount: 53%: unknown file system
umount: /usr/obj/synth-live/SL09/usr/local: not a file system root directory

Dismounting successful!
Old work directories detected; attempting to remove them automatically ...
Directory removal successful!
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
mount: /usr/obj/synth-live/SL09/boot/modules: No such file or directory

raised REPLICANT.SCENARIO_UNEXPECTED : /sbin/mount -t tmpfs -o size=100M tmpfs /usr/obj/synth-live/SL09/boot/modules => failed with code 64
```


----------



## marino (Mar 10, 2016)

france75006@, you don't have /boot/modules directory ?

if not, create it, but I thought /boot/modules existed on every system.


----------



## france75006 (Mar 10, 2016)

I didn't have that directory (it's an OVH dedicated server), but now that I've created it everything seems ok for synth: thanks!


----------



## Wapcaplet (Mar 11, 2016)

I'm in the process of building a new server with full-disk encryption and /boot on a USB drive.  Does this mean I will need to leave the /boot directory mounted whenever I run synth?  I had planned to remove the USB drive after each startup and only mount it when changes to /boot were necessary.


----------



## marino (Mar 11, 2016)

No, /boot is optional.  But if you have /boot, you also have to have /boot/modules.  At least, that's true for 1.30 and earlier.

edit: note that certain options (DTRACE) and certain ports (e.g. kernel modules) require /boot and /boot/modules to build, so not having it will ensure those ports do not build.  But they are fairly rare.


----------



## Synth Meat (Mar 16, 2016)

Deploying some new machines, and decided to take synth for a test ride...

So:
- I've set up my pkg to latest instead quarterly
- updated and upgraded
- snapped them ports
- there's nothing on the system yet, installed synth
- configured it to leverage_prebuilt=true
- did "synth status" to dry run it, it said that it'll fetch 121 packages and install 5 new ones
- now it's building 126 packages :/

I've kinda hoped it'll fetch prebuilt packages? It's installing nothing that's not readily available via pkg.


----------



## marino (Mar 16, 2016)

Synth Meat said:


> - did "synth status" to dry run it, it said that it'll fetch 121 packages and install 5 new ones
> - now it's building 126 packages :/


I'll take a look at the code.  `synth status` is supposed to be accurate and apparently it was able to query the remote database.


----------



## marino (Mar 16, 2016)

by the way, what command did you execute after `synth status`?


----------



## Synth Meat (Mar 16, 2016)

`synth upgrade-system`

It's still running, ~2h now.


----------



## marino (Mar 16, 2016)

Okay, here's my best guess:
Synth uses the host pkg(8) program to prefetch the packages.
This particular command continues quietly when it failed, but I think it should have said "failed to download at least one package" and stopped then.

so if the above is true, and host pkg program was missing, synth needs to abort immediately.  Also there should be some indication if the actual command fails.  I'll put this on the to do list regardless.


----------



## marino (Mar 16, 2016)

although I'm not sure it's possible to have synth installed w/o pkg .... so that might blow the theory ...


----------



## Synth Meat (Mar 16, 2016)

No, pkg(8) definitely exists. That's how I installed synth 1.30 in the first place.


----------



## marino (Mar 16, 2016)

so far I can't see how the dry run would work but the actual fetching would fail without stopping.
It has a list of packages it must fetch (from the remote query) and all have to be present before Synth starts building.

I'm not saying you're wrong, only that I have to set up a test case and try to reproduce it myself.


----------



## marino (Mar 17, 2016)

Synth Meat@,
During my attempt to reproduce, I got a different failure signature.  When I put in so print lines, I got a command which I re-ran on the system:

```
ironwolf# /usr/local/sbin/pkg rquery -U %q -r Wolfpond uhttpmock-0.5.0
Repository Synth has a wrong packagesite, need to re-create database
pkg: Repository Synth cannot be opened. 'pkg update' required
dragonfly:4.6:x86:64
```

In my case, a bad "synth" repo is interfering with the external repo check.  You might be experiencing the same thing.
It seems pkg(8) is doing something unexpected here -- if the command limits the repository to "Wolfpond", why are errors about other repositories being reported?


----------



## marino (Mar 17, 2016)

Hmm, look at what happens when I rearrange the command:

```
ironwolf# /usr/local/sbin/pkg rquery -r Wolfpond -U %q uhttpmock-0.5.0
dragonfly:4.6:x86:64
```


----------



## marino (Mar 17, 2016)

Synth Meat said:


> No, pkg(8) definitely exists. That's how I installed synth 1.30 in the first place.



I think the command would have worked if used `synth prepare-repository`.  What I think happened is that `synth upgrade-system` wrote the /usr/local/etc/pkg/repos/00_synth.conf before executing the build command, and due to the `pkg rquery` issue above, it caused error messages that weren't seen on the previous `synth status` output.

I've got this fixed in the repo and I'll push 1.31 shortly so you can re-test in your VPS.


----------



## marino (Mar 17, 2016)

```
ports-mgmt/synth: Upgrade version 1.30 => 1.31

Pkg(8) bug workaround:
  The pkg rquery man page is incorrect.  It lists the invocation of
  pkg rquery as "-U -r reponame" but if it's invoked in that order, it
  will squawk errors if any of the repositories are misconfigured.  The
  command must be "pkg rquery -r reponame -U" in order to limit pkg(8) to
  a single repository.

  The order sensitivity came into play with "synth upgrade-system" command
  on a brand new system.  The 00_synth.conf pkg(8) configuration file was
  created but there were no repository files created yet, cause pkg(8) to
  emit errors on Synth repo when checking the official FreeBSD package
  repo. This resulted in no packages prefetching even though "synth status"
  indicated some would be fetched.  Rearranging the internal rquery
  commands works around the misleading pkg(8) documentation (or bug?).

FreeBSD-specific system root validity check.
  The $SYSROOT/boot directory doesn't have to be present, but if it is,
  there must also be present $SYSROOT/boot/modules directory. Synth now
  checks for this on FreeBSD and stops with an informative message when
  the modules directory needs to be created by the root user.
```


----------



## Synth Meat (Mar 17, 2016)

You meant `synth prepare-system`. It worked. 

Haven't tried 1.31 yet though, but will these days.

Thanks for the help!


----------



## Wapcaplet (Mar 19, 2016)

I've discovered some strange behavior when changing the options in a port.

Let's say I have a port installed with the config settings all at defaults.  I then change the config file (`make config`) and run `synth upgrade-system`.  Synth proceeds to rebuild the port in the repository, but then the updated package doesn't get installed.  I then run `synth upgrade-system` again.  The port gets rebuilt again in the repository, but this time the newly-built package is installed in place of the old one.

I've confirmed this behavior on two separate systems with different ports.

Here's the output of Synth right after changing a port option (security/openssh-portable) and then immediately rerunning Synth a second time:


```
# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
openssh-portable-7.2.p2,1.txz failed option check.
progress: 90.83%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:34
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  260 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  70 KiB  71.4kB/s  00:01
Processing entries: 100%
Synth repository update completed. 245 packages processed.
Checking for upgrades (13 candidates): 100%
Processing candidates (13 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
progress: 87.07%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:34
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  260 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  71 KiB  72.2kB/s  00:01
Processing entries: 100%
Synth repository update completed. 246 packages processed.
Checking for upgrades (14 candidates): 100%
Processing candidates (14 candidates): 100%
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
  openssh-portable-7.2.p2,1 (options changed)

The operation will free 19 KiB.
[1/1] Reinstalling openssh-portable-7.2.p2,1...
You may need to manually remove /usr/local/etc/ssh/sshd_config if it is no longer needed.
[1/1] Extracting openssh-portable-7.2.p2,1: 100%
Message from openssh-portable-7.2.p2,1:
To enable this port, add openssh_enable="YES" in your rc.conf. To
prevent conflict with openssh in the base system add sshd_enable="NO"
in your rc.conf. Also you can configure openssh at another TCP port (via
sshd_config 'Port' and 'Listen' options or via 'openssh_flags'
variable in rc.conf) and run it in same time with base sshd.

'PermitRootLogin no' is the default for the OpenSSH port.
This now matches the PermitRootLogin configuration of OpenSSH in
the base system.  Please be aware of this when upgrading your
OpenSSH port, and if truly necessary, re-enable remote root login
by readjusting this option in your sshd_config.

Users are encouraged to create single-purpose users with ssh keys, disable
Password authentication by setting 'PasswordAuthentication no' and
'ChallengeResponseAuthentication no', and to define very narrow sudo
privileges instead of using root for automated tasks.
```
This is with Synth 1.31.  Is there anything else I can provide that would help debug this?


----------



## marino (Mar 19, 2016)

First, this is pkg(8), not synth.  Synth just makes the repository; pkg(8) is responsible for updating the system (Synth just spawns a command to pkg(8)).

Secondly, what is the output of `pkg -vv | grep CONS`?
Is it CONSERVATIVE_UPGRADE = true?

If so, that might the pkg(8) setting causing it.


----------



## Wapcaplet (Mar 19, 2016)

marino@ said:


> First, this is pkg(8), not synth.  Synth just makes the repository; pkg(8) is responsible for updating the system (Synth just spawns a command to pkg(8)).
> 
> Secondly, what is the output of `pkg -vv | grep CONS`?
> Is it CONSERVATIVE_UPGRADE = true?
> ...


Yes, CONSERVATIVE_UPGRADE=true is the result, but that's not the problem.

It turns out Synth does not copy the package into the repository after the first build attempt, leaving the repo without any package for that port.  It does copy the package on the second attempt, and only then can `pkg` find it to install it.


```
root@wintermute-pts/2(2:08pm)~/:ll /var/synth/live_packages/All/openssh-portable-7.2.p2,1.txz
-rw-r--r--  1 root  wheel  632400 Mar 19 08:27 /var/synth/live_packages/All/openssh-portable-7.2.p2,1.txz
root@wintermute-pts/2(2:08pm)~/:make -C /usr/ports/security/openssh-portable config

root@wintermute-pts/2(2:08pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
openssh-portable-7.2.p2,1.txz failed option check.
progress: 86.38%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:34
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(2:10pm)~/:ll /var/synth/live_packages/All/openssh-portable*
ls: No match.
root@wintermute-pts/2(2:12pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
progress: 85.84%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:34
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(2:13pm)~/:ll /var/synth/live_packages/All/openssh-portable*
-rw-r--r--  1 root  wheel  627176 Mar 19 14:12 /var/synth/live_packages/All/openssh-portable-7.2.p2,1.txz
root@wintermute-pts/2(2:16pm)~/:
```
As you can see, at first the package exists in the repo -- in this state, there is no config file.  I then run `make config` and change a few settings.  I then run `synth prepare-system`, and once the port is finished building and Synth completes its work, the openssh-portable package file is missing from the repo.  I run `synth prepare-system` again, Synth rebuilds the port a second time (because the package is now missing from the repo), and the package now appears, ready to be installed by `pkg`.

Can you reproduce this using the above steps?


----------



## marino (Mar 19, 2016)

Wapcaplet said:


> It turns out Synth does not copy the package into the repository after the first build attempt, leaving the repo without any package for that port.  It does copy the package on the second attempt, and only then can `pkg` find it to install it.



This makes no sense.  There's no copying, ever.  the packages are created in the proper place and the `pkg repo` command generates the digests, etc.




> As you can see, at first the package exists in the repo -- in this state, there is no config file.  I then run `make config` and change a few settings.



This also makes no sense.  no config file is the best state to be in.  The lack of a config file is a good thing.



> I then run `synth prepare-system`, and once the port is finished building and Synth completes its work, the openssh-portable file is missing from the repo.  I run `synth prepare-system` again, Synth rebuilds the port a second time (because the package is now missing from the repo), and the package now appears, ready to be installed by `pkg`.



Did you notice the openssh-portable file got removed during the repositiory creation because it was invalid?
The opposite of what you think is happening.  It's in place, but then removed because it fails validity checks.

If openssh-portable gets removed repeatedly, there's a problem with the port (e.g.  a LIB_DEPENDS that conditionally satisfied).  Synth is doing what it's supposed to do.  It doesn't do a full repository validation until right before the repo is created, and packages get deleted in the process (sometimes a lot of packages)


----------



## Wapcaplet (Mar 19, 2016)

marino@ said:


> This also makes no sense.  no config file is the best state to be in.  The lack of a config file is a good thing.


Now this makes no sense.  If I want to disable a default in a port config, or if I have to enable a different setting in a port, then config files are absolutely needed.  If I didn't change the configs, I'd rely solely on `pkg` and not even bother with Synth.

Even if I'm not describing the process properly, Synth is clearly doing something wrong here.  Here's another example, and I'll try to make it as clear as possible.

First, let's start with installing a brand-new port, say dns/ddclient because it has only one config option.


```
root@wintermute-pts/2(3:01pm)~/:synth install dns/ddclient



The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:10
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  264 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  71 KiB  72.2kB/s  00:01
Processing entries: 100%
Synth repository update completed. 246 packages processed.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  ddclient: 3.8.2_1

The process will require 178 KiB more space.
[1/1] Installing ddclient-3.8.2_1...
[1/1] Extracting ddclient-3.8.2_1: 100%
Message from ddclient-3.8.2_1:
***********************************************************

Copy
  /usr/local/etc/ddclient.conf.sample
to
  /usr/local/etc/ddclient.conf

and edit it to fit your needs.

If you would like to run ddclient as a daemon add the
following line to /etc/rc.conf

  ddclient_enable="YES"

If you would like to force ddclient to update your account
daily regardless of IP changes add the following line to
your /etc/periodic.conf

  daily_ddclient_force_enable="YES"

***********************************************************
root@wintermute-pts/2(3:02pm)~/:
```
So dns/ddclient is now installed, and the package file exists in the repo.

Right now, no port options are manually set:


```
root@wintermute-pts/2(3:03pm)~/:ll /var/db/ports/dns_ddclient*
ls: No match.
root@wintermute-pts/2(3:04pm)~/:
```

Let's change the port's only option, SSL, by disabling it (it's enabled by default):


```
root@wintermute-pts/2(3:04pm)~/:make -C /usr/ports/dns/ddclient config

root@wintermute-pts/2(3:05pm)~/:ll /var/db/ports/dns_ddclient*
total 2
-rw-r--r--  1 root  wheel  166 Mar 19 15:05 options
root@wintermute-pts/2(3:05pm)~/:cat /var/db/ports/dns_ddclient/options
# This file is auto-generated by 'make config'.
# Options for ddclient-3.8.2_1
_OPTIONS_READ=ddclient-3.8.2_1
_FILE_COMPLETE_OPTIONS_LIST=SSL
OPTIONS_FILE_UNSET+=SSL
root@wintermute-pts/2(3:05pm)~/:
```
Now that we've changed a port option, let's rerun Synth and update the repository:


```
root@wintermute-pts/2(3:05pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
ddclient-3.8.2_1.txz failed option check.
progress: 86.72%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:13
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(3:07pm)~/:
```
So dns/ddclient failed the option check when I disabled its one option, and the package should have been rebuilt with the new options selected (or deselected in this case).  But the newly-built package is missing from the repository and cannot be installed by `pkg`.


```
root@wintermute-pts/2(3:07pm)~/:ll /var/synth/live_packages/All/ddclient*
ls: No match.
```
This, as I understand it, is wrong.  Why is the package now missing from the repository after it was successfully built with `synth prepare-system`?  Where did it go?  Why do I have to run Synth twice to get the package file back in the repo and ready to be installed?


```
root@wintermute-pts/2(3:08pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
progress: 87.27%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:13
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(3:12pm)~/:ll /var/synth/live_packages/All/ddclient*
-rw-r--r--  1 root  wheel  40584 Mar 19 15:12 /var/synth/live_packages/All/ddclient-3.8.2_1.txz
root@wintermute-pts/2(3:12pm)~/:
```

Can you explain to me why this is normal?


----------



## marino (Mar 19, 2016)

Wapcaplet said:


> Now this makes no sense.  If I want to disable a default in a port config, or if I have to enable a different setting in a port, then config files are absolutely needed.  If I didn't change the configs, I'd rely solely on `pkg` and not even bother with Synth.



These replies are crazy long.
Okay, let's say you have 500 ports installed.
you didn't customize 500 ports.  You might have customized 8 ports, 8 of 500.
So you need 8 customized options.  You have 492 ports wthout options.
A port without options never has a bad config.  A port with cached options can go invalid with changes to the tree.  ergo, a ports wthout config files are in a better state.






> Even if I'm not describing the process properly, Synth is clearly doing something wrong here.  Here's another example, and I'll try to make it as clear as possible.
> 
> ddclient-3.8.2_1.txz failed option check.
> progress: 86.72%



why do you think the failed check is wrong?



> So dns/ddclient failed the option check when I disabled its one option, and the package should have been rebuilt with the new options selected (or deselected in this case).  But the newly-built package is missing from the repository and cannot be installed by `pkg`



hmmm?  it's not missing, it got deleted.



> This, as I understand it, is wrong.  Why is the package now missing from the repository?  Why do I have to run Synth twice to get the package file back in the repo and ready to be installed?



It just deleted, you see it on the screen.  

I'm not understanding your point.  You change the port options which immediately invalidate the package, which gets removed because of the invalidation. When you build it a second time, the package's stored options match the current ports tree so it passes the validation check.


----------



## Wapcaplet (Mar 19, 2016)

marino@ said:


> why do you think the failed check is wrong?


It's not wrong.  That's not the problem.  Synth correctly detects the options have changed and deletes the old package file. That's fine.

What's not fine is that Synth should then rebuild the package with the new option settings on the first invocation of `synth prepare-system`.  Yes, it should delete the old package file, because those options are now outdated, but then it proceeds to rebuild the package using the new option settings, but doesn't actually repopulate the repo with the new package file it just built a few seconds ago!  The newly-built package just disappears!



marino@ said:


> I'm not understanding your point.  You change the port options which immediately invalidate the package, which gets removed because of the invalidation. When you build it a second time, the package's stored options match the current ports tree so it passes the validation check.


OK, here's my point.  The deletion is fine.  When I run `synth prepare-system` after changing a port option, it should do two things: (1) Delete the old package with the outddated option setting, (2) Build the new package using the new option settings and place the new package file in the repo.  Running `synth prepare-system` once should accomplish both steps.  But it doesn't.  Only (1) gets completed.  I have to run `synth prepare-system` a second time to complete (2).  Are you saying that's by design?


----------



## marino (Mar 19, 2016)

i just looked at your dns_ddclient client example again.  The output isn't matching what you describe.

you ran `synth prepare-system`
It deleted the bad dns_ddclient package
It rebuilt the package
It rebuilt the repository
One pass.  It did 1) and 2).  Correct?


----------



## Wapcaplet (Mar 19, 2016)

marino@ said:


> i just looked at your dns_ddclient client example again.  The output isn't matching what you describe.
> 
> you ran `synth prepare-system`
> It deleted the bad dns_ddclient package
> ...


No, because the repository was missing the new dns/ddclient package file after the first invocation of `synth prepare-system` completed.  Here, I'll show you again by starting from scratch.  Bear with me on this, please -- I know this is long, but please follow along through all steps to reproduce:

Getting rid of dns/ddclient:


```
root@wintermute-pts/2(3:13pm)~/:pkg delete ddclient
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
  ddclient-3.8.2_1

The operation will free 178 KiB.

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling ddclient-3.8.2_1...
[1/1] Deleting files for ddclient-3.8.2_1: 100%
root@wintermute-pts/2(3:48pm)~/:rm /var/synth/live_packages/All/ddclient-3.8.2_1.txz
remove /var/synth/live_packages/All/ddclient-3.8.2_1.txz? y
/var/synth/live_packages/All/ddclient-3.8.2_1.txz
root@wintermute-pts/2(3:48pm)~/:make -C /usr/ports/dns/ddclient/ rmconfig
===> Removing user-configured options for ddclient-3.8.2_1
root@wintermute-pts/2(3:49pm)~/:
```
The package is now removed from the system, the options file has been deleted, and I manually deleted the package file from the repository.

Now we reinstall with `synth install dns/ddclient`:


```
root@wintermute-pts/2(3:49pm)~/:synth install dns/ddclient



The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:10
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  264 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  71 KiB  72.2kB/s  00:01
Processing entries: 100%
Synth repository update completed. 246 packages processed.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  ddclient: 3.8.2_1

The process will require 178 KiB more space.
[1/1] Installing ddclient-3.8.2_1...
[1/1] Extracting ddclient-3.8.2_1: 100%
Message from ddclient-3.8.2_1:
***********************************************************

Copy
  /usr/local/etc/ddclient.conf.sample
to
  /usr/local/etc/ddclient.conf

and edit it to fit your needs.

If you would like to run ddclient as a daemon add the
following line to /etc/rc.conf

  ddclient_enable="YES"

If you would like to force ddclient to update your account
daily regardless of IP changes add the following line to
your /etc/periodic.conf

  daily_ddclient_force_enable="YES"

***********************************************************
root@wintermute-pts/2(3:50pm)~/:ll /var/synth/live_packages/All/ddclient*
-rw-r--r--  1 root  wheel  40640 Mar 19 15:50 /var/synth/live_packages/All/ddclient-3.8.2_1.txz
root@wintermute-pts/2(3:51pm)~/:
```
The package is now installed, and ddclient-3.8.2_1.txz is in the repo.

Time to change the option:


```
root@wintermute-pts/2(3:51pm)~/:make -C /usr/ports/dns/ddclient config

root@wintermute-pts/2(3:52pm)~/:cat /var/db/ports/dns_ddclient/options
# This file is auto-generated by 'make config'.
# Options for ddclient-3.8.2_1
_OPTIONS_READ=ddclient-3.8.2_1
_FILE_COMPLETE_OPTIONS_LIST=SSL
OPTIONS_FILE_UNSET+=SSL
root@wintermute-pts/2(3:53pm)~/:
```
The option is now changed.  Now we run `synth prepare-system` to see if both (1) and (2) above are completed in one run:


```
root@wintermute-pts/2(3:53pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
ddclient-3.8.2_1.txz failed option check.
progress: 89.93%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:13
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(3:54pm)~/:ll /var/synth/live_packages/All/ddclient*
ls: No match.
root@wintermute-pts/2(3:55pm)~/:
```
And we see that only (1) happened -- the old package was deleted, but the new package was not put in the repo -- the new version of ddclient-3.8.2_1.txz is not in /var/synth/live_packages/All.  But we did build it during this run -- note that "packages built: 1" is in the final tally.  But where's the new package I just built?  We have to run `synth prepare-system` a second time for the new package to appear in the repo:


```
root@wintermute-pts/2(3:56pm)~/:synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
progress: 85.50%


The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:00:13
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 246 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
root@wintermute-pts/2(4:02pm)~/:ll /var/synth/live_packages/All/ddclient*
-rw-r--r--  1 root  wheel  40592 Mar 19 16:01 /var/synth/live_packages/All/ddclient-3.8.2_1.txz
root@wintermute-pts/2(4:02pm)~/:
```
The package was rebuilt again (packages built: 1), but now we see (finally!) that (2) has been completed -- the new package is in the repo and is ready to install.  I shouldn't have to run `synth prepare-system` twice for that to happen, right?


----------



## marino (Mar 19, 2016)

Wapcaplet said:


> And we see that only (1) happened -- the old package was deleted, but the new package was not put in the repo -- the new version of ddclient-3.8.2_1.txz is not in /var/synth/live_packages/All.  But we did build it during this run -- note that "packages built: 1" is in the final tally.  But where's the new package I just built?



That's what I was saying.  Something was built, I assume ddclient in this case, so
/var/synth/live_packages/All/ddclient-3.8.2_1.txz should exist at this point.

It also should be accounted for after `pkg repo` command completes.




> We have to run `synth prepare-system` a second time for the new package to appear in the repo:
> 
> 
> ```
> ...



What was just built?  ddclient should already be present so nothing should be rebuilt.
We didn't see it deleted before the repo build command.



> The package was rebuilt again (packages built: 1), but now we see (finally!) that (2) has been completed -- the new package is in the repo and is ready to install.  I shouldn't have to run `synth prepare-system` twice for that to happen, right?



no, you should not have to do that.  The question is did the package from the first build go?  it clearly built and was not deleted so it should still be there after the first run.[/code]


----------



## Wapcaplet (Mar 19, 2016)

marino@ said:


> That's what I was saying.  Something was built, I assume ddclient in this case, so
> /var/synth/live_packages/All/ddclient-3.8.2_1.txz should exist at this point.
> 
> It also should be accounted for after `pkg repo` command completes.


That's the bug I'm trying to report -- it doesn't exist at this point!  Look at the bottom of my log after the first time I run `synth prepare-system`: the file /var/synth/live_packages/All/ddclient-3.8.2_1.txz does not exist!



marino@ said:


> What was just built?  ddclient should already be present so nothing should be rebuilt. We didn't see it deleted before the repo build command.



dns/ddclient was built a second time.



marino@ said:


> no, you should not have to do that.  The question is did the package from the first build go?  it clearly built and was not deleted so it should still be there after the first run.


Right, but it's not there!

I'm not sure how else to explain this now.  Would you please try to reproduce the problem using the steps I outlined and let me know if the same thing happens to you?


----------



## marino (Mar 19, 2016)

Maybe tomorrow.  it's late here and the machine is busy with something else.

To reword this, you're saying that the first build of ddclient fails to package.   I don't see how that's possible if it's listed as a successful build.  The only other possibility is that it does successfully package, but then is immediately deleted, silently, before the repo command is executed.  Both are hard to believe.

After 1), the ddclient build log and the 00*.log doesn't have anything interesting in it?


----------



## Wapcaplet (Mar 20, 2016)

marino@ said:


> After 1), the ddclient build log and the 00*.log doesn't have anything interesting in it?


I just restarted the whole process again, and both look normal after (1).  Here's 00_last_results.log after the first `synth prepare-system`:


```
-=>  Chronology of last build  <=-
Started: Saturday, 19 MAR 2016 at 23:47:40 UTC
Ports to build: 241

Purging any ignored/broken ports first ...
00:00:05 Sanity check complete. Ports remaining to build: 1
00:00:05 [01] => dns/ddclient
00:00:14 dns/ddclient success
Finished: Saturday, 19 MAR 2016 at 23:47:54 UTC
Duration: 00:00:13
```

And after the second `synth prepare-system`:

```
-=>  Chronology of last build  <=-
Started: Saturday, 19 MAR 2016 at 00:05:11 UTC
Ports to build: 241

Purging any ignored/broken ports first ...
00:00:05 Sanity check complete. Ports remaining to build: 1
00:00:05 [01] => dns/ddclient
00:00:14 dns/ddclient success
Finished: Saturday, 19 MAR 2016 at 00:05:25 UTC
Duration: 00:00:13
```


----------



## marino (Mar 20, 2016)

Wapcaplet said:


> I just restarted the whole process again, and both look normal after (1).



I was able to reproduce this using your test case.  Indeed the newly-built package was immediately deleted and therefore not packaged.  Thanks for being persistent.  I think version 1.32 will resolve this for you.


```
ports-mgmt/synth: Upgrade version 1.31 => 1.32

prepare-system, upgrade-system package deletion bug:
  These two commands had a serious logic flaw that resulted in newly-built
  packages to be immediately deleted prior the pkg(8) repo command.  (And
  the obvious result of the system not getting updated without re-running
  the same command.)

  The cause was that the first package validity scan marked the package
  for deletion, and this setting was never reset.  The obsolete package
  was correctly deleted and rebuilt, but then it was quickly deleted
  again during the second package scan because the previous deletion mark
  was still present.  The fix was simple: Reset the deletion marks before
  rebuilding the repository so the second scan proceeds accurately.

  Test case provided by Wapcaplet in FreeBSD forums
```


----------



## fernandel (Mar 20, 2016)

I have the same problems as Wapcaplet. Thank you for update.


----------



## Wapcaplet (Mar 20, 2016)

marino@ said:


> I was able to reproduce this using your test case.  Indeed the newly-built package was immediately deleted and therefore not packaged.  Thanks for being persistent.  I think version 1.32 will resolve this for you.


Yes, that fixed it.  Thanks again for your help!


----------



## marino (Mar 27, 2016)

*ports-mgmt/synth: Upgrade version 1.32 => 1.33*

```
Prefetching option fix:
  Until now, if the "prefetch suitable packages" option was enabled, it
  could happen that "synth force" and "synth test" commands wouldn't do
  anything.  Moreover, even though it wouldn't build, it would delete the
  existing package and refetch it because force/test always pre-deletes
  existing packages.

  Now, ports listed with "force" and "test" commands are always built,
  even if suitable remote packages are available.

Improvements with test mode:
  1) Changes to /var/tmp are now ignored.  These are legal modifications
  2) Access time modification to the builder mount are now ignored.  This
     was possible if changes to /tmp or /var propagates upward (legally).
  3) A new environment variable is added for test command, "LOCK".  If it
     defined to any value, the builder's localbase will be remounted as
     read-only during the configure and build phases.  This is a
     diagnostic tool to help pinpoint where ports are committing file
     system violations by writing to localbase during these phases.

New notice to user regard CONSERVATIVE_UPGRADE:
  At the end of the "synth rebuild-repository" command, Synth will probe
  the host pkg(8) to see if it's configured with CONSERVATIVE_UPGRADE.
  If so, Synth will emit a notice because chances are good that the
  user built repository for the local system and may try to upgrade the
  system with pkg(8), and in this case, they may wish to turn off the
  CONSERVATIVE_UPGRADE setting.
```


----------



## xtaz (Mar 28, 2016)

Is this meant to happen now with this new version? It used to just say something about no packages needing to be upgraded so nothing to do. Now it exits with what looks like an error?


```
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
No packages qualify for prefetching from official package repository.
Unfortunately, the system upgrade failed.
```


```
# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
Unfortunately, the system upgrade failed.
```


----------



## marino (Mar 28, 2016)

it looks like fixing one issue caused a regression.
Can you test something for me?
do a `make clean; make patch` in ports-mgmt/synth, then find `src/portscan-pilot.adb` in the $WRKDIR and remove lines 278-280, then build/reinstall.  That should put it back to how it was.  I was trying to make it not ask to rebuild repository if nothing built.


----------



## marino (Mar 28, 2016)

I went ahead and commited that patch to ports.  Please let us know if that restores the previous behavior.


----------



## xtaz (Mar 28, 2016)

Apologies for me taking a while to test this. Been a bit busy today. I've just tested it with the same two commands as I mentioned in the post above. They both look good now. I get the original message about having nothing left to do. Thanks!


----------

