# Synth version 2.00: Flavor support



## marino (Dec 3, 2017)

After getting blindsided with the release of flavors, I spent a few hours upgrading Synth to support them.  You can upgrade to the latest version of Synth immediately as follows:


cd /usr/ports/ports-mgmt/synth
edit Makefile, change *DISTVERSION *from "1.71" to "2.00"
`make makesum`

`make clean deinstall`

`make install`

There is one major change to how Synth works now.  In order to handle the change from 1-to-1 to many-to-1 relationship between packages and ports, a new index is required.  Since ports doesn't supply this index, Synth has to generate it whenever the ports tree changes.  To do this, Synth has to scan every single port in the tree.  On DragonFly, this takes about 60 seconds.  On FreeBSD, it takes several minutes.  (The technical explanation is out of scope, but basically dports fixes several bottlenecks that never made it back to freebsd ports).

So when new new Synth starts up, it will immediately begin the scan.  Subsequent launches of Synth will work as before until the ports tree is updated.

For ports developers:
This new behavior can cause problems if you're modifying a port.  Synth will detect if a single file is newer than the current index.  For you, I've added the following logic:  As long as the ports tree directories match the new index (no port directories added or deleted), modifying a file will result in this message:


```
*******************************************************************
 The ports tree has at least one file newer than the flavor index.
 However, port directories perfectly match.  Should the index be
 regenerated? (Y/N)
```

Normally you'd select "N" which would "touch" the index and continue without performing a lengthy scan.  If your modifications define or remove flavors, you may want answer "yes" here. 

Regular users will not see this message though, and will only have to deal with the new index generation when the port tree changes.

Hopefully tobik@ will find this post and update ports soon.  The synth port maintainer already requested the update (as you could imagine).

I've done some testing and everything seems to work fine, but this version represents a LOT of new code so please report any strange behavior if you find any.

--  John


----------



## marino (Dec 3, 2017)

Oh, I forgot to mention another new behavior.
If a port features flavors, but you don't specify any specific favor, the framework picks the first listed flavor.  This is considered the default.
So lets take meld as an example.  In theory, you should be able to specify "textproc/meld" and synth will build it with the default flavor.  I decided to make the input really explicit, so Synth rejects flavored ports with no specified flavor.  Examples:

`synth synth force textproc/meld`

```
Error: port origin 'textproc/meld' is not recognized.
Perhaps you intended one of the following port flavors?
   - textproc/meld@py27
```

You will get suggestions if the flavor is missing or incorrect

`synth force accessibility/py-atspi@py28`

```
Error: port origin 'accessibility/py-atspi@py28' is not recognized.
Perhaps you intended one of the following port flavors?
   - accessibility/py-atspi@py27
   - accessibility/py-atspi@py36
```

This may cause some existing build lists to be modified.  pkg(8) will no longer provide good build lists out of the box


----------



## marino (Dec 3, 2017)

Hmm, I just realized that `synth status` (and related) is probably not going to work anymore.
The `pkg query %o` command only returns origins (without flavors).   This needs to be modified to have a per-package query for annotations to try to find flavors.  That will have to come in later versions.
For now, synth status is broken and you'll have to update manually.


----------



## Mayhem30 (Dec 3, 2017)

After doing all the updates, this shows up if I do a `synth prepare-system`

I believe most of these packages were just updated to the new py27 flavor. Other than that, everything works and looks great.


```
$ sudo synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scan of mail/py-pyspf failed, it will not be considered.
Scan of mail/py-authres failed, it will not be considered.
Scan of security/py-fail2ban failed, it will not be considered.
Scan of databases/py-sqlite3 failed, it will not be considered.
Scan of devel/py-pyinotify failed, it will not be considered.
Scan of devel/py-setuptools failed, it will not be considered.
Scan of mail/postfix-policyd-spf-python failed, it will not be considered.
Scan of devel/py-ipaddr failed, it will not be considered.
Scan of dns/py-dns failed, it will not be considered.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 142 ports serially.
Scan of mail/py-pyspf failed, it will not be considered.
Scan of devel/py-babel failed, it will not be considered.
Scan of mail/py-authres failed, it will not be considered.
Scan of textproc/py-docutils failed, it will not be considered.
Scan of security/py-fail2ban failed, it will not be considered.
Scan of textproc/py-pystemmer failed, it will not be considered.
Scan of devel/py-six failed, it will not be considered.
Scan of textproc/py-sphinx_rtd_theme failed, it will not be considered.
Scan of textproc/py-MarkupSafe failed, it will not be considered.
Scan of devel/py-pytz failed, it will not be considered.
Scan of databases/py-sqlite3 failed, it will not be considered.
Scan of devel/py-pyinotify failed, it will not be considered.
Scan of textproc/py-pygments failed, it will not be considered.
Scan of devel/py-setuptools failed, it will not be considered.
Scan of devel/scons failed, it will not be considered.
Scan of textproc/py-alabaster failed, it will not be considered.
Scan of devel/py-Jinja2 failed, it will not be considered.
Scan of textproc/py-sphinx failed, it will not be considered.
Scan of textproc/py-snowballstemmer failed, it will not be considered.
Scan of mail/postfix-policyd-spf-python failed, it will not be considered.
Scan of devel/py-ipaddr failed, it will not be considered.
Scan of dns/py-dns failed, it will not be considered.
Scan of graphics/py-imagesize failed, it will not be considered.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
```


----------



## rhsbsd (Dec 3, 2017)

Is pkg(8) going to take care of this 
	
	



```
DEFAULT_VERSIONS   +=   mysql=5.6 ssl=openssl
```
 in usr/local/etc/synth/LiveSystem-make.conf?


----------



## tobik@ (Dec 3, 2017)

marino said:


> Hopefully tobik@ will find this post and update ports soon. The synth port maintainer already requested the update (as you could imagine).


You know I would, but I see no PR or mail for it anywhere.

PR 224049


----------



## marino (Dec 3, 2017)

Mayhem30 said:


> After doing all the updates, this shows up if I do a `synth prepare-system`
> 
> I believe most of these packages were just updated to the new py27 flavor. Other than that, everything works and looks great.



As I said, `synth status` and related was anticipated to be broken.  `synth prepare-system` is just an extension of `synth status`.  It's also just a convenience command.  You can use a build list to build everything, including python (until it's fixed).
In fact, a build list is a better way to maintain the system.


----------



## marino (Dec 3, 2017)

tobik@ said:


> You know I would, but I see no PR or mail for it anywhere.
> 
> PR 224049


well, since it was a one-line change followed by `make makesum` (described at top), I didn't think a PR or patch was necessary.  I don't have access to bugzilla anyway.


----------



## marino (Dec 3, 2017)

rhsbsd said:


> Is pkg8 going to take care of this
> 
> 
> 
> ...



I will be reflected in the repository that you build, which pkg(8) respects (assuming you're not mixing repos or you turned off CONSERVATIVE_UPGRADE and put your local repository as the higher priority)


----------



## Snurg (Dec 3, 2017)

marino
OT: Yesterday I read a lot. It was very interesting to learn how you were thrown out of the FreeBSD team. Let me just say that I find it very sad how unoutspoken butthurt probably caused by narcissistic immaturities of some people led to the expulsion of an essential FreeBSD developer from the team.
So I am very glad that you seem mature and do not take immature of immature people personally and keep contributing to the FreeBSD community. (I refrain from going into detail, because this is obviously emotionally very sensitive to some people.)

Back to topic: As it looks like that my impression of Synth development being stopped was wrong, I will give Synth a try the next days.



marino said:


> pkg(8) will no longer provide good build lists out of the box


Thus I'd like to ask, is there a (simple) pkg command to rebuild all its lists to reflect the current system state etc if something goes wrong when playing with Synth?


----------



## nov1c3 (Dec 3, 2017)

marino said:


> Hmm, I just realized that `synth status` (and related) is probably not going to work anymore.
> The `pkg query %o` command only returns origins (without flavors).   This needs to be modified to have a per-package query for annotations to try to find flavors.  That will have to come in later versions.
> For now, synth status is broken and you'll have to update manually.



Thank you very much marino!

When you mention that you'll have to update manually, do you mean you have to 'synth just-build' port by port? Usually I feed the file with the list of ports to synth, but I get an error about port origin being not recognized:


```
# synth just-build /usr/local/etc/synth/ports.2.build
Error: port origin 'converters/libiconv' is not recognized.
```

Feeding converters/libiconv seems to work fine:


```
# synth just-build converters/libiconv
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
```


----------



## marino (Dec 3, 2017)

nov1c3 said:


> Thank you very much marino!
> When you mention that you'll have to update manually, do you mean you have to 'synth just-build' port by port? Usually I feed the file with the list of ports to synth, but I get an error about port origin being not recognized:



Hmm, that message about converters/libiconv not being recognized seems like a bug.  It if works via command line, it should work via file.  I'll try to reproduce locally while I work on v2.01


----------



## Mayhem30 (Dec 3, 2017)

Just a heads up, I'm having the same issue as nov1c3

```
$ sudo synth just-build build.list 
Error: port origin 'ports-mgmt/synth' is not recognized.
```


----------



## Eric A. Borisch (Dec 3, 2017)

And we're off and building. I had to use `synth just-build `cat build.list`` rather than `synth just-build build.list` -- otherwise I get the _port origin not recognized_ issue above -- and add flavors as appropriate, but now we're cooking. (And, for reference, `pkg query -e '%a = 0' %o > build.list` to populate the list to begin with...)

Thanks again for the great tool!!


----------



## rigoletto@ (Dec 3, 2017)

Eric A. Borisch

You could use `pkg prime-origins > build.list` instead of `pkg query -e '%a = 0' %o > build.list`.


----------



## marino (Dec 3, 2017)

lebarondemerde said:


> Eric A. Borisch
> 
> You could use `pkg prime-origins > build.list` instead of `pkg query -e '%a = 0' %o > build.list`.



Except that it won't work correctly anymore now that flavors is out.
To be clear, it won't work on flavored ports.  One origin, many ports. 
Since I decided to make the flavor explicit (rather than default to first flavor), the origin alone isn't enough anymore.


----------



## marino (Dec 4, 2017)

okay, I've got both bugs fixed locally.  They'll be included in v2.01.  I'll post when it's released.


----------



## marino (Dec 4, 2017)

Okay, I released version 2.01 on github just now.
Please let me know if flavor-related breakage is still showing up after upgrading to that version.


----------



## bhtooefr (Dec 4, 2017)

Thoughts on the flavored ports issue:

Could Synth, rather than skipping a flavored port that doesn't have a flavor specified, stop and ask which flavor to use?
Could Synth maintain a local database of historic flavor selections, so if it knows you want a certain flavor, use that flavor again unless otherwise specified?


----------



## marino (Dec 4, 2017)

bhtooefr said:


> Could Synth, rather than skipping a flavored port that doesn't have a flavor specified, stop and ask which flavor to use?



It doesn't skip now. It stops with an error message (and often a suggestion)



> Could Synth maintain a local database of historic flavor selections, so if it knows you want a certain flavor, use that flavor again unless otherwise specified?


no.  flavors must be explicitly specified.  anything else is asking for problems, especially when the previous selection changes (which it inevitably will over time).


----------



## Mayhem30 (Dec 4, 2017)

It's not too bad to rebuild an index on SSD's, but it sure takes a while on standard HDD. I realize that is beyond your control, but you'd think FreeBSD would have better tools in place so this wouldn't be necessary.

Is this how ports-mgmt/poudriere works as well to handle flavors? Building an index every time the port changes?


----------



## Mayhem30 (Dec 4, 2017)

Using Synth 2.01, this is still showing up each time `synth prepare-system` is run. Is this normal?


```
$ sudo synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 151 ports serially.
Scan of mail/py-pyspf failed, it will not be considered.
Scan of devel/py-babel failed, it will not be considered.
Scan of mail/py-authres failed, it will not be considered.
Scan of textproc/py-docutils failed, it will not be considered.
Scan of security/py-fail2ban failed, it will not be considered.
Scan of textproc/py-pystemmer failed, it will not be considered.
Scan of devel/py-six failed, it will not be considered.
Scan of textproc/py-sphinx_rtd_theme failed, it will not be considered.
Scan of textproc/py-MarkupSafe failed, it will not be considered.
Scan of devel/py-pytz failed, it will not be considered.
Scan of databases/py-sqlite3 failed, it will not be considered.
Scan of devel/py-pyinotify failed, it will not be considered.
Scan of textproc/py-pygments failed, it will not be considered.
Scan of devel/py-setuptools failed, it will not be considered.
Scan of devel/scons failed, it will not be considered.
Scan of textproc/py-alabaster failed, it will not be considered.
Scan of devel/py-Jinja2 failed, it will not be considered.
Scan of textproc/py-sphinx failed, it will not be considered.
Scan of textproc/py-snowballstemmer failed, it will not be considered.
Scan of mail/postfix-policyd-spf-python failed, it will not be considered.
Scan of devel/py-ipaddr failed, it will not be considered.
Scan of dns/py-dns failed, it will not be considered.
Scan of graphics/py-imagesize failed, it will not be considered.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
```


----------



## marino (Dec 4, 2017)

are you sure you're using v2.01?  try `synth` to verify


----------



## Mayhem30 (Dec 4, 2017)

```
$ synth

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 2.01
====================================================================
               Copyright (C) 2015-2017 John R. Marino

----

synth-2.01                         =   up-to-date with index
```


----------



## marino (Dec 4, 2017)

try `synth status` instead, and look in the logs directory for 06* log for errors.
What you're showing is not normal.  Without evidence, I'd suspect you've got something in your profile's make.conf that's breaking things.


----------



## Mayhem30 (Dec 4, 2017)

Ok, that seems to work just fine.

```
$ sudo synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/tmp/synth_status_results.txt
```
/usr/local/etc/synth/LiveSystem-make.conf

```
OPTIONS_UNSET = X11 CUPS

WITH_POSTFIX = yes

DEFAULT_VERSIONS += ssl=openssl php=7.1 mysql=5.6
```
/var/log/synth/06_obsolete_packages.log is empty


----------



## scottro (Dec 4, 2017)

Working for me. That is, I didn't get all the python related package errors, and I see it's already built some of them, for example getmail@py27


----------



## marino (Dec 4, 2017)

ah, it looks like the repository rebuilding is what is emitting the errors.  hang on.


----------



## marino (Dec 4, 2017)

I think I found the issue.  I'll have to issue v2.02 though, after I reproduce and test.


----------



## PacketMan (Dec 4, 2017)

marino said:


> `synth prepare-system` is just an extension of `synth status`.  It's also just a convenience command.



Maybe, but I see it more than that John and I use as my primary update command. Then when all has built successfully and I am ready to actually update my programs I then do my update.  Anyway,  thanks a bunch for your effort with this, it will likely be a few weeks before I do any updating on my main servers, but I do have a couple lab servers if you need extra 'eyes' to help test something.


----------



## rigoletto@ (Dec 4, 2017)

ports-mgmt/pkg just received some FLAVOR related fixes.


----------



## marino (Dec 5, 2017)

Getting close to releasing 2.02, but I need to finish testing locally.  One typo caused about 350 current packages in my repository to get deleted, so I'm building them back up.  However, I'm hopeful the current synth code is good.

If anyone is feeling adventurous, they can try it in the meantime.  I'd love the feedback.
to build, add the line
`GH_TAGNAME= f751b76`
to /usr/ports/ports-mgmt/synth/Makefile

then `cd /usr/ports/ports-mgmt/synth && make makesum`
then `make clean deinstall`
then `make install`


To clarify, this version tries to fix the `synth rebuild-repository` and `synth prepare-system` and similar commands.  The flavor stuff really threw a wrench in the works on those.


----------



## cvv (Dec 5, 2017)

marino said:


> Getting close to releasing 2.02, but I need to finish testing locally.  One typo caused about 350 current packages in my repository to get deleted, so I'm building them back up.  However, I'm hopeful the current synth code is good.
> 
> If anyone is feeling adventurous, they can try it in the meantime.  I'd love the feedback.
> to build, add the line
> ...



Tried it out on my system:


```
# synth rebuild-repository
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Stand by, building pkg(8) first ... done!
Stand by, prescanning existing packages.
Removed: pkg-1.10.3.txz
Stand by, recursively scanning 139 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Note: This system's pkg(8) is configured with CONSERVATIVE_UPGRADE = true
      You may wish to toggle that setting if this is a local repository.

root@forklift:~ # synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt

root@forklift:~ # synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 139 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
```
I'm not seeing flavor-related errors anymore. I'll play with it some more, but it looks okay so far.


----------



## Mayhem30 (Dec 6, 2017)

I'm not able to test this patch until it goes live.

The issue is happening on my production server, and I can't have the system installing all the Synth dependencies when the resources are needed elsewhere.


----------



## marino (Dec 6, 2017)

Mayhem30 said:


> I'm not able to test this patch until it goes live.
> 
> The issue is happening on my production server, and I can't have the system installing all the Synth dependencies when the resources are needed elsewhere.



I'm not sure what you mean.  It takes the same amount of resources to build a release as a git tag.  Unless you mean you have a wait a week for it to appear on FreeBSD official package servers ...


----------



## marino (Dec 6, 2017)

I pushed release 2.02.  It seems to be behaving with respect to rebuilding repositories and installing from synth.


----------



## Mayhem30 (Dec 6, 2017)

marino said:


> I'm not sure what you mean.  It takes the same amount of resources to build a release as a git tag.  Unless you mean you have a wait a week for it to appear on FreeBSD official package servers ...



When I don't use synth to install or upgrade a port, it has compile all the dependencies because they only exist in the repo - which is not used during `make install`. Then all of them are installed - whether or not they are "build only".

It was just using a fair amount of resources .. and a bit of work to clean everything up afterwards.


----------



## marino (Dec 6, 2017)

well, 2.01 synth can build 2.02 synth package, and you can install packages directly with `pkg add` and `pkg remove` commands.  So you don't have to build outside a jail.


----------



## PacketMan (Dec 6, 2017)

marino said:


> I pushed release 2.02.  It seems to be behaving with respect to rebuilding repositories and installing from synth.



Thanks John, I'll try to 'pound' this soon.


----------



## marino (Dec 6, 2017)

Mayhem30 said:


> It's not too bad to rebuild an index on SSD's, but it sure takes a while on standard HDD. I realize that is beyond your control, but you'd think FreeBSD would have better tools in place so this wouldn't be necessary.
> 
> Is this how ports-mgmt/poudriere works as well to handle flavors? Building an index every time the port changes?



well, it's not every time a port changes.  It's every time the ports tree changes.
poudriere doesn't do this.  The index is essentially a cache.  Synth does it once, then looks up the data.  Poudriere determines it every time.  The additional scan time is repeated on a limited number of ports (as opposed to the whole tree) every time poudriere is launched.  Synth is going to perform a lot fast during the actual building.

If freebsd actually provided an equivalent index with the tree, then it wouldn't be necessary.
There's no way of knowing how many potential packages there are without a full scan.  Remember, there's no longer a 1:1 ratio between packages and ports.  The only way to figure out how many packages per port there is is to do a VERY lengthy per port scan.

Incidently, see http://www.ravenports.com.
Ravenports does a full tree scan in less than 1 second.
Extrapolating to a tree the size of freebsd, I think ravenports could scan the tree in about 2, maybe 3 seconds.
Ravenports blows freebsd ports away in areas like these (and freebsd blows pkgsrc away in the same areas)


----------



## marino (Dec 6, 2017)

marino said:


> Ravenports does a full tree scan in less than 1 second.
> Extrapolating to a tree the size of freebsd, I think ravenports could scan the tree in about 2, maybe 3 seconds.
> Ravenports blows freebsd ports away in areas like these (and freebsd blows pkgsrc away in the same areas)



By the way, the ravenports scan is done serially (one port at a time) while the freebsd ports scan is done in parallel (e.g. 12 builders at a time on a 4-core system).  Without the parallel scanning, the scan would take like 10 times longer (conversely if the ravenports scan was done in parallel, it would cut 90% off the < 1 second time).


----------



## Mayhem30 (Dec 6, 2017)

This is what I'm seeing with the new version this morning.

```
$ synth

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 2.02
====================================================================
               Copyright (C) 2015-2017 John R. Marino

$ sudo synth status
Querying system about current package installations.
Installed package ignored, mail/py-authres package unmatched
Installed package ignored, dns/py-dns package unmatched
Installed package ignored, security/py-fail2ban package unmatched
Installed package ignored, devel/py-ipaddr package unmatched
Installed package ignored, mail/postfix-policyd-spf-python package unmatched
Installed package ignored, devel/py-pyinotify package unmatched
Installed package ignored, mail/py-pyspf package unmatched
Installed package ignored, devel/py-setuptools package unmatched
Installed package ignored, databases/py-sqlite3 package unmatched
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt


$ sudo synth prepare-system
Querying system about current package installations.
Installed package ignored, mail/py-authres package unmatched
Installed package ignored, dns/py-dns package unmatched
Installed package ignored, security/py-fail2ban package unmatched
Installed package ignored, devel/py-ipaddr package unmatched
Installed package ignored, mail/postfix-policyd-spf-python package unmatched
Installed package ignored, devel/py-pyinotify package unmatched
Installed package ignored, mail/py-pyspf package unmatched
Installed package ignored, devel/py-setuptools package unmatched
Installed package ignored, databases/py-sqlite3 package unmatched
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 142 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
```


----------



## marino (Dec 6, 2017)

mayhem30@
I think this is fallout from flavors.
What you need to do follow the bulletproof technique outlined here:
https://www.dragonflybsd.org/docs/howtos/HowToDPorts/#index4h1

You may have to replace some origins from your list with actual package names if you're needed non-default flavors.

Once you replace all packages with those that you've built, the system query errors should go away.


----------



## marino (Dec 6, 2017)

in other words, the unmatched packages were installed from pre-flavors, so there are no flavor annotations provided with the internal "pkg query" commands that synth is issuing.


----------



## Mayhem30 (Dec 6, 2017)

Ok thanks, I will give that a go sometime later today and report back.


----------



## Mayhem30 (Dec 6, 2017)

Thank you, removing / installing all the ports solved the issue.

```
$ sudo synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt

$ sudo synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, prescanning existing packages.
Stand by, recursively scanning 142 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
```


----------



## driesm (Dec 7, 2017)

John marino, I'm having no trouble building flavored ports, for example; net/py-speedtest-cli@py36 builds all dependencies with python3.6 ports. However when I want to build a non flavored port like net/samba46 that depends on python against python3.6 it doesn't work?
I have this in LiveSystem-make.conf:

```
DEFAULT_VERSIONS+=python=3.6
```
When I use `synth status build.list`
I get this:

```
N => devel/py-iso8601@py27
N => dns/py-dnspython@py27
```
Which are dependencies of samba. Are the DEFAULT_VERSIONS entries ignored now that flavors are here?


----------



## marino (Dec 7, 2017)

doubtful.  I suspect the py27 versions are specified.  The "default" is used when no version is specified.  However, a good deal of s/w doesn't build with python higher than 2.7, so for those, python 2.7 has to be required (and for those, the requires version of python ignored).   Looks normal to me.  You're going to have multiple versions of python modules installed.


----------



## PacketMan (Dec 8, 2017)

So using my home 'play' server. I deleted all the packages that were previously in here:

```
$ ls -lhG /var/synth/live_packages/All
total 0
```

...then....


```
# synth

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 2.02
====================================================================
               Copyright (C) 2015-2017 John R. Marino


# synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Querying system about current package installations.
Installed package ignored, graphics/py-pillow package unmatched
Installed package ignored, missing from ports: devel/py27-setuptools
Installed package ignored, x11-toolkits/py-tkinter package unmatched
Stand by, comparing installed packages against the ports tree.
Stand by, building pkg(8) first ... done!
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
These are the packages that would be fetched:
```

Removed / reinstalled 'prime' packages (see post from John).


```
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
These are the packages that would be fetched:
  => ccache-3.3.4_8.txz (devel/ccache)
  => indexinfo-0.3.1.txz (print/indexinfo)
  => gettext-runtime-0.19.8.1_1.txz (devel/gettext-runtime)
  => gettext-tools-0.19.8.1.txz (devel/gettext-tools)
  => pkgconf-1.3.10,1.txz (devel/pkgconf)
  => gmake-4.2.1_1.txz (devel/gmake)
<more>


# synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
The following packages will be fetched:

New packages to be FETCHED:
   ccache-3.3.4_8 (95 KiB: 0.00% of the 5 GiB to download)
   indexinfo-0.3.1 (6 KiB: 0.00% of the 5 GiB to download)
   gettext-runtime-0.19.8.1_1 (147 KiB: 0.00% of the 5 GiB to download)
   gettext-tools-0.19.8.1 (2 MiB: 0.04% of the 5 GiB to download)
   pkgconf-1.3.10,1 (51 KiB: 0.00% of the 5 GiB to download)
   gmake-4.2.1_1 (378 KiB: 0.01% of the 5 GiB to download)
<more>

..then eventually
The task is complete.  Final tally:
Initial queue size: 6
    packages built: 6
           ignored: 0
           skipped: 0
            failed: 0

Duration: 00:08:55
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 259 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
```

The "regenerating flavor index" on my machine takes about 10 minutes.


----------



## marino (Dec 8, 2017)

ouch.  There is low and moderate hanging fruit that could easily speed the scans up 5 fold (e.g. 2 minutes rather than 10) but so far port manager hasn't seen the current implementations as a problem.  sorry; I can't fix ports here.


----------



## PacketMan (Dec 8, 2017)

No worries John, I'm just thankful that you made Synth.  My 'play' server is a Dell PowerEdge 1950 (2x quad-core 2.66GHz Xeon E5430 Harpertown processors, 16 GB RAM, 2x 143GB SAS hard drives).  I am researching hardware (and trying to convince my wife) to let me build a brand new box using Intel Core i5-8400 but even that will have only a moderate gain over dual E5430 Harpertown processors. I use the same server (a 2nd one actually) with net-p2p/btsync to do the building for my slower machines.

Thanks for your effort with this FLAVORS thing; I'm far from sold on it, but it's early days. Wish there was more I could read up on it, but no one has offered any URLs yet.

Thanks again.


----------



## Mayhem30 (Dec 8, 2017)

This is the only URL I'm aware of :

http://www.at.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/flavors.html


----------



## PacketMan (Dec 8, 2017)

Yeah Mayhem30 I already posted that link in Thread 63509. Thanks just the same.


----------



## Mayhem30 (Dec 14, 2017)

Just saw that ports-mgmt/portmaster was updated and now supports flavors. I see that portmaster checks the /usr/ports/MOVED file to work things out.

Couldn't Synth use that file instead of having to build it's own index? (which is quite time consuming).

```
devel/py34-setuptools|devel/py-setuptools@py34|2017-11-30|Moved to a flavored, generic, version
devel/py35-setuptools|devel/py-setuptools@py35|2017-11-30|Moved to a flavored, generic, version
devel/py36-setuptools|devel/py-setuptools@py36|2017-11-30|Moved to a flavored, generic, version
```


----------



## marino (Dec 14, 2017)

no.


----------



## marino (Dec 15, 2017)

For people that this index generations seems like a problem, I suggest doing the following:

Write a small shell script that first updates the ports tree, then executes `synth status`.  Update cron to run that script at like 0500 each morning.  Then your machine will have a current index every time you need to use it.

If somebody wants to publish such a script and post it here, maybe that would help others.


----------



## Mayhem30 (Dec 15, 2017)

That's exactly what I've done. I have cron set to run `portsnap cron update` at 5AM - and then at 6:30AM it runs my script.

Note that `portsnap cron update` sleeps a random amount of time between 1 and 3600 seconds. So you need to wait a minimum of 1 hour after it fires before running the script.

```
0  5 * * * /usr/sbin/portsnap cron update >/dev/null 2>&1
30  6 * * * /bin/echo y | /home/check-updates.sh
```
When running the script in cron, you need to add "/bin/echo y" to tell synth to rebuild the index (if prompted), otherwise it will hang and do nothing.

If updates are available, I "echo" it so it emails me the results. If no updates are available, it echo's nothing and then exits.

check-updates.sh

```
#!/usr/local/bin/bash

read UPDATES <<< $(/usr/local/bin/synth status | awk '/Total packages that would be built:/ { print $7 }')

if [ "$UPDATES" -eq "0" ]; then
   exit;
fi

if [ "$UPDATES" -eq "1" ]; then
   UPDSTR="package update is"
else
   UPDSTR="package updates are"
fi

echo "Synth status: $UPDATES $UPDSTR available for download"
```
I hope this helps someone, as it took me a bit to figure it all out.


----------



## marino (Dec 15, 2017)

i would have added the "portsnap" command to the check-updates.sh script so that you don't have to time it. 
Or in other words, the second part doesn't run if the first part fails.


----------



## Eric A. Borisch (Dec 15, 2017)

Some tweaks. Remove the portsnap cron line from crontab, and just run this every morning.


```
#!/bin/sh

portsnap auto > /dev/null || exit 1

[ -x /usr/local/bin/synth ] || { echo "Error: synth not installed?"; exit 1 ; }

SYNOUT=$(echo y | /usr/local/bin/synth status)

BLDS=$(echo -e "${SYNOUT}" | awk '/Tot.*built:/{print $7}')
DLS=$(echo -e "${SYNOUT}" | awk '/Tot.*fetched:/{print $7}')

[ "$BLDS" = "0" ] && [ -z "$DLS" ] && exit

[ -z "$DLS" ] || DLSTAT=" ${DLS} to download,"

echo "Synth status:${DLSTAT} $BLDS to build."
```


----------



## Mayhem30 (Dec 16, 2017)

`portsnap auto` is not valid command on FreeBSD 10.4

```
$ sudo portsnap auto
usage: portsnap [options] command ... [path]

Options:
 ...
 ...
 ...
Commands:
  fetch        -- Fetch a compressed snapshot of the ports tree,
                  or update an existing snapshot.
  cron         -- Sleep rand(3600) seconds, and then fetch updates.
  extract      -- Extract snapshot of ports tree, replacing existing
                  files and directories.
  update       -- Update ports tree to match current snapshot, replacing
                  files and directories which have changed.
```


----------



## Eric A. Borisch (Dec 16, 2017)

Mayhem30 said:


> `portsnap auto` is not valid command on FreeBSD 10.4
> 
> ```
> $ sudo portsnap auto
> ...



True. All my systems are on 11.1-p*. Adapt as needed. 

[edit]
Fixed typo in version #.
For those with 11, “auto” is just what you would hope: “Run *fetch* or *cron* depending on whether stdin is a terminal; then run *update* or *extract* depending on whether _portsdir_ exists.”


----------



## Mayhem30 (Dec 16, 2017)

Marino, if you have a minute could you look at this thread? I'm having an issue with Synth after upgrading to FreeBSD 11.1

https://forums.freebsd.org/threads/63731/


----------



## Cthulhux (Dec 17, 2017)

On one of my machines, I have an auto-update cronjob script:


```
# ...
export TERM=xterm
# ...
echo "***" | tee -a ${LOG_FILE}
echo "*** Looking for ports to update..." | tee -a ${LOG_FILE}
echo "***" | tee -a ${LOG_FILE}
/usr/local/bin/synth rebuild-repository | tee -a ${LOG_FILE}
/usr/local/bin/synth upgrade-system | tee -a ${LOG_FILE}
/usr/local/bin/synth purge-distfiles | tee -a ${LOG_FILE}
```

However, Synth just halts:



> ***
> *** Looking for ports to update...
> ***
> Synth is already running on this system.
> ...



Executing the commands in the file manually does not have this effect, I am not prompted for anything either. What am I missing?


----------



## marino (Dec 17, 2017)

That's a dangerous script.  You should limit this to `synth prepare-system`.
Plus it doesn't do what I think you want it to do.
"Rebuild repository" doesn't build anything, so "upgrade" isn't going to do anything.
But if you meant to put `synth prepare-system` instead of `synth rebuild-repository` I would not recommend that.
If there's a build failure and the full ports build doesn't work, the system upgrade will be partial at best.  System upgrades should always be manual in my option.

As for the actual question, there's a active synth pid somewhere (at least there was).  I think the messages are legit at the time of the cron job.  When you try it later, there wasn't any instance of synth running.


----------



## Cthulhux (Dec 17, 2017)

Thank you - I have misunderstood the README, probably.
But why are there suddenly active synth PIDs every three days (the cronjob time) without me starting any? I thought that maybe synth doesn't find itself, it just made no sense to me.

Yes, system upgrades _should_ be manual and I actually _do_ them manually on all non-playground servers. 
But I'm lazy sometimes.


----------



## marino (Dec 17, 2017)

evidence suggests another cron job is starting synth up at the same time.  Or your script is forking and racing.  but something is going on and the error messages are most likely real.


----------



## Eric A. Borisch (Dec 18, 2017)

For the record, here's where I ended, which seems to be working when called via cron: updates portfiles, rebuild synth flavors index, report status back to cron via stdout...

(For 10.x, change 'portsnap auto' to two lines with 'portsnap cron' and 'portsnap update'...)


```
#!/bin/sh

# Synth requires TERM to be set
export TERM=xterm

portsnap auto > /dev/null || exit 1

[ -x /usr/local/bin/synth ] || \
    { echo "Error: synth not installed?"; exit 1 ; }

SYNOUT=$(echo y | /usr/local/bin/synth status)

BLDS=$(echo -e "${SYNOUT}" | awk '/Tot.*built:/{print $7}')
DLS=$(echo -e "${SYNOUT}" | awk '/Tot.*fetched:/{print $7}')

[ "$BLDS" = "0" ] && [ -z "$DLS" ] && exit

[ -z "$DLS" ] || DLSTAT=" ${DLS} to download,"

echo "Synth status:${DLSTAT} $BLDS to build."
```


----------



## Mayhem30 (Dec 18, 2017)

What is this "Total fetched" line the code is trying to scrape? I have never seen it in any Synth output.

```
DLS=$(echo -e "${SYNOUT}" | awk '/Tot.*fetched:/{print $7}')
```


----------



## Eric A. Borisch (Dec 18, 2017)

Mayhem30 said:


> What is this "Total fetched" line the code is trying to scrape? I have never seen it in any Synth output.
> 
> ```
> DLS=$(echo -e "${SYNOUT}" | awk '/Tot.*fetched:/{print $7}')
> ```



If you have ‘leverage pre-built packages’ enabled in synth.


----------



## Mayhem30 (Dec 19, 2017)

Eric A. Borisch said:


> If you have ‘leverage pre-built packages’ enabled in synth.



Ok, that is one option that I'm still not clear about. When enabled, providing my port does not have custom options, it will grab the pre-build package (if available) from an external repo?

What file do I need to define the external repo in?


----------



## Eric A. Borisch (Dec 19, 2017)

Mayhem30 said:


> Ok, that is one option that I'm still not clear about. When enabled, providing my port does not have custom options, it will grab the pre-build package (if available) from an external repo?
> 
> What file do I need to define the external repo in?



In such a way the pkg is aware of them.

See synth(1) for the description on when a pre-compiled port is considered usable. It’s my favorite part of synth over “the other builder.” Only those ports where I need a non-standard option (and ports depending on them) need to be built. Most of the ports where I need a non-standard option are high on the dependency tree (apps, rather than libs) so many ports can be fetched.


----------



## Mayhem30 (Dec 19, 2017)

Ok, so only 14 of my ports will have to be built, and the other 130 (if available and considered usable) will use a pre-built package.

Why is this option not enabled by default?


----------



## Mayhem30 (Dec 19, 2017)

Synth appears to have crashed this morning.

```
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.

culprit: databases/postgis24
Scan aborted because 'make' encounted an error in the Makefile.
databases/postgis24 (return code = 1)
```
Also, it left all of these behind :

```
$ df -h
Filesystem                  Size    Used   Avail Capacity  Mounted on
/dev/mirror/gm0s1a          105G    7.2G     89G     7%    /
devfs                       1.0K    1.0K      0B   100%    /dev
tmpfs                        13G    228K     13G     0%    /usr/obj/synth-live/SL09
/bin                        105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/bin
/sbin                       105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/sbin
/lib                        105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/lib
/libexec                    105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/libexec
/usr/bin                    105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/bin
/usr/include                105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/include
/usr/lib                    105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/lib
/usr/libdata                105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/libdata
/usr/libexec                105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/libexec
/usr/sbin                   105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/sbin
/usr/share                  105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/share
/usr/ports                  105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/xports
/var/db/ports               105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/options
/var/synth/live_packages    105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/packages
/usr/ports/distfiles        105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/distfiles
tmpfs                        12G    4.0K     12G     0%    /usr/obj/synth-live/SL09/construction
tmpfs                        12G    4.0K     12G     0%    /usr/obj/synth-live/SL09/usr/local
/usr/lib32                  105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/lib32
/boot                       105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/boot
tmpfs                       100M    4.0K    100M     0%    /usr/obj/synth-live/SL09/boot/modules
/usr/games                  105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/games
/usr/src                    105G    7.2G     89G     7%    /usr/obj/synth-live/SL09/usr/src
devfs                       1.0K    1.0K      0B   100%    /usr/obj/synth-live/SL09/dev
```
Do I need to `umount` all of them manually?


----------



## marino (Dec 19, 2017)

no. any synth command, e.g. `synth configure` cleans them up.

It didn't crash.  The port tree is bad, so it stopped (as it should).


----------



## Mayhem30 (Dec 19, 2017)

Sorry, poor choice of words. Thank you - all is well again.


----------



## Mayhem30 (Dec 20, 2017)

Any ideas why PKG wants to install (2) build-only dependencies?

These files are currently not installed, but do reside in the /var/synth/live_packages/All directory and are up to date with the latest versions.

```
$ sudo pkg upgrade      
Updating Synth repository catalogue...
Synth repository is up to date.
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking for upgrades (53 candidates): 100%
Processing candidates (53 candidates): 100%
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        gmp: 6.1.2 [FreeBSD]
        mpfr: 3.1.6 [FreeBSD]

Number of packages to be installed: 2

The process will require 4 MiB more space.
799 KiB to be downloaded.

Proceed with this action? [y/N]:
```
If I disable the FreeBSD repo, everything is showing up to date.

```
$ sudo pkg upgrade      
Updating Synth repository catalogue...
Synth repository is up to date.
All repositories are up to date.
Checking for upgrades (1 candidates): 100%
Processing candidates (1 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.

$ sudo synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt
```


----------



## marino (Dec 20, 2017)

just a typical mismatch between a local repository and the official freebsd repository which is normally 3-7 days behind.
Unless you're leveraging the official packages to avoid building everything locally, i'd stick to the local repository only.


----------



## Mayhem30 (Jan 16, 2018)

Does anything need to be done on my need or I just have to wait it out?

```
$ sudo synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
 progress: 63.08%            
culprit: net/unison-nox11
Scan aborted because 'make' encounted an error in the Makefile.
net/unison-nox11 (return code = 1)
```
Also, it was a good time to test the cleanup feature - which worked great.

```
$ sudo synth configure
Builder mounts detected; attempting to remove them automatically ...
Dismounting successful!
```


----------



## marino (Jan 16, 2018)

Wasn't the same thing posted to the freebsd ports list (which I don't suscribe to)?

As mentioned in my response, either the port net/unison-nox11 is busted (likely) or you've got something in your <profile>-make.conf file that is causing it to be busted.  In the first case, `make -C /usr/ports/net/unison-nox11 -V PORTNAME` should immediately show the broken makefile errors.

Bottom line: This is not a synth error.  It never is.  "culprit" means the ports tree is at fault (or your make.conf is).


----------



## marino (Jan 16, 2018)

okay, I clicked on the link.  net/unison-nox11 has been deleted.
You're using the `synth status`.  On your system, you have the deleted port installed.  If you want `pkg(8)` to stop telling synth to build a deleted port, deinstall that port from your system.  

But net/unison-nox11 shouldn't be on the flavor index, so it should be ignored anyway.  Is your port tree up to date and has the flavor index been regenerated?


----------



## tobik@ (Jan 16, 2018)

net/unison-nox11 was deleted, but is still hooked up in net/Makefile... I'll remove it.


----------



## marino (Jan 16, 2018)

tobik@ said:


> net/unison-nox11 was deleted, but is still hooked up in net/Makefile... I'll remove it.


Yep, that would do it.  madpilot didn't delete the port correctly (fixing the category makefile is mandatory when removing a port as documented in the handbook)


P.S. maybe you can close out the topic on the freebsd ports mail list.


----------



## tobik@ (Jan 16, 2018)

Should be fixed now: https://svnweb.freebsd.org/changeset/ports/459179


----------



## Mayhem30 (Jan 17, 2018)

Everything is working great now.


----------



## loki2012 (Feb 17, 2018)

First, thank you so much Marino for developing synth, it is such a great tool!

I am running into a situation I am not sure is normal. I am trying to install shells/scponly with unison selected in the config, but I want unison to be build with nox11 flavor. So I specified it in the config files:


```
$ cat /usr/local/etc/synth/LiveSystem-make.conf

DEFAULT_VERSIONS+=unison=nox11
```

but `synth install shells/scponly` will still install the default flavor of net/unison which is with x11. Anything am I missing here?


----------



## marino (Feb 17, 2018)

loki2012, you would specify "net/unison@nox11" to build it, e.g. `synth build net/unison@nox11`
I'm not sure `synth install net/unison@nox11` works though.  You can try it and let us know.
In fact, I don't think synth even accepts `synth install net/unison`.  It should give you an error that it needs a flavor.  Are you sure you using version 2.02 ?

edit: hold on, you're talking about scponly.  let me look for a second.


----------



## marino (Feb 17, 2018)

It's the fault of scponly.  It's specifying net/unison@x11 (it's specifying net/unison, but without the flavor specification, so it defaults to x11).
It's not synth's fault.  You can modify the scponly makefile to add "@nox11" to the makefile.  That's the only way to change it.


----------



## loki2012 (Feb 18, 2018)

Marino, thanks for having looked at that.

Actually I initially thought setting `DEFAULT_VERSIONS+=unison=nox11` in /etc/make.conf and building manually the port shells/scponly would build unison@nox11, but further testing indicates it is not the case, it still builds unison@x11 (the default flavor). So indeed it is not a `synth` related issue.

I am still trying to figure out how to define the default flavor of a port in /etc/make.conf. Once I get it working, I suspect updating /usr/local/etc/synth/LiveSystem-make.conf will also work.


----------



## marino (Feb 18, 2018)

the default flavor is the first one listed, by definition.
Ravenports (which has had variants a year before flavors came out) doesn't have "defaults".  You specify the exact variant.  FreeBSD ports does it like it does because they didn't want to fix hundreds if not thousands of ports when the feature was introduced.  I don't like the design decision on defaults myself.


----------



## irukandji (Feb 20, 2018)

I am having an issue since the beginning of the flavors comming in into ports tree, synth always begins with 

```
Regenerating flavor index: this may take a while ...                                                                                                                                                                                     Scanning entire ports tree.                                                                                                                                                                                                                                           Querying system about current package installations.                                                                                                                                                                                           Installed package ignored, devel/py-Automat package unmatched                                                                                                                                                                     Installed package ignored, devel/py-constantly package unmatched                                                                                                                                                                 Installed package ignored, www/py-hyperlink package unmatched                                                                                                                                                                     Installed package ignored, devel/py-incremental package unmatched                                                                                                                                                               Installed package ignored, security/py-pycrypto package unmatched                                                                                                                                                                 Installed package ignored, mail/pyzor package unmatched                                                                                                                                                                                 Installed package ignored, devel/py-six package unmatched
Installed package ignored, databases/py-sqlite3 package unmatched                                                                                                                                                               Installed package ignored, devel/py-twisted package unmatched                                                                                                                                                                       Installed package ignored, devel/py-zope.interface package unmatched                                                                                                                                                           Installed package ignored, missing from ports: security/py3-certifi                                                                                                                                                                     Installed package ignored, missing from ports: devel/py3-cffi                                                                                                                                                                             Installed package ignored, missing from ports: textproc/py3-chardet                                                                                                                                                                 Installed package ignored, missing from ports: devel/py3-click                                                                                                                                                                           Installed package ignored, missing from ports: devel/py3-coloredlogs                                                                                                                                                               Installed package ignored, missing from ports: security/py3-cryptography                                                                                                                                                       Installed package ignored, missing from ports: textproc/py3-humanfriendly                                                                                                                                                     Installed package ignored, missing from ports: dns/py3-idna                                                                                                                                                                               Installed package ignored, missing from ports: devel/py3-libzfs                                                                                                                                                                         Installed package ignored, missing from ports: security/py3-openssl                                                                                                                                                                 Installed package ignored, missing from ports: devel/py3-pyasn1                                                                                                                                                                       Installed package ignored, missing from ports: devel/py3-pycparser                                                                                                                                                                   Installed package ignored, missing from ports: net/py3-pysocks                                                                                                                                                                         Installed package ignored, missing from ports: devel/py3-pytest-runner
Installed package ignored, missing from ports: www/py3-requests
Installed package ignored, missing from ports: devel/py3-six
Installed package ignored, missing from ports: textproc/py3-texttable
Installed package ignored, missing from ports: misc/py3-tqdm
Installed package ignored, missing from ports: net/py3-urllib3
Installed package ignored, missing from ports: devel/py3-verboselogs
```

And finishes with:


```
Error: port origin 'devel/py-pip' is not recognized.
Perhaps you intended one of the following port flavors?
   - devel/py-pip@py27
   - devel/py-pip@py36
```

I dont know how to fix it


----------



## PacketMan (Feb 20, 2018)

Did you run a version of Synth previous to version 2.02 ?  If so you really should do the process that John outlined earlier.  Found here:  Synth Bullet-proof (conflict-proof) upgrade technique  Note its written for DragonFlyBSD but you should be able to see what needs to be done.

If you are a new synth user and have never used a version previous to 2.02 then I'm not sure what the issue might be.  You could try following that process for fun, but I wouldn't expect it to make a difference.


----------



## marino (Feb 20, 2018)

It's caused by having very old packages (predating flavors) still installed on the system.  Removing all packages and reinstalling newly built ones (as would happen with the bullet-proof technique) will indeed fix that.  

The problem is the prepare-system command, which uses pkg(8) to get information about the system by reading the packages.  If you don't want to install freebsd.org packages to solve this, you need to come up with a manual build list (as seen in bullet-proof technique).


----------



## irukandji (Feb 20, 2018)

marino said:


> It's caused by having very old packages (predating flavors) still installed on the system.  Removing all packages and reinstalling newly built ones (as would happen with the bullet-proof technique) will indeed fix that.
> 
> The problem is the prepare-system command, which uses pkg(8) to get information about the system by reading the packages.  If you don't want to install freebsd.org packages to solve this, you need to come up with a manual build list (as seen in bullet-proof technique).



Is there any way to just delete the packages that are old? I seriously wouldnt like to leave my server building packages for next two weeks...


----------



## PacketMan (Feb 20, 2018)

Synth will build what needs to built, and if prefetch option is enabled, it will prefetch packages when it can.  That process does just that; deletes old packages (well actually it deletes them all), and a few other things.  If its a 'lab' machine I suppose you could try deleting some stuff, but experience tells me that could make more troubles for you.  Its actually one of the things I like about FreeBSD, it requires more discipline but its worth it.


----------



## marino (Feb 20, 2018)

irukandji said:


> Is there any way to just delete the packages that are old? I seriously wouldnt like to leave my server building packages for next two weeks...



Then use the official freebsd packages that already exist.
You really should change out all the packages at once.  If you change just one or two key packages, it will try to uninstall everything anyway.  Don't take shortcuts and do it right.


----------



## irukandji (Feb 21, 2018)

Hmmm, ok, with shaking hands i will do it, just one more question, how the fetching works, where does it take official repo from? Currently I have /etc/pkg/FreeBSD.conf disabled and additional one LocalSynth.conf set to /var/synth/live_packages/

In case of synth fetching the binaries the FreeBSD.conf needs to be enabled or there is an additional config file just for synth?


----------



## marino (Feb 21, 2018)

see `man pkg.conf`
Re-enable the freebsd repository and disable the synth repository.  
You can reverse that later.

Basically everyone really needed to reinstall all packages after flavors came out because flavors are annotated in package notes.  Old packages don't have those annotations.


----------



## jrm@ (Feb 22, 2018)

loki2012 said:


> I am still trying to figure out how to define the default flavor of a port in /etc/make.conf.


Not ideal, but putting this in make.conf should work.

```
.if ${.CURDIR:M*/net/unison*}
  FLAVOR=nox11
.endif
```


----------



## irukandji (Feb 23, 2018)

PacketMan said:


> Did you run a version of Synth previous to version 2.02 ?  If so you really should do the process that John outlined earlier.  Found here:  Synth Bullet-proof (conflict-proof) upgrade technique  Note its written for DragonFlyBSD but you should be able to see what needs to be done.
> 
> If you are a new synth user and have never used a version previous to 2.02 then I'm not sure what the issue might be.  You could try following that process for fun, but I wouldn't expect it to make a difference.



Hmm, been studying this a bit... and i am really grateful for zfs on root... =/

- get list of prime packages
- wipe out all packages from the system ()
- reinstall packages
- probably same procedure for jails

The only thing i dont understand is where does synth come into play? I need to wipe /var/synth/live_packages/ too and what then, will the list of packages being used stay intact or i need to provide prime list back to synth using synth install?


----------



## PacketMan (Feb 23, 2018)

irukandji said:


> The only thing i dont understand is where does synth come into play? I need to wipe /var/synth/live_packages/ too and what then, will the list of packages being used stay intact or i need to provide prime list back to synth using synth install?



Reinstall packages via synth using prime list, using a fresh ports tree.
`portsnap fetch update`
`synth install primelist`

EDIT: When you say you wiped out the packages that includes actually uninstalling them, and their dependencies, from your system right?

`pkg delete pkgs_names_here`
`pkg autoremove`


----------



## marino (Feb 23, 2018)

synth doesn't come into play if you're just reinstalling official packages.  You can (and should) do it all via pkg(8)


----------



## marino (May 2, 2018)

Synth version 2.03 was just released.
It's main improvement is cutting down flavor index generation from around 20 minutes to 5 minutes (On DF server with 8 CPUs it's 84 seconds, so there is still fat to be removed in the scan).

A minor change per request is now synth will exit with code 1 if a build attempt aborts due to synth detecting out of date saved options.


----------

