# Synth: Introducing new custom package repository builder for FreeBSD and DragonFly



## marino (Jan 11, 2016)

I've been working on new software for the past few weeks.  It's called "Synth" and one of its main purposes is to replace portmaster/portupgrade.  It it similar to poudriere in that it supports concurrent building and local respository creation, but it's aimed at the regular user.  It is a drop-in tool, and there is no "jail building" or "ports tree" installing or anything like that.  It uses the features of the system it builds on by default, but it builds everything in a clean environment and it builds exactly what is needed.

It's particular aimed at users that:

Insist on building ports from source no matter what.

Need ports with options other than defaults.

Want system upgrades to be painless.

Have multiple systems that they want to share a common, customized
package repository.
Think postmaster is great.

Want to gain concurrent package building locally.

Don't like package surprises.

Don't understand or want to deal with poudriere.
I've written a "HowTo" with graphics that is published on github: https://github.com/jrmarino/synth  (scroll down to README.md).  There is also a man page, but it's a drier version of the HowTo.

I pushed synth to FreeBSD ports last night (as ports-mgmt/synth).  That means it will not be available as a pre-built package for a few days.

For those that don't want to wait, they can build it themselves pretty easily.

Install the dependencies: `pkg install gcc6-aux ncurses`
Build it: `cd /usr/ports/ports-mgmt/synth && make install`

Check with: `synth configure`
Synth should actually work without configuring, but check the default configuration to make sure it's going to use areas that have plenty of hard disk space.

It has not had widespread testing for FreeBSD yet.  If you want to provide detailed feedback, the best place is GitHub ( https://github.com/jrmarino/synth/issues ) but if you don't have an
account you can just write back here and I'll check back periodically.

Known (current) limitations:

It won't leverage pre-built official binaries at this time.  It will build every dependency that it needs.  I'll look into seeing how feasible downloading already-built dependency packages is in the near future.

The "test mode" isn't complete.  It doesn't check for leftover files or filesystem violations (the latter isn't a big deal because most of the build area is read-only so violators generally fail during the build).

There's no "hang" watchdog yet.  Any process that gets stuck won't be automatically killed and it could result in hanging mounts.  For stock port trees, this shouldn't come up though, it's mostly a port developer issue.
I highly recommend that you review the README link above.  There are screenshots!  I hope people find Synth useful and a suitable replacement for PortMaster and PortUpgrade.


----------



## SirDice (Jan 11, 2016)

I'm very happy with my poudriere set up but I'm definitely going to play around with this.


----------



## kpa (Jan 11, 2016)

As far as I understand there is nothing in it to guarantee clean build environments because no jail(8)s or chroot(8)s are used so pay special attention to /usr/ports/UPDATING for build time dependencies.


----------



## cpm@ (Jan 11, 2016)

Thanks for all your hard work it is much appreciated, John!


----------



## marino (Jan 11, 2016)

kpa said:


> As far as I understand there is nothing in it to guarantee clean build environments because no jail(8)s or chroot(8)s are used so pay special attention to /usr/ports/UPDATING for build time dependencies.



It uses chroot(8)s extensively.  A clean environment is constructed for each and every port built.  Execute "mount" command while synth in running and you'll see dozens of mounts.  So yes, the environment is guaranteed to be clean and always the same on repeated runs.


----------



## kpa (Jan 11, 2016)

marino said:


> It uses chroot(8)s extensively.  A clean environment is constructed for each and every port built.  Execute "mount" command while synth in running and you'll see dozens of mounts.  So yes, the environment is guaranteed to be clean and always the same on repeated runs.



Ok  The details of how it works just didn't jump at me at the first glance on the documentation. This sounds very promising, portmaster as it should have been from the get go.


----------



## marino (Jan 11, 2016)

I don't want to be too hard on Portmaster because it was written long before poudriere and pkg.  Also, since I've never used portmaster myself (only portupgrade back in the day), when I reviewed the man page my eyes glazed over.  I wondered how anybody could grok it quickly.

With that in mind, I intentionally made synth have a seemingly simple interface with a limited number of understandable commands.  I aimed it at the everyday user, not the ports committer or expert.  While it appears simple at the interface, it's got a lot going on underneath.  I just didn't want to get too technical with the presentation, but rather preferred the goal of having it "just work" out of the box.


----------



## Crivens (Jan 11, 2016)

Out of curiosity, what is the gcc dependency for and why is it needed?


----------



## marino (Jan 11, 2016)

Crivens said:


> Out of curiosity, what is the gcc dependency for and why is it needed?


Synth is written in Ada.
gcc*-aux ports (gcc-aux, gcc5-aux, gcc6-aux) are versions of GCC that contain the Ada front-end.  It's needed to compile the program (build dependency only)


----------



## kpa (Jan 11, 2016)

Trying it out the first time I noticed that it auto-detected some settings from my /etc/make.conf such as DISTDIR. It did not auto-detect the ports tree directory PORTSDIR and I got an error about /usr/ports/ports-mgmt/pkg not existing on first `synth configure` run and I had to nullfs mount the ports tree at /usr/ports. Is it by any chance testing for the wrong variable?


----------



## marino (Jan 11, 2016)

If PORTSDIR is defined in the environment, it's picked up, with the highest priority.  If it's defined in make.conf, it's not picked up.  Basically it checks PORTSDIR (env), then /usr/dports, then /usr/ports and then it just defaults to /usr/ports.  (this probably needs to eject before writing synth.ini and tell the user to set PORTSDIR).

You should use `synth configure` now to fix the portsdir setting and then you can remove the nullfs.

I'll put more graceful handling of this situation on the to-do list.


----------



## kpa (Jan 11, 2016)

Yes, that's exactly what I did, it now uses my custom ports tree correctly after saving the configuration.


----------



## marino (Jan 11, 2016)

Okay, I just a private email with a question that revealed there's a bug for people with default setups.

On FreeBSD (but not DragonFly), the default location for distfiles is /usr/ports/distfiles.  This directory passes the criteria as a ports category, so Synth will search it when scanning the ports tree, and quickly fails.

I need to update the Synth to avoid this directory.

If somebody wants an immediate workaround, it would be to move /usr/ports/distfiles to /my/new/location and set "DISTDIR=/my/new/location" in /etc/make.conf and run `synth configure` to change locations there as well.  I'll push v98.1 soon though.


----------



## marino (Jan 11, 2016)

I've updated the port to version 0.98_1.
It handles kpa's case now (and properly aborts if valid PORTSDIR can't be found anywhere) and it should skip ${PORTSDIR}/distfiles during the portstree scanning.

https://svnweb.freebsd.org/ports?view=revision&revision=405790


----------



## marino (Jan 11, 2016)

I've updated the port to version 0.98_2.
For systems that don't have $LOCALBASE/etc/pkg/repos directory, synth would fail to install anything, because the pkg conf file that synth installs (00_synth.conf) failed.  The latest version creates the path if it's missing.  (This can't happen on DragonFly, thus it took a FreeBSD user to flush out the oversight)


----------



## kpa (Jan 11, 2016)

Couple of things. How can I use an existing /etc/make.conf to set the port options? It looks like synth is now creating its own /etc/make.conf under the chroot with preset options that can not be controlled. I really don't want to create the options under /var/db/ports because the saved options are a bad idea in my opinion.

Next is control-c and signal handling in general. It would be nice if the tool tried to clean up properly after being shut down by a signal, now it leaves the nullfs mounts behind if killed with control-c.


----------



## marino (Jan 11, 2016)

The man page and the graphic howto both address both those topics.

signal-handling: Use the "Escape" key.  I had to abandon SIGINT capture because it propagates to the builders and breaks when conftests segfaults.  You should almost never use control-C.  However, if you do, run any major synth command, like `synth configure` and it will clean up the dirty mounts automatically.

See FILES on man page for make.conf customization (or miscellaneous on graphic howto).


----------



## talsamon (Jan 12, 2016)

Congratulations! Yesterday I tried it first time (yesterday it fails, but it seems it was something with my system. Today in the morning it runs fine). At the first look seems a fine tool.
What I am missing also on poudriere, I would like to have (before it starts to compile) the output of a list of the dependencies which will compiled (with start and/or cancel option/question).


----------



## marino (Jan 12, 2016)

talsamon said:


> I would like to have (before it starts to compile) the output of a list of the dependencies which will compiled (with start and/or cancel option/question).



That's what the `synth status` commands do.  There are 3 flavors: `synth status`, `synth status [ports]`, and (ignore this), `synth status-everything`.

See `synth help`, `man 1 synth` or the graphic howto on the GitHub site.


----------



## marino (Jan 12, 2016)

(to be clear, status just gives you the list that you want, it's a dry-run.  You need to give the command to build to actually do anything)


----------



## kpa (Jan 12, 2016)

talsamon said:


> What I am missing also on poudriere, I would like to have (before it starts to compile) the output of a list of the dependencies which will compiled (with start and/or cancel option/question).



I did suggest this to the poudriere developers but their response was that you can always use `make all-depends-list` (and the other two for run and build dependencies) either by running that in the jail or directly from the host.


----------



## marino (Jan 12, 2016)

kpa said:


> I did suggest this to the poudriere developers but their response was that you can always use `make all-depends-list` (and the other two for run and build dependencies) either by running that in the jail or directly from the host.



Except that tells you every dependency, not what is going to build.
If there are no packages, then "every dependency" is the same as the build list

If everything has already been rebuilt and only a couple of packages change, then maybe the build list is say, 3 packages long.
*Synth status* gives a list of what will build after evaluating which packages are still good and don't need rebuilding.  So I don't necessarily agree with poudriere dev stance, if that's what was said.


----------



## talsamon (Jan 12, 2016)

Ok,  `synth status category/port` does what I want, thanks.


----------



## protocelt (Jan 12, 2016)

Hi.

First, thanks for this and the work you put into it. It definitely seems like a large improvement on ports-mgmt/portmaster so far.

One question, is there a reason synth(1) doesn't nullfs(5)? mount the /usr/lib32 directory as well? At least on my machines running 11-CURRENT and 10.2-STABLE this is the case. Ports like emulators/virtualbox-ose will bail on building due to requiring access to the lib32 base libraries. I just left the configuration at the defaults for testing it out on both machines. I'm rebuilding all installed packages on both systems with it and so far haven't had a problem otherwise.


----------



## marino (Jan 12, 2016)

lib32 just wasn't implemented.  The linux support is brand new but I forgot about lib32.  DragonFly doesn't have these.  I imagine it's not difficult to implement (e.g. one additional if on freebsdFreeBSD).

I ran into a potential issue with java/openjdk8 myself (socket timeout trying to connect to VM) so I'd be interested to hear if anyone has trouble with that one.  I suspect the builders minimal /etc/services is missing a port/protocol there.


----------



## protocelt (Jan 12, 2016)

marino said:


> lib32 just wasn't implemented. The linux support is brand new but I forgot about lib32. DragonFly doesn't have these. I imagine it's not difficult to implement (e.g. one additional if on freebsd)


Ok, thanks.



marino said:


> I ran into a potential issue with java/openjdk8 myself (socket timeout trying to connect to VM) so I'd be interested to hear if anyone has trouble with that one. I suspect the builders minimal /etc/services is missing a port/protocol there.



FWIW, I just finished building java/openjdk8 on CURRENT with synth(1) and had no obvious issues on my machine, at least not that I noticed thus far. It's still got 400+ ports to go however.


----------



## fernandel (Jan 12, 2016)

talsamon said:


> Ok,  `synth status category/port` does what I want, thanks.



I get dependencies but I like it to have  before start compiling options  for settings (make config).


----------



## marino (Jan 12, 2016)

I don't understand what you mean.
If you don't `make config` or you issue `make rmconfig` then `synth status` gives you the standard dependencies (again assuming there are zero packages already built).  Which is not a good assumption.  "status" always give the incremental list of built ports, not the absolute.

I you want the absolute list, then you're back to the standard `make all-depends-list` ports command.


----------



## fernandel (Jan 12, 2016)

marino said:


> I don't understand what you mean.
> If you don't `make config` or you issue `make rmconfig` then `synth status` gives you the standard dependencies (again assuming there are zero packages already built).  Which is not a good assumption.  "status" always give the incremental list of built ports, not the absolute.
> 
> I you want the absolute list, then you're back to the standard `make all-depends-list` ports command.



I was not clear...
When I run for example `portmaster multimedia/ffmpeg` it will first open options (as you wrote "2.Need ports with options other than defaults.").


----------



## Beastie7 (Jan 13, 2016)

Just by glancing over it seems a lot more straight-forward and intuitive than poudriere/portmaster. Will definitely try this out. Good stuff.

edit: quick dumb question. Why the name Synth?


----------



## marino (Jan 13, 2016)

fernandel said:


> I was not clear...
> When I run for example `portmaster multimedia/ffmpeg` it will first open options (as you wrote "2.Need ports with options other than defaults.").


Synth doesn't do any of that.  It assumes that you already went to the port and redefined options (which caches them), otherwise it assumes the default options.   It does not handle the options dialog at all.  This is intentional.

What it will do is flag a port with cached options that are out of date.  If it's requested to build that port, it will not start the build but will list the port(s) which bad options and tell the user to either `make config` or `make rmconfig` that port and try again.


----------



## marino (Jan 13, 2016)

protocelt said:


> FWIW, I just finished building java/openjdk8 on CURRENT with synth(1) and had no obvious issues on my machine, at least not that I noticed thus far.



I tried again on a new machine and openjdk8 built fine.  My previous issue must be something specific to the first machine, thanks.


----------



## marino (Jan 13, 2016)

Beastie7 said:


> edit: quick dumb question. Why the name Synth?



short, unique, memorable, pronounceable (/me looks at poudriere) and is the root of "synthesis", which the first definition probably most apt: 





> The formation of something complex or coherent by combining simpler things.


 or "synthesize" which is a nearly a synonym for "make" only mostly for chemistry


----------



## Beastie7 (Jan 13, 2016)

Very sound.  Cool


----------



## garry (Jan 13, 2016)

marino said:


> Synth is written in Ada.
> gcc*-aux ports (gcc-aux, gcc5-aux, gcc6-aux) are versions of GCC that contain the Ada front-end.  It's needed to compile the program (build dependency only)



In fact the synth code is so beautiful it got me interested in learning ada and I ordered a couple of the Ada '95 books.  Thanks for synth, and for showing us a good example with your Ada code.  Synth was easy to configure with the interactive `synth configure`.  It was easy to tell it to use my non-standard location of distfiles, to use ccache, etc.  I like the use of "profiles" and the per-profile make.conf.  It chose good values for the number of builders and jobs and really loads my cpus well -- it was fast in building my repository.  I like the continuously updating status displayed during the build.  Showing the "lines" of progress into the build for each package as it's building is helpful.

I've been using poudriere(8) for building a local repository of about 3000 packages and may now be switching to synth(1).


----------



## garry (Jan 13, 2016)

marino said:


> short, unique, memorable, pronounceable (/me looks at poudriere) and is the root of "synthesis", which the first definition probably most apt:  or "synthesize" which is a nearly a synonym for "make" only mostly for chemistry



After looking "synth" over I thought the command should have been named "synthesize" and the "upgrade-system" subcommand could have been simply "system" so that the commonest command could have been `synthesize system`.


----------



## fernandel (Jan 14, 2016)

marino said:


> Synth doesn't do any of that.  It assumes that you already went to the port and redefined options (which caches them), otherwise it assumes the default options.   It does not handle the options dialog at all.  This is intentional.
> 
> What it will do is flag a port with cached options that are out of date.  If it's requested to build that port, it will not start the build but will list the port(s) which bad options and tell the user to either `make config` or `make rmconfig` that port and try again.



Thank you. I gave it a try and it works very fast and I didn't get any errors yet .


----------



## marino (Jan 16, 2016)

protocelt said:


> One question, is there a reason synth(1) doesn't nullfs(5)? mount the /usr/lib32 directory as well? At least on my machines running 11-CURRENT and 10.2-STABLE this is the case. Ports like emulators/virtualbox-ose will bail on building due to requiring access to the lib32 base libraries.



I think I've implemented support for this if you want to try it out.

if so, change line 24 of Makefile from

```
GH_TAGNAME=    v1.3:bundle c83a9d9
```
to

```
GH_TAGNAME=    v1.3:bundle b21d685
```
then `make makesum`
then the standard `make deinstall ; make clean ; make install` command.


----------



## Beastie7 (Jan 16, 2016)

The use of a concurrent primitive language is really interesting to me which caused me to do some research into languages like it (ie, Erlang, Elixir, etc). I'd wager for something like a build system where you're handling many heavy tasks at once and in a world of multiple cores, the synchronous message passing helps with speed. At least to my understanding. Ada looks to be very readable also.


----------



## protocelt (Jan 16, 2016)

marino said:


> I think I've implemented support for this if you want to try it out.


Just tested it out. Seems to work as suggested. Thanks!

On a tangential note, is there any future possibility for adding some flags(overrides) for the build commands such as concurrent builders and max jobs? Certainly not a necessity but would be a convenience for a user that wanted to change things on the fly depending on how a system is being used at that time. Having said that, I do understand the aim was to be simple so just asking mostly out of curiosity.


----------



## Beastie7 (Jan 16, 2016)

protocelt said:


> Just tested it out. Seems to work as suggested. Thanks!
> 
> On a tangential note, is there any future possibility for adding some flags(overrides) for the build commands such as concurrent builders and max jobs? Certainly not a necessity but would be a convenience for a user that wanted to change things on the fly depending on how a system is being used at that time. Having said that, I do understand the aim was to be simple so just asking mostly out of curiosity.



This.


----------



## marino (Jan 16, 2016)

I'm not planning on doing that.  It's a slippery slope that will result in an invocation mess as you can see with portmaster man page.  You could always create another profile or two so it's simply a matter of switching profiles (and no, I haven't throught about selecting profiles via command either although one switch would be preferable to several)


----------



## protocelt (Jan 16, 2016)

marino said:


> I'm not planning on doing that.  It's a slippery slope that will result in an invocation mess as you can see with portmaster man page.  You could always create another profile or two so it's simply a matter of switching profiles (and no, I haven't throught about selecting profiles via command either although one switch would be preferable to several)


Understandable. I haven't tested the profile feature(s) yet so it wasn't on my mind when I wrote the above reply. I would think profiles would suffice just fine. Thanks


----------



## marino (Jan 17, 2016)

I've just updated synth in ports.  From the log:

```
The following changes have been implemented:
    * The builder command executer had been upgraded with a watchdog.  It
      does not monitor the overall time of a phase (things like fetch /
      checksum vary depending on the internet connection and the volume it
      needs to download ranges from bytes to gigabytes), but it does monitor
      log progress.  Each phase has a maximum amount of time allowed for the
      log to be static (measured in lines, not file size).  If the log is
      static for too long for that phase, the processes of the builder will
      be killed, and the builder log updated accordingly.
    * The load indicator was stuck at 0.00 for some named locales (only on
      FreeBSD) so this was resolved.
    * Ports tree scanning time was cut nearly in half by caching make
      variables on each builders make.conf
    * Support for /usr/lib/lib32 was added for FreeBSD
    * purge-distfiles command was improved by handling potential exceptions
      and fixing the case of 100-1023 Mb purged (range was too narrow)
    * Typos on man page fixed
    * The directory ${PORTSDIR}/packages are now ignored.  This is the
      default package location and any existing packages were getting
      treated as port directories.
    * Skip some additional questions/actions if a graceful shutdown was
      previously detected
```

Some of those are big items, so I think most people would be interested.
I'm not moving to 0.99 until the functionality to use prebuilt binaries for dependencies is implemented.


----------



## marino (Jan 17, 2016)

hmm, there may be a regression on ports using scons.  I'm not sure what it is, scons just freezes as soon as it's invoked with no error message.  Maybe something with the watchdog setting the process group?  unclear.


----------



## marino (Jan 17, 2016)

yes, it's this addition "setpgid (0, 0);"
at https://github.com/jrmarino/synth/commit/71d83acba0adc51b3c54ba1ab1a54caf6638c96b
that makes scons just break without output.  I've no idea why.


----------



## marino (Jan 17, 2016)

When I use the forking version of synthexec, scons works again, so I think that's what I'll have to do.  Apparently scons doesn't like to be the process leader.


----------



## marino (Jan 18, 2016)

I could not find a solution to the scons problem, so I've disabled the watchdog until future notice.  As far as I know it's a bug with scons, but there are tool many ports dependent on scons.  It may take a while to get resolved and re-enabled.  I pushed 0.98_4.


----------



## kpa (Jan 20, 2016)

It seems to work fine but I'm wondering why I keep getting a "Unfortunately, the system upgraded failed" -message (it should say "system upgrade") everytime on `# synth upgrade-system` even if everything seems to work? What's the condition that is tested for failure? I have no other repositories enabled but the one created by synth. System is:


```
FreeBSD freebsd10 10.2-RELEASE-p10 FreeBSD 10.2-RELEASE-p10 #13 r294085: Fri Jan 15 16:35:16 EET 2016     root@freebsd10:/usr/obj/usr/src/sys/GENERIC  amd64
```


----------



## marino (Jan 20, 2016)

I'll fix the typo.  
There are three places it could emit that, see: https://github.com/jrmarino/synth/blob/master/src/portscan-pilot.adb#L767
look for "sorry", there are three places where "sorry" could be emitted.  However, it should never emit that upon success, so I have to doubt upgrade-system succeeded.
It might help to see the what is printed out prior to the sorry message.


----------



## marino (Jan 20, 2016)

ah, it might be line 803.  It seems it prints sorry upon success instead of failure.


----------



## kpa (Jan 20, 2016)

This was the update of today:


```
..snip...
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%    260 B   0.3kB/s    00:01    
Fetching packagesite.txz: 100%   54 KiB  54.9kB/s    00:01    
Processing entries: 100%
Synth repository update completed. 207 packages processed.
Checking for upgrades (3 candidates): 100%
Processing candidates (3 candidates): 100%
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        synth: 0.98_2 -> 0.98_4

Installed packages to be REINSTALLED:
        curl-7.46.0_2 (options changed)

The process will require 8 KiB more space.
[1/2] Upgrading synth from 0.98_2 to 0.98_4...
[1/2] Extracting synth-0.98_4: 100%
[2/2] Reinstalling curl-7.46.0_2...
[2/2] Extracting curl-7.46.0_2: 100%
Unfortunately, the system upgraded failed.
freebsd10 ~ %
```


----------



## marino (Jan 20, 2016)

well, if you want to try, run "make patch" and then change line 803 from

```
if Unix.external_command (command)
```
to 

```
if not Unix.external_command (command)
```
and "make deinstall; make install"

I expect that is the fix.  it may be a couple of days before the next update because I'm doing an extended test.


----------



## kpa (Jan 20, 2016)

I don't have time to test the fix but it looks right, under a shell the status code 0 means success but in almost every programming language 0 means false that translates to failure.


----------



## kpa (Jan 20, 2016)

As a programmer I would write constructs like that in this fashion to make it clear to anyone reading the code what the success status code is (in C):


```
char *externalcommand = ... ;
int status;
...

status = system(externalcommand);

if (status != 0) {
    /* Failure */
...
}
```


----------



## Chris_H (Jan 20, 2016)

WOW! Brilliant work, marino@ !
The options are many, but are intuitive enough so as to not be overwhelmed -- thanks! Thank you also for the work you put into the FAQ/documentation. It's lengthy, yet concise -- perfect! While I haven't _yet_ actually used it to upgrade/install. I _have_ used it to evaluate the list of ~960 ports I want it to update -- `synth status ./PORTLIST`. Which brings me to my only nit, so far;
It doesn't appear that synth evaluates `make configure` the same way that the FreeBSD ports() framework does. Case in point; synth evaluated the list I fed it, and returned ~60 ports that it indicated I need to (re)configure -- make configure || rmconfig/configure. So I went through the list, and about half of those returned "configuration found for blah-blah-NN". So after finishing the list, I figured that synth would also mark those off the list. It didn't. So it appears to me, I will now have to memorize the options for each of those, prior to a `rmconfig` followed by a `configure`. A bit of a PITA, IMHO. Nothing I can't manage. Just wished I didn't have to. 
If you can think of anything here. I'd love to hear about it -- aside from keeping the ports updated more often. 

A *huge* thanks again, John! _Really_ appreciate all the work, and thought you put into this!

--Chris


----------



## marino (Jan 20, 2016)

Chris_H said:


> So after finishing the list, I figured that synth would also mark those off the list. It didn't.



But it does.  you can't even advance it if didn't.  From the symptoms, I'd say you might have two locations for options, especially if you had a list of 60 obsolete cached options.

to iterate, if you `make -C /usr/ports/<cat>/<port> rmconfig` and then synth status returns it as still bad, synth isn't looking at the same place as `/usr/ports` is storing them.


----------



## Chris_H (Jan 20, 2016)

marino@ said:


> But it does.  you can't even advance it if didn't.  From the symptoms, I'd say you might have two locations for options, especially if you had a list of 60 obsolete cached options.
> 
> to iterate, if you "make -C /usr/ports/<cat>/<port> rmconfig" and then synth status returns it as still bad, synth isn't looking at the same place as /usr/ports is storing them.


Thanks for the prompt reply, John.
I think it must be my explanation. Let me re-iterate;
`synth status ./PORTLIST` returned a list of ~60 ports that the configure options were out of date, and hence; needed a `make configure`. I went through them all, performing a `make configure` about two thirds of them brought up the configuration dialog(). The other third returned `Found configuration for port-blah-NN...`. My point is; if the configuration is considered up to date enough for the port itself. Then why doesn't synth agree (it still says I need to (re)configure the port)?

Thanks again, John.

--Chis


----------



## marino (Jan 20, 2016)

by the way, you should rmconfig any saved configuration that is the same as the default options.  synth assumes the defaults.  most people have unnecessary cached options because they just hit "yes", "yes", yes" when the dialog window keeps coming up.


----------



## marino (Jan 20, 2016)

i should have read further.  





> Then why doesn't synth agree (it still says I need to (re)configure the port)?



the port isn't checking the validity of the config file, only that it exists.
If synth says it's bad, it's bad.  It checked the options against the current port and something doesn't match up.
If you "make config" and save, it will be fixed.  Likewise "rmconfig" will obviously fix it.


----------



## marino (Jan 20, 2016)

> The other third returned Found configuration for port-blah-NN....



"make config" brings up the dialog no matter what.  It won't print "found configuration" and not bring up the dialog.


----------



## marino (Jan 20, 2016)

make sure you use `make config`, NOT `make configure`.


----------



## Chris_H (Jan 20, 2016)

OK. Got it! That makes sense enough. In the end; if I'd kept this particular box more up to date, synth would currently building/updating my ports, instead of warning (scolding) me. 

Really, John. I can't thank you enough for this tool. It's a *real* godsend. I can't believe the load it takes off me.
Job well done!

--Chris

Yea. I'm using `make config`. 

...and it's performing a `build ./PORTLIST` now!


----------



## Chris_H (Jan 21, 2016)

OK in case anyone's interested. I cobbled up a script to build a complete list of ports installed on the system. So I could feed it to ports-mgmt/synth
It goes as follows, it's heavily commented, and _should_ run without error. but has no _real_ accommodations for error -- you have been warned! 


```
#!/bin/sh -
# first create a list of all currently installed ports
pkg info >>./PORTLIST-RAW
```


```
#!/bin/sh -
# pkg(8) also prints a description of the ports listed
# we only want the port's names, get them
awk '{print $1;}' ./PORTLIST-RAW >./PORTLIST-CLEAN
```


```
#!/bin/sh -
# now all we need to do is prepend the category to
# those port names. I created the following script to
# accomplish just that

# create a categorized ports list to feed to
# ports-mgmt/synth

# place a list of port names, one per line *between*
# the following quotes

ports="

"
for name in $ports

do
   pkg origin $name >>./PORTLIST-DO
# the following line is just to produce some status/feedback
   echo $name
done
```

That's actually 3 scripts. You could actually run the first 2 commands w/o
the use of sh. I just ran the commands pkg(8), and awk(1) at my console. It's up to you. The third one should be run as a shell script. As you can see, there is some manual intervention involved. You will need to feed it the contents of PORTSLIST-CLEAN. It should finally look something like the following:

```
ports="
xterm-314
xtrans-1.3.5
zip-3.0_1
zziplib-0.13.62_2
"
```
Just put the list just like that between the 2 double-quotes. You can then run the script to create your final list, to feed to ports-mgmt/synth.
Worked well for me. Hope it works well for you! 

--Chris


----------



## protocelt (Jan 21, 2016)

Yikes! That seems a lot of work. 

As you said, you could just use pkg(8) and awk(1) in a one-liner, something like the following: `# synth build/force `pkg info -oa | awk -F ' ' '{ print $2 }'``


----------



## Chris_H (Jan 21, 2016)

Indeed, protocelt
It was more of a brain dump. Those were all the pieces I used to arrive at a final list. But when I pasted them all in the post, and saw them all together. I quickly arrived at much the same as you. 
Oh well. It's the thought that counts, right? 

--Chris


----------



## protocelt (Jan 21, 2016)

Chris_H said:


> Oh well. It's the thought that counts, right?


Indeed it is.


----------



## marino (Jan 21, 2016)

Chris_H said:


> OK in case anyone's interested. I cobbled up a script to build a complete list of ports installed on the system. So I could feed it to ports-mgmt/synth



I don't understand.  Why don't you simply use `synth rebuild-repository` or `synth upgrade-system`?

Synth knows what you have installed, and it just builds those installed ports with those commands.

It seems to me that the whole script is unnecessary.


----------



## Chris_H (Jan 21, 2016)

marino@ said:


> I don't understand.  Why don't you simply use `synth rebuild-repository` or `synth upgrade-system`?
> 
> Synth knows what you have installed, and it just builds those installed ports with those commands.
> 
> It seems to me that the whole script is unnecessary.


Thank you for the information, marino@
While that is indeed good to know, and I would surely be better off using that method. The ports-mgmt/synth exit status seems contrary:

```
The task is complete.  Final tally:
Initial queue size: 1024
  packages built: 852
  ignored: 1
  skipped: 168
  failed: 3

Duration: 14:27:24
The build logs can be found at: /var/log/synth
Would you like to rebuild the local repository (Y/N)? Y
Scanning entire ports tree.
progress: 98.05%
culprit: packages/All
Scan aborted for an unknown reason.
bad input for 'Value: ""
Failed to scan ports tree  (Synth must exit)
```
Aside from the obvious portions. I'm not exactly sure what to make of it.  Can you please translate why the rebuild of the repository failed? Clearly packages/All wasn't properly prepended with /usr/ports/. But my synth.ini completely checks out. No anomalies, what so ever.

Thanks again, John.

--Chris


----------



## Chris_H (Jan 21, 2016)

Additional information, in case it helps:


```
FreeBSD dev-box 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r294112:
Mon Jan 18 14:25:01 PST 2016 root@dev-box:/usr/obj/usr/src/sys/DEVBOX amd64

Path: /usr/src
Working Copy Root Path: /usr/src
URL: svn://svn.freebsd.org/base/head
Relative URL: ^/head
Repository Root: svn://svn.freebsd.org/base
Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Revision: 294112
Node Kind: directory
Schedule: normal
Last Changed Author: ak
Last Changed Rev: 294111
Last Changed Date: 2016-01-15 15:13:01 -0800 (Fri, 15 Jan 2016)

Path: /usr/ports
Working Copy Root Path: /usr/ports
URL: svn://svn.freebsd.org/ports/head
Relative URL: ^/head
Repository Root: svn://svn.freebsd.org/ports
Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
Revision: 406193
Node Kind: directory
Schedule: normal
Last Changed Author: madpilot
Last Changed Rev: 406193
Last Changed Date: 2016-01-15 14:38:36 -0800 (Fri, 15 Jan 2016)

synth-0.98_2
Name  : synth
Version  : 0.98_2
Installed on  : Wed Jan 20 07:43:43 2016 PST
Origin  : ports-mgmt/synth
Architecture  : freebsd:11:x86:64
Prefix  : /usr/local
Categories  : ports-mgmt
Licenses  : ISCL
Maintainer  : marino@FreeBSD.org
WWW  : https://github.com/jrmarino/synth
Comment  : Custom package repository builder for FreeBSD and DragonFly
```

Thanks, again.

--Chris


----------



## marino (Jan 21, 2016)

Chris_H said:


> Additional information, in case it helps:
> 
> 
> ```
> ...



That's been fixed in 0.98_4.  The version you installed is too old.  You can `pkg upgrade synth` because portsmon says 0.98_4 is available everywhere


----------



## Chris_H (Jan 21, 2016)

I'll get on it.

Thank you, John!

--Chris


----------



## Chris_H (Jan 22, 2016)

OK. I started a 2nd run, after telling pkg() to upgrade ports-mgmt/synth, and you were right; the package clusters had 0.98_4. Anyway I chose another `synth build /BUILDLIST`. Before you remind me about `rebuild-repository`, or `upgrade-system`, I just want to know that It'll finish a task completely, and uneventfully, before I make a full commitment, and let alter my system. 
Anyway, it, or rather, one of the builds failed. Shortly after, a (system?) message was cast across the bottom of the screen:

```
ELF interpreter /libexec/ld-elf.so.1 not found, error 2
```
As you might guess from the number of ports it needs to build, this box is lagging a bit. But still, if it had anything to do with the synth install, I would have figured that pkg() would reconcile any dependencies, or forbid the install, for lack of prerequisites. So I'm not sure where to point, or look, exactly. I've instructed synth to finish what it's doing, and end the current build session (ESC). So I'll give the log(s) a look (system && synth), when synth's finished. But thought I'd mention it, in case you've seen, or heard anything like this, where synth is concerned.

Thanks!

--Chris


----------



## marino (Jan 22, 2016)

> Before you remind me about rebuild-repository, or upgrade-system, I just want to know that It'll finish a task completely, and uneventfully, before I make a full commitment, and let alter my system


`synth build-repository` doesn't alter the system.  It's just assembling information about all the build packages, but that's it.



> I would have figured that pkg() would reconcile any dependencies.


  Why would you think that?  It's Synth's job.

The error sometimes happens as a result of failed builds.  I've see it before on occasion.

Synth isn't at release status yet,  I'm still looking for, finding, and fixing some bugs (many reported here)


----------



## Chris_H (Jan 22, 2016)

N.P. I _know_ that it's still a young application. But I _really_ like it, and am anxious to provide anything I can to help it become stable. 
As to pkg() being responsible for chasing dependencies. Yes. I knew. But was _sure_ synth couldn't possibly be wrong. 
Seriously. After posting my message, and going back to monitor the progress. It occurred to me that the only thing that wouldn't/couldn't find /libexec/ld-elf.so.1 would be something in a chroot(), or jail(). So concluded that either the chroot wasn't setup correctly, or something in a chroot crashed badly.
Anyway. I'll report back with anything that looks like it might be of any interest to you.

Thanks for the reply, John.

--Chris


----------



## fernandel (Jan 22, 2016)

I switched to synth from portmaster and it works great.
I had some problem. I got failure for building lang/sbcl and math/maxima which I thing is not related to synth but I have also
devel/libc++ and archivers/file-roller which I get "failed dependency check" but building is successful.
On the and I have:


> Scanning existing packages
> Checking integrity...done (2 conflicting)
> pkg: Cannot solve problem using SAT solver:
> Checking integrity...done (0 conflicts)
> Unfortunately, the system upgraded failed



Everything is updated and it works but every time it rebuilt devel/libc++ and archivers/file-roller.
Thank you.


----------



## marino (Jan 22, 2016)

fernandel said:


> Everything is updated and it works but every time it rebuilt devel/libc++ and archivers/file-roller.



I'll check those two ports out.  Very often, the problem is with the port itself.  I've fixed several problems with ports that synth detected (but poudriere did not) already.


----------



## marino (Jan 22, 2016)

fernandel said:


> archivers/file-roller



I spotted a problem on file_roller.  It only affects FreeBSD.  Since DragonFly is not affected, I can't easily test my fix:  http://www.freshports.org/commit.ph...=201601221319.u0MDJbbm075196@repo.freebsd.org

Please update ports and rebuild file_roller one more time and let me know if if it stops rebuilding after that.


----------



## marino (Jan 22, 2016)

fernandel, I see a problem with libc++ as well.  I assume /usr/lib/libcxxrt.so exists on your system so 
	
	



```
LIB_DEPENDS=    libcxxrt.so:${PORTSDIR}/devel/libcxxrt
```
 is a bad specification.
I won't fix this myself, but I'll tell dim@ about it.


----------



## Chris_H (Jan 22, 2016)

> Unfortunately, the system upgraded failed


I think this is an indication that you aren't on the most recent version. I also know personally, that the newest version fixed an issue I had. Just thought I'd mention it. 

--Chris


----------



## marino (Jan 22, 2016)

no, it's not.  That bug has been fixed in the repository, but it's not been pushed to ports.  It's not the only fix either that's not in ports yet.


----------



## Chris_H (Jan 22, 2016)

marino@ , I sent an `ESC` to synth, last night. It sent me the message that it was shutting down. However, it's still running this morning. It had two targets in the queue. One finished about a half hour later, but it's still working on the other one, apparently (screen is corrupt, so I can't tell what the other one is).
I'm thinking at this point I should probably send it a HUP signal. But am wondering what sort of cleanup I have to deal with, afterwards. Any pointers?

Thanks, John.

--Chris


----------



## Chris_H (Jan 22, 2016)

marino@ said:


> no, it's not.  That bug has been fixed in the repository, but it's not been pushed to ports.  It's not the only fix either that's not in ports yet.


Oh, OK. I remembered a discussion about that, earlier in this thread, and (mistakenly) assumed that it had (already) been pushed to the current version. My bad. I'll shut up now. 

--Chris


----------



## marino (Jan 22, 2016)

Chris_H said:


> marino@
> I'm thinking at this point I should probably send it a HUP signal. But am wondering what sort of cleanup I have to deal with, afterwards. Any pointers?



Just hit control-C, then type `synth configure` and hit "enter".  Synth will see that the previous build didn't clean up after itself and it will clean it up, if it can.

(obviously the "enter" just leaves configure screen without doing anything)


----------



## Chris_H (Jan 22, 2016)

Perfect! Thank you, John!

--Chris


----------



## Chris_H (Jan 22, 2016)

Feature request
Given that this (currently) requires being root. (and) given that (I, as root) receive messages at the console; was wondering if you could somehow buffer those messages, for display after synth has finished it's tasks. Somewhat like ports-mgmt/portmaster does with the ports(7) messages that normally are displayed after the port has been installed. My issue is; that the (root) messages sent to the screen, corrupt the synth output. I could overcome this by temporarily diverting the messages to syslog(3) via syslog.conf(5) . But thought this might be nice feature. At least until this can be run by a normal user. But even then, I still think this would be a nice feature.

Thanks for your time, John.

--Chris


----------



## Chris_H (Jan 22, 2016)

OK. Just got another /libexec/ld-elf.so.1 error. Would you advise I bail, and start a new session? Is there any information that I could supply, that would help you find the culprit, to overcome this error?

Thanks!

--Chris


----------



## marino (Jan 22, 2016)

1) It's not supposed to output at all
2) Every 100 seconds, the display completely redraws, so the display is only corrupted for a maximum of 1 minute, 40 seconds.

I would figure out what package is causes the ld.elf error and what is special about it.  (e.g. linux?)


----------



## Chris_H (Jan 22, 2016)

Thanks for the quick response, John.


marino@ said:


> 1) It's not supposed to output at all
> 2) Every 100 seconds, the display completely redraws, so the display is only corrupted for a maximum of 1 minute, 40 seconds.


Odd. That's not my experience. Things get redrawn, alright. But the layout is all screwed up. The top 1/3rd of the screen is all out-of-whack, which includes the current builds. So time, and such don't jive with what's being displayed, and the top statistics display numbers, but the labels are gone.


marino@ said:


> I would figure out what package is causes the ld.elf error and what is special about it.  (e.g. linux?)


Yes. I have the Linux ABI loaded. If that's what you mean.

I'll bail, and start a new session.

Thanks again, John, for taking the time to respond.

--Chris


----------



## marino (Jan 22, 2016)

make sure the window is wide enough for 80 characters (for virtual terminals).  If it's more narrow it may not redraw right.

I wasn't asking if you *have* linux, I was given an example of a "special" port, e.g. one that requires linux emulation.


----------



## Chris_H (Jan 22, 2016)

Ahh. OK.
1) I'm running it on a (p)tty (Xorg, or other is not running)
2) OK. No, it wasn't building a Linux (related) port. So don't know what else to think. Could it be chroot(8) related? Are they large enough? I already have tmpfs(5) managing /tmp. But I've never observed swap exceed 23%. So I'm not sure what to expect. Load has never exceeded 4.3, either.

Thanks, John.

--Chris

OH! Forgot to mention;
I'm using an nVidia card, and as a result, using `kern.vty=sc`, as opposed to `vt`. In case it matters.


----------



## Beastie7 (Jan 22, 2016)

someone should show this off at a conference this year. This is really good stuff.


----------



## fernandel (Jan 22, 2016)

marino@ said:


> fernandel, I see a problem with libc++ as well.  I assume /usr/lib/libcxxrt.so exists on your system so
> 
> 
> 
> ...



It does:



> locate libcxxrt.so
> /lib/libcxxrt.so.1
> /usr/lib/libcxxrt.so
> /usr/lib32/libcxxrt.so
> ...


----------



## fernandel (Jan 22, 2016)

marino@ said:


> I spotted a problem on file_roller.  It only affects FreeBSD.  Since DragonFly is not affected, I can't easily test my fix:  http://www.freshports.org/commit.ph...=201601221319.u0MDJbbm075196@repo.freebsd.org
> 
> Please update ports and rebuild file_roller one more time and let me know if if it stops rebuilding after that.



It works .
Thank you.


----------



## marino (Jan 22, 2016)

Chris_H said:


> Things get redrawn, alright. But the layout is all screwed up.


I am starting to think the ncurses redraw doesn't work like I thought it did.  From the symptoms, it seems to only redrawn what it thinks has changed.  In other words, if it thinks the line is "xx00045" and you redraw it to "xx10045", only the "1" is updated.  I expected a full overwrite, but the image only fully redraws if I adjust the window of the virtual terminal.

I'm not a big fan of ncurses...


----------



## Chris_H (Jan 22, 2016)

marino@ said:


> I am starting to think the ncurses redraw doesn't work like I thought it did.  From the symptoms, it seems to only redrawn what it thinks has changed.  In other words, if it thinks the line is "xx00045" and you redraw it to "xx10045", only the "1" is updated.  I expected a full overwrite, but the image only fully redraws if I adjust the window of the virtual terminal.


Ahh. Sure. That makes sense.


marino@ said:


> I'm not a big fan of ncurses...


Hmm, I wonder if it can be improved in such a way as to keep a map of it's current/last state, and fully redraw itself?
I''l have a look at the source, and see if I can see any possibilities.
EDIT
Never mind. I spoke too soon. After giving it more thought, I realized that it would obviously be the responsibility of your (or any) application to keep/maintain the current state, and (in this case) redraw the _entire_ screen/framework. sorry for the noise.

Thanks for the reply, John!

--Chris


----------



## Chris_H (Jan 22, 2016)

> I'm not a big fan of ncurses...


What about forth?


----------



## marino (Jan 24, 2016)

I've pushed a significant bug fix and new feature update, 0.98_5, to ports:


```
ports-mgmt/synth: Finish test mode and various fixes

The following changes have been implemented:
  * The "test" command checks for file system violations between
    the configure and build targets (inclusive)
  * The "test" command checks for leftover (extra), missing, and
    unexpectedly modified files and directories between the stage and
    deinstall targets (inclusive)
  * Fix bug where successful system-upgrade was indicated as a failure
  * Bring in procfs mounts for x11-toolkits/gnustep-gui (only!)
    It appears to the be only port that needs it, but procfs appears to be
    pretty unstable, so we don't mount/dismount it unconditionally
  * Similarly, change linprocfs mounts/dismounts to only occur when when
    linux ports are building.  Linprocfs stability is unknown (and I can't
    test it on DF) so be conservative and use it as little as possible.
  * Fix bug on builders /etc/group file (some groups were missing)
  * Install /etc/master.passwd in builders, it is required for at least
    one port
  * Install /etc/rc.d and /etc/defaults/rc.conf in builders.  It is
    required for at least one port
  * Disable repository rebuild after synth-everything.  Twice it has
    removed all packages (over 23,000!) after a build, so there's a bug
    or missing safeguard there.
  * Watchdog status: Situation is better if scons ports are unwatched, but
    python3* freezes along with a handful of other ports.  It works 99% of
    the time, but not reliably enough yet to re-enable.
```


----------



## talsamon (Jan 24, 2016)

Seems `synth status` does not what I want. It tells me 
	
	



```
Total packages that would be built: 1436
```
 and `pkg version|wc -l` says 1426 (And I don't want rebuild all packages before 11.0-RELEASE).


----------



## marino (Jan 24, 2016)

You have zero packages.
If you upgrade system, it will build all of them.  You should expect this.
Obviously it builds more than it installs.  It will not install build dependencies.  So you should also expect the number built to be greater than the number currently installed.

The option to use prebuilt packages if available is not there yet.

So it seems to be this is exactly what you are asking for.  The first time will be a full build (1400+ packages).  The next time will be incremental.

Right?  What else could you be expecting?


----------



## fernandel (Jan 24, 2016)

talsamon said:


> Seems `synth status` does not what I want. It tells me
> 
> 
> 
> ...



It is not bad. I did before bed time and in the morning was done .


----------



## fernandel (Jan 24, 2016)

I have one question as former portmaster user. What should I do in case:


> Default implementation of jpeg has been switched from graphics/jpeg to
> graphics/jpeg-turbo.  To perform the upgrade, use instructions below.
> 
> If using binary packages: 'pkg upgrade' will do the right thing.  If it
> ...



Thank you.


----------



## marino (Jan 24, 2016)

I don't think you do anything special.  The standard `synth upgrade-system` command should do it (it sets up the local repository and pkg(8) will upgrade from it, and it will "do the right thing" according to your text).


----------



## Chris_H (Jan 25, 2016)

Great news, marino@ , about the new revision!
regarding:


> Similarly, change linprocfs mounts/dismounts to only occur when when
> linux ports are building.  Linprocfs stability is unknown (and I can't
> test it on DF) so be conservative and use it as little as possible.


I can confirm this ro have been an issue for me. I'll be happy to provide any feedback/experiences on this, to help. 

Thanks again, for all the time, and effort you've put into this, John!

--Chris


----------



## talsamon (Jan 25, 2016)

> > What else could you be expecting?



Don't understand why recompile. That is not necesseraily, if packages exists yet.


----------



## talsamon (Jan 25, 2016)

Another question is, how to exclude something from update?


----------



## marino (Jan 25, 2016)

> Don't understand why recompile. That is not necesseraily, if packages exists yet.


What packages?  You starts with zero (0) packages in the local repository when synth is first installed.  Before you can upgrade the system with *only* synth-built packages, you have to actually build the packages.   Remember, this is the command you are issuing.  (note that just because the package is built, that doesn't mean it's used.  pkg(8) can easily decide the installed package satisfies the requirement)



> Another question is, how to exclude something from update?


I think you are getting confused with the commands.  `synth status` is a specific command that means "what would happen if I rebuilt everything with `synth upgrade-system`.  By definition, mean everything that is on the system currently and you can't change the definition or action of this command.

If you want to "exclude" then you need to feed in a list of ports to build manually.  E.g. output the list from pkg(8) to a file and remove the ones you don't want and then feed that list back to synth (e.g. `synth build <filename>` or `synth status <filename>`).


----------



## marino (Jan 25, 2016)

talsamon: and one more thing:  You only need list the ports you want.  You don't need to list the dependencies because those will be calculated and added automatically.  So maybe something like `pkg prime-list > /tmp/myfile` would work for you to get a list of packages you actually want.


----------



## talsamon (Jan 25, 2016)

This will be a long long thread....
`synth` does not mount linprocfs(5), can't build Linux depend ports.
Make the /usr/compat dir, make the link, linprocfs(5) is normal mounted. What to do?

=>

```
00:00:01 [--] skipped  x11/nvidia-driver-304  --:--:--
00:00:01 [--] ignored  emulators/linux_base-c6
```


```
cat /var/log/synth/03_ignored_list.log
00:00:01 Reason: linuxulator is not (kld)loaded
```


```
kldstat
.....
11  1 0xffffffff82a11000 9da0  linprocfs.ko
l....
```


```
cat /var/log/synth/04_skipped_list.log
x11/nvidia-driver-304 by emulators/linux_base-c6
```


why?


----------



## marino (Jan 25, 2016)

linprocfs(5) is specifically mentioned in the changes of 0.98_5, so saying it does not support linprocfs(5) is false.  It *does* mount it, under the specific case that USE_LINUX is defined.  If a port fails to define that when it does need Linux, then it's a ports problem.  It used to be mounted always.

Due to instability with procfs(5) in general, it is not mounted when it's not needed.

That being said, if emulators/linux_base-c6 doesn't define USE_LINUX, then we need an additional detection for that.


----------



## talsamon (Jan 25, 2016)

Where is USE_LINUX to define?  I don't really understand your answer.
Another question:
After I have build ~700 packages:
`synth status`

```
Total packages that would be built: 1347
```

What is this ??


----------



## marino (Jan 25, 2016)

the ports define it.  All linux ports should define it.  *you* don't define it.
But it's possible the linux base doesn't define it.


----------



## Chris_H (Jan 25, 2016)

talsamon said:


> This will be a long long thread....
> `synth` does not mount linprocfs(5), can't build Linux depend ports.
> Make the /usr/compat dir, make the link, linprocfs(5) is normal mounted. What to do?
> 
> ...


I'm not sure why you would see anything like this.
I *just this moment* finished a build, and up(date/grade) of over 1025 ports on my -CURRENT (as of 5 days ago) system.
My system *also* needed to update the nvidia driver, as well as the emulators/linux_base_c6 && all that depend upon it. My fstab(5) has always had the linproc file system listed/mounted.
The upgrade just finished with "flying colors". Bouncing the box with the up(dated|graded) system && ports parts in place, was without event/error -- All is well! 

Puzzled.

--Chris


----------



## talsamon (Jan 25, 2016)

Nothing is well, as you reed above. I have edited my last post......



> After I have build ~700 packages:
> 
> synth status
> Total packages that would be built: 1347



In the moment seems the tool is not useable.
Or explain me how to use. I can't see it . Does linux-ports work or not , and why it don't work on my system.
I have `linprocfs` in /etc/fstab and with `kldload` loaded. What happend?


----------



## marino (Jan 25, 2016)

talsamon: You reported a regression due to a recent change.  I'll already have a fix internally that hasn't been published yet.  Is that clear?

Of course the tool is "usable".  I don't have Linux packages and it works fine.  It's broken for the moment specifically (and only) on emulators/linux_base-c6 and emulators/linux_base-f10, but that doesn't mean it's not usable.


----------



## Chris_H (Jan 25, 2016)

marino@ regarding: 0.98_5 -- smooth as silk!
Just completed updating my box, using that version. Everything went flawlessly. Failures were handled gracefully, as well as all other events. Well; save the screen corruption, due to (kernel) messages being sent to it. But I didn't see anything in the Changelog, to indicate there were any changes in that regard -- wishful thinking on my part. 

Thank you *very* much, John. My dev box thanks you, also. 

--Chris


----------



## marino (Jan 25, 2016)

Chris_H said:


> I'm not sure why you would see anything like this.
> I *just this moment* finished a build, and up(date/grade) of over 1025 ports on my -CURRENT (as of 5 days ago) system.
> My system *also* needed to update the nvidia driver, as well as the emulators/linux_base_c6 && all that depend upon it. My fstab(5) has always had the linproc file system listed/mounted.



You probably already have the port built so it's not trying to build it again.


----------



## marino (Jan 25, 2016)

Chris_H said:


> marino@ regarding: 0.98_5 -- smooth as silk!
> Just completed updating my box, using that version. Everything went flawlessly. Failures were handled gracefully, as well as all other events. Well; save the screen corruption, due to (kernel) messages being sent to it. But I didn't see anything in the Changelog, to indicate there were any changes in that regard -- wishful thinking on my part.



Check github, screen corruption has been fixed, but not released yet.
I might push 0.99 soon though, for linux, and the new "fetch prebuilt packages" feature.


----------



## talsamon (Jan 25, 2016)

> but that doesn't mean it's not usable.


 <= not this (linux).
But this:


> After I have build ~700 packages:
> 
> synth status
> Total packages that would be built: 1347


This means I have lost  7 hours and had to begin from the start.


----------



## marino (Jan 25, 2016)

what do you mean "lost 7 hours"?
You didn't lose anything
When I push the next update, you can continue where it left off.


----------



## Chris_H (Jan 25, 2016)

talsamon
Ahh... I *just* saw the difference; on *my* system I have in /boot/loader.conf
linux_load="YES", as well as having linux_enable="YES" in rc.conf(5).

Hope this helps.

--Chris


----------



## talsamon (Jan 25, 2016)

If `synth status` tells me after I build 700 packages , there are 1347 packages to do (I have 1436 packages on my system), means that I have to begin from the start. Something not works.


----------



## marino (Jan 25, 2016)

Chris, it's not mounting linprocfs internally.  It doesn't matter what the system has.

Talsamon, why guess?  cd to the packages directory and look at the new packages.  They are there, not lost.  You do not have to restart from the beginning.  

And you know you always build more packages than you install, right?  They are called "build dependencies".  You need to build them, but you don't install them.


----------



## Chris_H (Jan 25, 2016)

talsamon
Just for the record, on *my* system, it always started from where it left off. It never had to *re*build any of the ports it had already built, _unless_ the build had failed. This also goes for packages I already had that were _current_.
This was my _recent_ experience, anyway.

Hope this helps.

--Chris


----------



## marino (Jan 25, 2016)

I've booted in FreeBSD.  I'm going to update it to the version in GitHub.  When I verify that linux_base-c6 builds (and maybe a Linux port or 2) I'll push synth v0.99 (release candidate).


----------



## Chris_H (Jan 25, 2016)

> Chris, it's not mounting linprocfs internally. It doesn't matter what the system has.


Oh. I see. I was just appreciating the differences from what talsamon showed, and my system. As my system happily built linux_base_c6 && nvidia-driver-304.

Thanks for the clarification, John.

--Chris


----------



## marino (Jan 25, 2016)

Chris_H said:


> As my system happily built linux_base_c6 && nvidia-driver-304.



past tense.  It probably doesn't build c6 base now (if linprocfs really is required)


----------



## Chris_H (Jan 25, 2016)

marino@ said:


> You probably already have the port built so it's not trying to build it again.


Nope. The /usr/ports/packages folder didn't even exist, until I created it, prior to launching ports-mgmt/synth. Which is why I was confused by talsamon's post.

--Chris


----------



## talsamon (Jan 25, 2016)

```
ls /var/synth/live_packages/All/|wc -l
  219
```
Sorry, 7 hours lost.


----------



## marino (Jan 25, 2016)

Chris_H said:


> Nope. The /usr/ports/packages folder didn't even exist, until I created it, prior to launching ports-mgmt/synth. Which is why I was confused by talsamon's post.



synth(1) doesn't use that directory.  launch `synth configure` if you don't know where the packages directory is.

I can reproduce this kldloaded error.  I think it's a chicken-egg issue with ports variables (it's not really what the issue is)


----------



## talsamon (Jan 25, 2016)

```
Math/atlas ignored
00:00:03 Reason: has to be built manually: Optimizes for the local machine
```

What is this?


----------



## marino (Jan 25, 2016)

talsamon said:


> Sorry, 7 hours lost.


Again, what is it that you lost?
Maybe you only built 219 ports in 7 hours.  you should have timestamps on all the logs to confirm.


----------



## marino (Jan 25, 2016)

talsamon said:


> ```
> Math/atlas ignored
> 00:00:03 Reason: has to be built manually: Optimizes for the local machine
> ```
> ...


Exactly what it says.  The port says it can't be built as a package.
This is a PORTS SPECIFIC issue.
You might want to start looking at the port Makefiles for questions like that.


----------



## talsamon (Jan 25, 2016)

```
ls /var/log/synth/|wc -l
  925
```

Where are the rest?

Edit: had I to buld for each package one thousand of dependencies and the next time once more?


----------



## protocelt (Jan 25, 2016)

emulators/linux_base-c6 does not build for me with version 0.98_5 as expected. (Just tried)

Not to break up a good conversation, but is the feature for using official packages when possible going to be optional? I assume so, but would just like to know for sure. If that's been mentioned already I missed it.


----------



## marino (Jan 25, 2016)

I don't know what "optimal" means. 
If the package meets the requirements, it downloads it.  It might not build anything at all, but rather download everything.


----------



## talsamon (Jan 25, 2016)

It is horrible. I want to make the packages in more parts not "in one". Now I start again and it build the third time the compiler, so I come never to an end.


----------



## marino (Jan 25, 2016)

talsamon said:


> ```
> ls /var/log/synth/|wc -l
> 925
> ```
> ...



I can't parse your "edit", I don't know what you are asking.

Did you update your ports tree after you started building?  You know that if something like perl or python gets updated, it pretty much wipes out all the packages on the next build and you have to rebuild almost everything right?  That's how these things cascade.


----------



## talsamon (Jan 25, 2016)

Why it wipes out the packages I made?
How it works?


----------



## protocelt (Jan 25, 2016)

marino@ said:


> i don't know what "optimal" means.


I wrote "optional"


----------



## marino (Jan 25, 2016)

talsamon said:


> Ir is horrible. I want to make the packages in more parts not "in one". Now I start again and it build the third time the compiler, so I come never to an end.



Unless you are using "force", it does not rebuild packages unless it has to.  I'm not sure what you are doing, but what you are describing is not happening to anyone else.  You probably need to describe EXACTLY the process that you are using.


----------



## marino (Jan 25, 2016)

talsamon said:


> Why it wipes out the packages I made?
> How it works?


I can't keep up with this and try to fix linux at the same time.  Can you not make 100 posts?


----------



## marino (Jan 25, 2016)

protocelt said:


> I wrote "optional"



Yes, it is optional and turned off by default.


----------



## talsamon (Jan 25, 2016)

The only thing which is not updated is the devel/qt5 , all other ports are updated (with the  devel/qt5 update there was yesterday a problem).


----------



## protocelt (Jan 25, 2016)

talsamon, it should probably be noted here that local modifications to port Makefiles can undoubtedly affect building some packages at some point depending on what you changed and are outside the control of synth(1). Maybe that is part of the problem here? I've not had any such problems as your having on my machines. If you do have local modifications you should start with a clean slate and the issues may go away.


----------



## talsamon (Jan 25, 2016)

Local modifications should not matter to synth(1), that is against the port philosophy. This would mean I can nothing change  in the ports.
Now I try to make all packages  for all ports I have installed at once (this is what I not want, cause I can't user my machine not for hours). If this not will work I give it up.


----------



## marino (Jan 25, 2016)

I think he means that you may have modified the ports tree, not that you are using custom options.


----------



## talsamon (Jan 25, 2016)

But every second user modify ports. And there are not really many ports I have changed (mostly not important).


----------



## marino (Jan 25, 2016)

Okay, I've confirmed what caused the linux regression, I'll push the fix shortly and make a new release of synth(1).


----------



## marino (Jan 25, 2016)

talsamon said:


> But every second user modify ports.



So you use a text editors and change Makefiles?

Have you forked ports?

No, only developers "modify" ports.  Users use ports.


----------



## talsamon (Jan 25, 2016)

Okay, thank you.


----------



## talsamon (Jan 25, 2016)

No, I have not forked ports,  only changes in Makefiles or patches.


----------



## Chris_H (Jan 25, 2016)

marino@ said:


> synth doesn't use that directory.  launch `synth configure` if you don't know where the packages directory is.
> 
> I can reproduce this kldloaded error.  I think it's a chicken-egg issue with ports variables (it's not really what the issue is)


Sorry, John. I know you're "buried" right now.
But just for the record; synth *does* use that directory because I *told* it to when I first fired up synth. I used `synth configure`. I knew to do this *first*, because I read this thread, and all the _informative_ information you thoughtfully provided on your GitHub site. 

Thank you.

--Chris

I'll shut up now.


----------



## marino (Jan 26, 2016)

I've pushed version 0.99 which should fix the linux issue:


```
ports-mgmt/synth: v0.99 (RC), fixes / use official pkgs feature

Now Synth has all the features envisioned for the first release.  This
edition add an option (off by default) to fetch prebuilt packages if
they are suitable (ABI, options, dependencies match).  This feature is
aimed at people that only want to build ports with customized options,
but for ports with default configurations, they are happy to use the
official packages.  The feature is not heavily tested yet.

Once this version is sufficently tested, Release 1.00 will follow.

Other changes since 0.98_5:
  * curses will redrawn itself (correctly now) every 30 seconds to
    fix any corruption that may have occured
  * Synth everything will build the repository without deleting packages
    now (this was disabled on the last update)
  * The repos directory for pkg is read from pkg config instead of using
    the hardcoded defaults
  * The man page has been updated with new feature descriptions
  * linprocfs is mounted for linux_base ports
  * The regression that prevented linux ports from building has been
    fixed.  It was caused by caching LINUX_OSVERSION, so this variable is
    no longer cached.
```


----------



## talsamon (Jan 26, 2016)

Seems the explanation for my problem is/was:
I tried to make the packages in parts , and this seems not to work. You have do it at once.  In the moment it looks this works, I will report later if it is solved.
Thanks for patience and the update.


----------



## fernandel (Jan 26, 2016)

On my system synth(1) works not bad and I am satisfied much much more than with portmaster(8).
I have some skipped ports and libc++ has still problem with dependencies check.
Skipped log:

```
x11/gnome-shell by math/openblas
x11/gnome-shell-extensions by math/openblas
x11/gnome-terminal by math/openblas
graphics/mypaint by math/openblas
devel/py-pyopencl by math/openblas
deskutils/gnome-shell-extension-audio-output-switcher by math/openblas
deskutils/gnome-shell-extension-calculator by math/openblas
deskutils/gnome-shell-extension-coverflow by math/openblas
deskutils/gnome-shell-extension-dashtodock by math/openblas
deskutils/gnome-shell-extension-filesmenu by math/openblas
deskutils/gnome-shell-extension-hidetopbar by math/openblas
deskutils/gnome-shell-extension-lockkeys by math/openblas
math/py-matplotlib by math/openblas
deskutils/gnome-shell-extension-mediaplayer by math/openblas
deskutils/gnome-shell-extension-openweather by math/openblas
math/suitesparse by math/openblas
deskutils/gnome-shell-extension-overlay-icons by math/openblas
deskutils/gnome-shell-extension-panel-osd by math/openblas
deskutils/gnome-shell-extension-trash by math/openblas
math/py-numpy by math/openblas
deskutils/gnome-shell-extension-weather by math/openblas
deskutils/gnome-shell-extra-extensions by math/openblas
math/octave by math/openblas
graphics/gimp by math/openblas
graphics/gimp-ez-perspective-plugin by math/openblas
textproc/ibus by math/openblas
sysutils/gnome-control-center by math/openblas
graphics/py-gimp by math/openblas
devel/py-notify by math/openblas
x11/gdm by math/openblas
x11-toolkits/py-gtk2 by math/openblas
multimedia/totem by math/openblas
sysutils/gnome-settings-daemon by math/openblas
graphics/gimp-resynthesizer by math/openblas
graphics/gimp-refocus-plugin by math/atlas
```

Thank you.


----------



## talsamon (Jan 26, 2016)

No, does not work ....


----------



## Chris_H (Jan 26, 2016)

marino@ -- Just a quick thank you, for all the time, work, and support you're putting into this.
The new addition, that corrects "corrupt screens", is the icing on the cake. IMHO ports-mgmt/synth is now _complete_. Thanks!

--Chris


----------



## marino (Jan 26, 2016)

fernandel said:


> On my system synth(1) works not bad and I am satisfied much much more than with portmaster(8).
> I have some skipped ports and libc++ has still problem with dependencies check.



1) libc++ is a problem with the PORT.  Until the port is fixed, it's going to keep rebuilding.  It's been reported.

2) if everything is skipping because of math/openblas, that means math/openblas failed to build.  What you would do is inspect the openblas log and find out why.  Synth is working normally.  that's what is supposed to happen.


----------



## marino (Jan 26, 2016)

talsamon said:


> No, does not work ....


You will have to do a *much* better job at communicating what issues you are having.  Saying "it doesn't work" or mysterious comments like "I was building in parts" is not at all useful. We have no idea what you are doing, or what is failing.

e.g. are packages rebuilding again on a new run?  If so, WHICH package EXACTLY?
Try turning off ncurses mode and seeing the text.  It might say which packages are being removed and the general reason of it.

The only reason for packages to get removed is if a dependency is removed so you need to figure out exactly which one it causing it.  (This is a guess because you have been extremely vague about what the problem actually is)


----------



## PacketMan (Jan 26, 2016)

So am I right in thinking that in the latest version (0.99) synth will fetch prebuilt packages if I have not deviated from the defaults? But if I have changed some of the defaults it will download the source and do a compile? In other words I can save lots of time by compiling only the ports that I have changed the defaults on? Is that correct?


----------



## marino (Jan 26, 2016)

PacketMan said:


> So am I right in thinking that in the latest version (0.99) synth will fetch prebuilt packages if I have not deviated from the defaults?


PacketMan, yes, you understand the situation exactly right.
However, you have to set option "[N]" on `synth configure` to "true" to enable the feature.

Twice I've hit a message where it says it failed to download something and stopped before the build, but it both cases the error message was false and running the command again completed the task.  (FYI, this feature is brand new thus it's not tested very much)


----------



## marino (Jan 26, 2016)

PacketMan said:


> So am I right in thinking that in the latest version (0.99) synth will fetch prebuilt packages if I have not deviated from the defaults?



Let me expand on this.  It's not only the defaults that it's checking.  It's checking ABI (this is almost always good though) and the dependencies that the remote package wants.  It could be that a dependency has changed thus invalidating the remote package.  so really the two main reasons for building from source are different / mismatching options or a failure cascade from a dependency.


----------



## PacketMan (Jan 26, 2016)

Sold!!  My single biggest beef with using ports and portmaster was the recompiling of every port every time there were updates to be done.  I will start to test this in my environment and report what I find. 



marino@ said:


> PacketMan, yes, you understand the situation exactly right.
> However, you have to set option "[N]" on `synth configure` to "true" to enable the feature.



Understood......stay tuned/


----------



## kpa (Jan 26, 2016)

PacketMan said:


> Sold!!  My single biggest beef with using ports and portmaster was the recompiling of every port every time there were updates to be done.  I will start to test this in my environment and report what I find.
> 
> 
> 
> Understood......stay tuned/



Note that synth is not really different to anything else there in this regard, if an important and widely used port has a new version there is absolutely no way to avoid rebuilding everything that depends on that port directly or indirectly. There was lot of discussion about that with Poudriere and initially Poudriere tried to avoid rebuilding everything but that resulted in broken builds because of subtle changes at binary level that can not be detected automatically.


----------



## Chris_H (Jan 26, 2016)

protocelt said:


> emulators/linux_base-c6 does not build for me with version 0.98_5 as expected. (Just tried)
> 
> Not to break up a good conversation, but is the feature for using official packages when possible going to be optional? I assume so, but would just like to know for sure. If that's been mentioned already I missed it.


protocelt -- this is interesting to me. It was thought that this package was simply fetched from the official ports repo, when I declared that synth built it, and the nvidia-driver for me. But I just checked, and this is what pkg(8) told me:

```
linux_base-c6-6.6_6
Name  : linux_base-c6
Version  : 6.6_6
Installed on  : Mon Jan 25 14:13:34 2016 PST
Origin  : emulators/linux_base-c6
Architecture  : freebsd:11:x86:64
Prefix  : /compat/linux
Categories  : linux emulators
Licenses  :
Maintainer  : emulation@FreeBSD.org
WWW  : UNKNOWN
Comment  : Base set of packages needed in Linux mode for i386/amd64 (Linux CentOS 6)
Annotations  :
   repo_type  : binary
   repository  : Synth
Flat size  : 170MiB
```
Note the "repository", it says "Synth".
I say "interesting", because I wonder what would/might allow it to be built on one system, but not another?

--Chris


----------



## protocelt (Jan 26, 2016)

Yeah, I may have something going on on my system. lang/gcc6-aux just failed to build during a `# synth upgrade-system` as well. Looking into it now.


----------



## Chris_H (Jan 26, 2016)

I _should_ have noted that this was on synth 0.98_5. Sorry, and good luck, protocelt! 

--Chris


----------



## protocelt (Jan 26, 2016)

So it looks like the lang/gcc6-aux is spitting out some assembler errors for missing instructions. This is above my pay grade so I may need to file a PR against the port. This of course prevents synth(1) from being upgraded as well.

For reference: lang___gcc6-aux.log


----------



## Chris_H (Jan 26, 2016)

protocelt said:


> So it looks like the lang/gcc6-aux is spitting out some assembler errors for missing instructions. This is above my pay grade so I may need to file a PR against the port. This of course prevents synth(1) from being upgraded as well.
> 
> For reference: lang___gcc6-aux.log


Well, protocelt , looks like you're _also_ tracking -CURRENT. I'm on r294112 (kernel), checked out on 2016-01-15, and on PORTS revision 406193. My lang/gcc6-aux is version is 20151227, which is the same as yours. We've used the same OPTIONS. Mine built && installed fine. SYNTH build a package for it. If you're even remotely interested, I can provide a link to it, and you can download it to your repo, and install it from there, if you wish.  I can also provide the SYNTH log for it.

All the best, protocelt!

--Chris


----------



## marino (Jan 26, 2016)

Chris_H said:


> protocelt -- this is interesting to me. It was thought that this package was simply fetched from the official ports repo, when I declared that synth built it, and the nvidia-driver for me. But I just checked, and this is what pkg(8) told me:
> Note the "repository", it says "Synth".
> I say "interesting", because I wonder what would/might allow it to be built on one system, but not another?



There's no mystery.
1) Synth needs a package and prebuilt option is active
2) Synth queries remote repository and finds a suitable package
3) Synth tells pkg(8) to download the package and stick it in Synth's package directory
4) Synth rebuilds local repository
5) The Synth repository shows the downloaded package and it now owns it.

if/when the system is updated, both the external repo and synth repo have the package, but it will get installed from the synth repo which has higher priority.


----------



## marino (Jan 26, 2016)

protocelt said:


> So it looks like the lang/gcc6-aux is spitting out some assembler errors for missing instructions. This is above my pay grade so I may need to file a PR against the port. This of course prevents synth(1) from being upgraded as well.
> For reference: lang___gcc6-aux.log



What system?  I can't fetch that log.   I did a small build test on the new gcc6-aux before commit it and everything worked alright.


----------



## Chris_H (Jan 26, 2016)

marino@ said:


> What system?


I have the log he linked to in one of the tabs in my browser. It says:

```
Platform: 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r294762: Mon Jan 25 23:57:45 CST 2016     root@ZION:/usr/obj/usr/src/sys/GENERIC-NODEBUG  amd64
```
You weren't able to reach that link?

--Chris


----------



## protocelt (Jan 26, 2016)

marino@ said:


> What system?  I can't fetch that log.   I did a small build test on the new gcc6-aux before commit it and everything worked alright.


The log is a pastebin link(http://pastebin.com/AeP041S6)

Also added as an attachment to this post.


----------



## marino (Jan 26, 2016)

Chris_H said:


> I have the log he linked to in one of the tabs in my browser. It says:
> You weren't able to reach that link?



No, I still can't.

```
This webpage is not available


DNS_PROBE_FINISHED_NXDOMAIN
```


----------



## marino (Jan 26, 2016)

Okay, my first build attempt was gcc6-aux on poudriere in Release 10.2 jail.  It built fine.
I suspect this is a CURRENT regression.  Just for fun, I'll try on synth on FreeBSD 10.2 release system, and I expect it to succeed.


----------



## marino (Jan 26, 2016)

By they way, ports builders already have gcc6-aux-20160124 on 9.3 and 10.2, both amd64 and i386 platforms.


----------



## marino (Jan 26, 2016)

gcc6-aux builds fine in synth on 10.2-amd64 system as well.
This looks like a FreeBSD-CURRENT regression.  If it's old, maybe it's time to update the system (the regression may have been fixed since the last update)


----------



## marino (Jan 26, 2016)

> Mon Jan 25 23:57:45 CST 2016



That's not old at all.  Regression might be very recent.


----------



## protocelt (Jan 26, 2016)

marino@ said:


> No, I still can't.
> 
> ```
> This webpage is not available
> ...


...Not sure why the site is being blocked however we try to discourage people from posting long logs directly to posts here as not to clutter up threads. I'll send you a PM with the log contents to read since your accessing the forums fine.


----------



## marino (Jan 26, 2016)

I could read the attachment, I've already seen it.
I just can't reproduce the error on releases.
The only recent "CURRENT" have I is the latest i386 version and that's a couple of days older than what you are showing.


----------



## Chris_H (Jan 26, 2016)

marino@ said:


> No, I still can't.
> 
> ```
> This webpage is not available
> ...


FWIW this is what I get:

```
drill pastebin.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 8889
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 4
;; QUESTION SECTION:
;; pastebin.com.  IN  A

;; ANSWER SECTION:
pastebin.com.  300  IN  A  104.20.63.56
pastebin.com.  300  IN  A  104.20.64.56

;; AUTHORITY SECTION:
pastebin.com.  112119  IN  NS  todd.ns.cloudflare.com.
pastebin.com.  112119  IN  NS  sue.ns.cloudflare.com.

;; ADDITIONAL SECTION:
sue.ns.cloudflare.com.  112119  IN  A  173.245.58.145
todd.ns.cloudflare.com. 31240  IN  A  173.245.59.146
sue.ns.cloudflare.com.  112119  IN  AAAA  2400:cb00:2049:1::adf5:3a91
todd.ns.cloudflare.com. 31240  IN  AAAA  2400:cb00:2049:1::adf5:3b92
```


----------



## protocelt (Jan 26, 2016)

Ok. I haven't tried on 10.2-STABLE or 10.2-RELEASE yet so I'll take your word it builds fine. Since this is an issue with CURRENT, not sure where to go from here other than the mailing lists.


----------



## marino (Jan 26, 2016)

protocelt said:


> Ok. I haven't tried on 10.2-STABLE or 10.2-RELEASE yet so I'll take your word it builds fine. Since this is an issue with CURRENT, not sure where to go from here other than the mailing lists.


Maybe monitor portsmon --  it must try to build gcc6-aux on CURRENT very soon.  And I'll get fallout logs mailed to me if they fail.  And if they succeed?  Then I'd guess the jail is older than what you have, prior to the regression. Either way that's useful information.


----------



## protocelt (Jan 26, 2016)

Sounds good. I'll keep an eye out. Thanks.


----------



## PacketMan (Jan 26, 2016)

kpa said:


> Note that synth is not really different to anything else there in this regard, if an important and widely used port has a new version there is absolutely no way to avoid rebuilding everything that depends on that port directly or indirectly. There was lot of discussion about that with Poudriere and initially Poudriere tried to avoid rebuilding everything but that resulted in broken builds because of subtle changes at binary level that can not be detected automatically.



Any time saved, is time saved. 

But now I gotta ask:  By using the [N] option do I run the risk of a messed up system because synth rebuilt/compiled some ports, but for other just straight downloaded the package?

And if the answer is "yes" then this is not really caused by synth but simply by packages not being updated as often as ports right? In other words no different then me manually using port build for some and packages for others.  (Which is what got me in trouble a year ago when I was new to FreeBSD.) But I'm not installing here, I am just upgrading so does the scenario still apply?

I'm willing to use the [N] option on one of my machines (FreeBSD 10.2-RELEASE + XCFE4 + some other stuff) to see what happens over time. Gladly will provide any info that is useful to others,


----------



## marino (Jan 26, 2016)

PacketMan said:


> But now I gotta ask:  By using the [N] option do I run the risk of a messed up system because synth rebuilt/compiled some ports, but for other just straight downloaded the package?



Not in my opinion.  The dependencies to exact package names, options configuration, and ABI are all verified.  they should work just as if they were built by synth.  In theory.


----------



## PacketMan (Jan 27, 2016)

marino@ said:


> Not in my opinion.  The dependencies to exact package names, options configuration, and ABI are all verified.  they should work just as if they were built by synth.  In theory.



We're gonna find out. Hang on!


----------



## talsamon (Jan 27, 2016)

> No, does not work ...


#
was only a loud sigh.

After two further failed tries, now it works. I don't really know was it caused it, or what I am made wrong.

Only two packages failed (editors/abiword and sysutils/htop). But that's no problem, both are buggy and I can live without them.
Interesting the error messages of sysutils/htop:

```
htop(1) requires linprocfs(5) to be mounted. If you don't
have it mounted already, please add this line......
```

but all other  linux-programs works.


----------



## talsamon (Jan 27, 2016)

Only problem left:


```
jackit-0.124.1_5.txz failed dependency check.
ffmpeg-2.8.5,1.txz failed dependency check.
gstreamer1-libav-1.6.3.txz failed dependency check.
fluidsynth-1.1.6_2.txz failed dependency check.
mplayer-1.2.r20151219_2.txz failed dependency check.
opencv-2.4.9_7.txz failed dependency check.
alsa-plugins-1.1.0.txz failed dependency check.
chromaprint-1.1.txz failed dependency check.
gstreamer1-plugins-opencv-1.6.3.txz failed dependency check.
gstreamer1-plugins-chromaprint-1.6.3.txz failed dependency check.
gstreamer1-plugins-jack-1.6.3.txz failed dependency check.
gstreamer1-plugins-all-1.4_3.txz failed dependency check
```

It does not find one (or more) dependcies of gstreamer & co.
Don't know how to locate this. Synth should say which package is missing.

*Edit: *found one, reduces the list to:

```
jackit-0.124.1_5.txz failed dependency check.
ffmpeg-2.8.5,1.txz failed dependency check.
gstreamer1-plugins-jack-1.6.3.txz failed dependency check
```


----------



## marino (Jan 27, 2016)

htop: That's a run-time message right?  You shouldn't need linprocs to build htop

The "failed dependency check" messages: I think you are confused about these.
It is finding existing packages that are no longer useful.  It doesn't matter exactly why they are not useful.  If not a "status" command it will DELETE these packages and then rebuild them.

If these package show up as failed check after rebuilding, then there's a problem with the port itself (which is quite possible).  In cases like that, I have a debug mode that can tell me exactly why and I can fix the port.


----------



## talsamon (Jan 27, 2016)

Debug-mode? `man synth` says nothing about that.


----------



## marino (Jan 27, 2016)

I said *I* can do it.  I did not say it is a user feature.  It would be used to help identify a problem with a port.  It is out of scope of a synth user, they don't fix ports.


----------



## talsamon (Jan 27, 2016)

I think it is a problem with audio/jack - maybe it is the name. The package is named jackit and the port jack.
The problem is: it tries each run to rebuild 10 or 12 audio/multimedia related packages.


----------



## marino (Jan 27, 2016)

the same 12, over and over?


----------



## talsamon (Jan 27, 2016)

`htop` Makefile:

```
CONFIGURE_ARGS= --with-proc=/compat/linux/proc
```


----------



## talsamon (Jan 27, 2016)

Yes, the same 12.


----------



## marino (Jan 27, 2016)

okay, synth will have to be modified to support htop.
I'll have to confirm jack rebuilds over and over and if confirmed, fix the port.


----------



## marino (Jan 27, 2016)

I've pushed version 0.99_1 to ports:


```
ports-mgmt/synth: minor fixes plus linprocfs fix

1) When using prefetch option, list the packages that failed to download
   rather than just say, "at least one failed to download"
2) sysutils/htop requires linprocfs but doesn't set USE_LINUX.  Set this
   port to mount linprocfs based on its origin
3) Fix linprocfs implementation, it was mounting out of order, basically
   resulting in that it was non-functional
4) Close all the logs in the case where no packages are built.  In that
   case, the logs were never modified.  Changes discarded?
```


----------



## marino (Jan 27, 2016)

talsamon: please check if version 0.99_1 solves the htop issue for you.

everyone else: While testing on FreeBSD release 10.1 (not 10.2) I noticed synth just exits leaving up the curses screen.  It's not supposed to do that, it is supposed to restore the terminal to how it was before synth was executed.  Morever, that's hiding the "tally" that shows a summary of the build results.

Is everyone seeing the same thing?  Or do some people see the terminal restore as expected?  I was using syscons, not a vt.


----------



## marino (Jan 27, 2016)

talsamon said:


> *Edit: *found one, reduces the list to:
> 
> ```
> jackit-0.124.1_5.txz failed dependency check.
> ...



talsamon: I can't reproduce a problem with these 3 ports.  I build them, then run "just-build" and synth says everything is fine (no package is pruned).

Have you customized options on jack or the other two ports?  Maybe it works fine with default options but you are tickling a port problem with custom options.  I'm just guessing here.


----------



## talsamon (Jan 27, 2016)

emulators/virtualbox-ose failed:

```
.... -L/usr/local/lib  /construction/xports/emulators/virtualbox-ose/work/VirtualBox-4.3.34/out/freebsd.amd64/release/bin/VBoxVMM.so  /construction/xports/emulators/virtualbox-ose/work/VirtualBox-4.3.34/out/freebsd.amd64/release/bin/VBoxRT.so  -lpthread
kmk: *** Waiting for unfinished jobs....
kmk: *** Exiting with status 2
*** Error code 2

The failing command:
```

compiles fine in the port.
devel/ghc failed:

```
/usr/local/bin/ar: creating libraries/haskeline/dist-install/build/libHShaskeline-0.7.2.1-1dVCRhdIH7hAQWJrKwByYv.a
<<ghc: 254116248 bytes, 78 GCs, 4389772/11881320 avg/max bytes residency (5 samples), 32M in use, 0.000 INIT (0.000 elapsed), 0.186 MUT (0.507 elapsed), 0.469 GC (0.219 elapsed) :ghc>>
  HC [stage 1] libraries/Cabal/Cabal/dist-install/build/Distribution/Simple/Test/LibV09.p_o
"rm" -f libraries/haskeline/dist-install/build/libHShaskeline-0.7.2.1-1dVCRhdIH7hAQWJrKwByYv.a.contents
  HC [stage 1] libraries/Cabal/Cabal/dist-install/build/Distribution/Simple/GHCJS.p_o
  HC [stage 1] libraries/Cabal/Cabal/dist-install/build/Distribution/Simple/GHC.p_o
compiler/ghc.mk:655: recipe for target 'compiler/stage2/build/DynFlags.p_o' failed
gmake[2]: *** [compiler/stage2/build/DynFlags.p_o] Killed
gmake[2]: *** Waiting for unfinished jobs....
Makefile:71: recipe for target 'all' failed
```

(VirtualBox looks like a port-error, with GHC I don't think so - both standard options).

I have to test both in the port,
devel/ghc
I have installed a month ago , this time it compiled fine (don't think there were much changes).
The installation of of
emulators/virtualbox-ose
is longer ago.

Jackit: I will test it with standard options, but this need some time.
Edit: Could be, it is a circular dependency with audio/alsa-plugins.

=> We need an exclude option/parameter or exclude-list.
If I had a package which compiles in the port , but not in Synth. Synth tries every time to compile it, it is not practical (and needs a lot of time). And every update make a new "compile.list" is tedious (I hope this is the right word for "mühsam" in german). I can't deinstall all, who is failing in Synth (e.g. emulators/virtualbox-ose). And for ports I have for what reason ever downgraded (see VirtualBox, there are a lot of problems on Bugzilla in the past).


----------



## marino (Jan 27, 2016)

Have you tried the failing ports multiple times?  Sometimes ports aren't really jobs safe.


----------



## talsamon (Jan 27, 2016)

The compiles now runs three days.... let me and the box some time.
Two times each (emulators/virtualbox-ose and lang/ghc). But first time I had several failure with three builders. Second time I try it with one builder.
Now all other failed packages compiles except this two (and jackit and htop of course).


----------



## marino (Jan 27, 2016)

talsamon: there's a new version of synth available (0.99_1) that will build those two ports.  You didn't see the announcement?  Just install it!

*edit: *well, htop anyway.


----------



## talsamon (Jan 27, 2016)

Thank you, I have not look on Freshports the last hours.
Jackit does not work (with standard options) and multimedia/ffmpeg with standard options make no sense. I have removed jack and try now again.

Another question: What means "Impulse"?


----------



## marino (Jan 27, 2016)

audio/jack builds fine for me on 10.1.  What release and arch are you using?


by the way, I announced it here!


----------



## talsamon (Jan 27, 2016)

10.2-RELEASE-p10 FreeBSD 10.2-RELEASE-p10 amd64.

Please, wait with jack, maybe I made an error I try it later, but this need some time, it seems the update of synth recompiles now all compilers.


----------



## marino (Jan 27, 2016)

For now I'm test building emulators/virtualbox-ose and lang/ghc.
"impulse" is described in the howto and the man page.  Basically it's the build rate over the last 500 seconds (versus pkg/hour which is the build rate averaged over the entire elapsed time)


----------



## marino (Jan 27, 2016)

talsamon: emulators/virtualbox-ose built just fine for me.
lang/ghc will take a lot longer to build-check though.


----------



## marino (Jan 27, 2016)

talsamon: Also lang/ghc builds fine for me.
Both of these also built on a DragonFly machine.
Maybe you have a failing drive?  some kind of h/w issue?


----------



## talsamon (Jan 27, 2016)

Very sorry for that, seems it was a swap-error on my machine. For sure I know it after next try.


----------



## PacketMan (Jan 27, 2016)

marino@, I seem to have misunderstood something.  I was expecting `synth status` to tell me that my system is all up to date, but it didn't.  Why does portmaster say one thing but synth something very different?


```
# portmaster -L --index-only| egrep '(ew|ort) version|total install'
===>>> 692 total installed ports
   ===>>> There are no new versions available
```


```
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
These are the ports that would be built:
  => print/indexinfo
  => textproc/expat2
.......
Total packages that would be built: 696
```

Portmaster is saying I am good to go, but synth is saying I have 696 ports failing validity checks??? What am I misunderstanding? I am running 10.2-RELEASE-p7 and synth version 0.99_1.


----------



## talsamon (Jan 27, 2016)

Seems it was not the swap-error. emulators/virtualbox-ose compiles. lang/ghc fails with the same error above.
Can I build a package with ports-mgmt/poudriere and copy it to /var/synth, to silence this?


----------



## talsamon (Jan 28, 2016)

I am short before to give it up. synth(1) compiles the updates but not installs it (`# synth upgrade-system`), maybe that's the reason for the lang/ghc fail, cause I have the old Synth version.
The next update Synth tries to compile 10 or 12 Qt4 or Qt5 packages '(only needed x11-toolkits/qt5-gui and x11-toolkits/qt4-gui). The new versions e.g of devel/binutils or ports-mgmt/synth are in /var/synth but not installed), and .....


```
vlc-2.2.1_6,4.txz failed dependency check.
transmission-gtk-2.84_5.txz failed dependency check.
qt5-declarative-5.5.1.txz failed dependency check.
qgit-qt4-2.3_1.txz failed dependency check.
virtualbox-ose-4.3.34_1.txz failed dependency check.
goldendict-1.0.1_8.txz failed dependency check.
qt5ct-0.21.txz failed dependency check.
stellarium-0.14.2.txz failed dependency check.
xarchiver-0.5.4_1.txz failed dependency check.
```
and more of this.....

What has multimedia/vlc or archivers/xarchiver to do with an update of devel/xdg-utils. Portmaster does not show this dependencies (synth(1) - 53 packages, portmaster(8) - 6 packages), or is now each update the recompile of the half system?

A question on the lowest level:
Tell me the steps to get a proper update with synth(1).


----------



## PacketMan (Jan 28, 2016)

PacketMan said:


> Marino, I seem to have misunderstood something.  I was expecting `synth status` to tell me that my system is all up to date, but it didn't.  Why does portmaster say one thing but synth something very different?
> 
> 
> ```
> ...



Is it because synth(1) uses a completely different directory than portmaster(1)? Is /usr/obj/synth-live  that directory which for me is currently empty? Thus if I do `# synth upgrade-system` it will rebuild everything, and the next time around it will report good to go (assuming all is indeed good to go).

So does this mean I should make a hard switch to synth and once I do so I should stop using portmaster(8)?


----------



## fernandel (Jan 28, 2016)

talsamon said:


> I am short before to give it up. Synth compiles the updates but not installs it (`upgrade-system`), maybe that's the reason for the ghc fail, cause I have the old synth version.
> The next update synth tries to compile 10 or 12 qt4 or qt5 packages '(only needed qt5-gui and qt4-gui). The new versions e.g of binutils or synth are in /var/synth but not installed), and .....
> 
> 
> ...



I got many ports with failed dependency too but before was everything okay. Now I start again with updated version 0.99_1.


----------



## marino (Jan 28, 2016)

PacketMan said:


> Marino, I seem to have misunderstood something.  I was expecting `synth status` to tell me that my system is all up to date, but it didn't.  Why does portmaster say one thing but synth something very different?
> 
> Portmaster is saying I am good to go, but synth is saying I have 696 ports failing validity checks??? What am I misunderstanding? I am running 10.2-RELEASE-p7 and synth version 0.99_1.



Have you ever built packages before?
Remember, Synth is building a REPOSITORY.
Afterwards, it tells pkg(8) to update from that repository

If you haven't done anything, then 1) the repository doesn't exist and 2) if it did, it would be empty.

My assumption is that you haven't built anything yet, so it wants to build the entire thing first (or at least download as many official packages as it can if you use that option)

On the next run, that will be incremental.  On the first run, you have establish the repo.


----------



## marino (Jan 28, 2016)

talsamon said:


> Seems it was not the swap-error. emulators/virtualbox-ose compiles. lang/ghc fails with the same error above.
> Can I build a package with ports-mgmt/poudriere and copy it to /var/synth, to silence this?



talsamon, I'm having a hard time telling you what to do because it seems there is something seriously wrong with your system.  I have nearly the same release as you and I have no problems building things on it or other machines that simply don't build for you.  So whatever advice I give you is based on a normal working machine and you don't have one.

How am I supposed to troubleshoot if I can't trust anything that happens?

Maybe I'll add an option to turn on debug mode to show why the dependencies are failing.  That might help.


----------



## marino (Jan 28, 2016)

fernandel said:


> I got many ports with failed dependency too but before was everything okay. Now I start again with updated version 0.99_1.



Did you update your ports tree?  If so, that would be normal.
(or does your ports tree get updated with a cron job?)


----------



## marino (Jan 28, 2016)

PacketMan said:


> Is it because synth(1) uses a completely different directory than portmaster(1)? Is /usr/obj/synth-live  that directory which for me is currently empty? Thus if I do `# synth upgrade-system` it will rebuild everything, and the next time around it will report good to go (assuming all is indeed good to go).
> So does this mean I should make a hard switch to synth and once I do so I should stop using portmaster(8)?



You have the right idea, but the wrong directory.  It the directory of option [ B ] *, *Packages directory, that needs to be full.

yes, you would use one or the other.  Synth is meant to replace portmaster, and doesn't s share parameters.


----------



## marino (Jan 28, 2016)

I pushed a minor release this morning to ports tree: 0.99_2


```
ports-mgmt/synth: two minor but important bug fixes

1) Fixed false "fetched failed" messages that always appear after
   prebuilt packages are fetched
2) Fix bug where "synth configure" command would not run if any directories
   were invalid.  For new systems, /usr/ports/distfiles is always invalid
3) Following 2), greatly improve error message by saying exactly which
   directory is missing and which configuration letter it pertains to
4) If synth is configured to a non-existent /usr/ports/distfiles directory,
   also add a recommendation to consider a better location outside of the
   ports tree and remind them to set DISTDIR in /etc/make.conf too.
```


----------



## PacketMan (Jan 28, 2016)

marino@ said:


> Have you ever built packages before?


No not on this machine and any of my others, I am a strict user of make and portmaster. But I was sure the end result of make was a built package, I was sure folks here on this forum told me this.  But you are right this will be my first run of synth.  I will upgrade to 0.99_2 first though.


----------



## marino (Jan 28, 2016)

PacketMan said:


> But I was sure the end result of make was a built package, I was sure folks here on this forum told me this.



You were informed incorrectly.  on FreeBSD, a package is only made with the explicit `make package` command.  (On pkgsrc a package is always made and this is arguably a better approach).  

Most users with standard defaults don't have packages on their machine when they build from source.  Things are installed directly from the stage directory and registered I think.


----------



## marino (Jan 28, 2016)

by the way, since you will be using the prebuilt option, it's not actually going to build that many ports.  It will download most of them (or all of them if you don't have custom options).  When it gets to the install stage, pkg will see that they are the same as what is already installed and just ignore them.


----------



## marino (Jan 28, 2016)

> No not on this machine and any of my others,



Did you know you can share this synth repository with all your machines?
You can use one machine for synth and then add a pkg repository conf file on your other machines that point to it.  That will save a ton of time (all have to have the same ABI though, e.g. FreeBSD 10.2/amd64)


----------



## PacketMan (Jan 28, 2016)

marino@ said:


> You were informed incorrectly.  on FreeBSD, a package is only made with the explicit "make package" command.  (On pkgsrc a package is always made and this is arguably a better approach).  most users with standard defaults don't have packages on their machine when they build from source.  Things are installed directly from the stage directory and registered I think.





marino@ said:


> by the way, since you will be using the prebuilt option, it's not actually going to build that many ports.  It will download most of them (or all of them if you don't have custom options).  When it gets to the install stage, pkg will see that they are the same as what is already installed and just ignore them.



This also says that ports creates packages, here see #4:
http://www.wonkity.com/~wblock/docs/html/pkg.html

Yes I am looking forward to trying using the [N] - prebuilt option. I am interested in seeing how much time is saved.  I am optimistic that using ports to first-time build a program and then using synth to maintain updates on the system will be a winning combination. I use older hardware and after a month or so a GNOME3 machine can take easily over 12 to 16 hours (might be closer to 24 hours) to update.

Stay tuned....I just need to wait a few weeks for more updates to hit the ports tree.


----------



## marino (Jan 28, 2016)

wblock@'s FAQ is not correct.  You can test for yourself.  packages have a ".txz" extension, right?
Build and install a port from source, then try something like "`find <wrkdir> -name "*.txz`".  You won't find anything.
I also checked the pkg cache dir and no txz file is added after a test install.


----------



## PacketMan (Jan 28, 2016)

Okay, so if you are right, then anytime I install a port using make then the next time I run synth its gonna burp saying I need to 'do' that one.  So can I use one of those synth options, maybe `synth build port`, to actually build the port.  can I use synth to set any options I want and it will build for me? Would I need to turn off the [N] option, or like during `synth upgrade-system` it automatic determine that options have been changed and build locally? (Or no options have been changed, and download the pre-built package?)


----------



## marino (Jan 28, 2016)

> Okay, so if you are right, then anytime I install a port using make then the next time I run synth its gonna burp saying I need to 'do' that one.


Well, if you use synth(1), you wouldn't use "`make`" anymore, other than "`make config`" to customize options.

Example: say you want to install 3 ports.  You would:
`# synth install <port1> <port2> <port3>`

There are other combinations but that would be the most efficient.

But yes, if you build/install via make(1), then synth(1) will want to rebuild it the next time you run `# synth upgrade-system`


----------



## PacketMan (Jan 28, 2016)

Righto, got it. Thanks a ton, and stay tuned.


----------



## basbebe (Jan 28, 2016)

My `# synth rebuild-repository` is also stuck at lang/gcc6-aux.
I'm on the latest release of synth(1) and FreeBSD-10.2-RELEASE-p11


----------



## marino (Jan 28, 2016)

What is the error?   I've personally confirmed lang/gcc6-aux builds in synth(1) (on 10.1) and it's also building on FreeBSD  cluster according to portsmon.


----------



## basbebe (Jan 28, 2016)

I don't get an error.
It just doesn't seem to finish compiling:


----------



## marino (Jan 28, 2016)

1) There is a build log for every build.  Check the logs directory (see `# synth configure` if you don't know where it is)
2) How do you know it wasn't working?  maybe it was linking which takes a long time.


----------



## marino (Jan 28, 2016)

Also it would be helpful to know things about your machine (arch, memory, filesystem, cpu, etc) old/new/powerful/creaky


----------



## basbebe (Jan 28, 2016)

marino@ said:


> 1) there is a build log for every build.  Check the logs directory (see `synth configure` if you don't know where it is)
> 2) how do you know it wasn't working?  maybe it was linking which takes a long time.


I'm sorry, I'm taking everything back.
After 40 minutes, the build finished.
I was just too impatient.

But this is what I got when it was finished:


```
Scanning existing packages.
Scanning entire ports tree.
mktemp: mkstemp failed on /tmp/[username]/portslicense.867hCAMO: No such file or directory
make: "/xports/Mk/bsd.licenses.mk" line 510: warning: "mktemp -ut portslicense" returned non-zero status
progress: 70.76%
culprit: devel/p5-Perl-Unsafe-Signals
Scan aborted for an unknown reason.
bad input for 'Value: ""
Failed to scan ports tree   (Synth must exit)
```

I'm on FreeBSD 10.2-RELEASE-p11 on a vServer:
2 Cores (Intel Xeon E5-26xxV3)
6GB RAM
SSD


----------



## marino (Jan 28, 2016)

How clean is your ports tree?  Other than possibly distfiles and packages directory, is there anything installed in the tree that it did not come with?

Finally, does that portlicense message mean anything to you at all?


----------



## marino (Jan 28, 2016)

I am also interested in this "/tmp/basbebe"
do you have a non standard tmp area or something like that?


----------



## basbebe (Jan 28, 2016)

marino@ said:


> i am also interested in this "/tmp/[username]"
> do you have a non standard tmp area or something like that?


No I don't…


marino@ said:


> how clean is your ports tree?  Other than possibly distfiles and packages directory, is there anything installed in the tree that it did not come with?
> 
> finally, does that portlicense message mean anything to you at all?


I didn't add anything to the ports tree by myself, it should be clean besides your exceptions.
port license doesn't mean anything to me…


----------



## marino (Jan 28, 2016)

You don't find a tmp subdirectory with the same name as your username in the forum kind of coincidental?


----------



## basbebe (Jan 28, 2016)

marino@
I just entered it as a placeholder username – changed it now.


----------



## marino (Jan 28, 2016)

I don't care about your username, removing it is not help.,
it's that mktemp(1) wants to use /tmp/[username] and that's not normal.  There must be something in environment or some kind of setting causing that.


----------



## kpa (Jan 28, 2016)

marino@ said:


> You were informed incorrectly.  on FreeBSD, a package is only made with the explicit "make package" command.  (On pkgsrc a package is always made and this is arguably a better approach).
> 
> most users with standard defaults don't have packages on their machine when they build from source.  Things are installed directly from the stage directory and registered I think.



Yes, a common source of confusion in FreeBSD ports is package creation vs. package registration. The latter is of course always done when you do `make install` on a port by calling pkg-register(8). The former, creating an actual package file is only done as you say when `make package` is used.


----------



## basbebe (Jan 28, 2016)

marino@ said:


> I don't care about your username, removing it is not help.,
> it's that mktemp wants to use /tmp/[username] and that's not normal.  There must be something in environment or some kind of setting causing that.


Looks like this is prezto setting an environment variable:
https://github.com/sorin-ionescu/prezto/blob/master/runcoms/zprofile


----------



## basbebe (Jan 28, 2016)

I set the env(1) variable manual to `/tmp`

The rest of the error remains:


```
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Failed!
The external repository could not be updated.
No prebuilt packages will be used as a result.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Scanning entire ports tree.
progress: 69.80%
culprit: devel/p5-Perl-Unsafe-Signals
Scan aborted for an unknown reason.
bad input for 'Value: ""
Failed to scan ports tree   (Synth must exit)
```


----------



## marino (Jan 28, 2016)

Hmm, it think synth(1) is external repository.  We can trouble shoot that later, for now turn option [N] off to get rid of the first set of errors (I probably need to add an additional guard in Synth to prevent it).

I'm not sure what's wrong with
devel/p5-Perl-Unsafe-Signals yet


----------



## wblock@ (Jan 28, 2016)

marino@ said:


> wblock's faq is not correct.  You can test for yourself.  packages have a ".txz" extension, right?



This is semantics.  If it wasn't a package, a built port could not be installed by the package system.  That it is not saved as a separate package file is a different matter.

I am willing to reword that document on pkg(8) if it can be explained more clearly.


----------



## marino (Jan 28, 2016)

basbebe can you tell me output of this command?
`/usr/bin/make -C /usr/ports/devel/p5-Perl-Unsafe-Signals  -VPKGVERSION -VPKGFILE:T -VMAKE_JOBS_NUMBER -VIGNORE -VFETCH_DEPENDS -VEXTRACT_DEPENDS -VPATCH_DEPENDS -VBUILD_DEPENDS -VLIB_DEPENDS -VRUN_DEPENDS -VSELECTED_OPTIONS -VDESELECTED_OPTIONS -V_INCLUDE_USES_SCONS_MK -VUSE_LINUX`

Note that I edited it, it was missing /usr/ports after -C


----------



## marino (Jan 28, 2016)

wblock@ said:


> This is semantics.  If it wasn't a package, a built port could not be installed by the package system.  That it is not saved as a separate package file is a different matter.
> 
> I am willing to reword that document on pkg(8) if it can be explained more clearly.


I disagree it's semantics.  Making a package is a specific thing with specific meaning.  A package is not being made.  The installation occurs directly from stage directory.

Case in point: if a package is being made, where is it?


----------



## basbebe (Jan 28, 2016)

marino@ said:


> basbebe can you tell me output of this command?
> `/usr/bin/make -C devel/p5-Perl-Unsafe-Signals  -VPKGVERSION -VPKGFILE:T -VMAKE_JOBS_NUMBER -VIGNORE -VFETCH_DEPENDS -VEXTRACT_DEPENDS -VPATCH_DEPENDS -VBUILD_DEPENDS -VLIB_DEPENDS -VRUN_DEPENDS -VSELECTED_OPTIONS -VDESELECTED_OPTIONS -V_INCLUDE_USES_SCONS_MK -VUSE_LINUX`


The output is empty…
I also tried
	
	



```
make config
make: don't know how to make config. Stop

make: stopped in /usr/ports/devel/p5-Perl-Unsafe-Signals
```

I also found out that the folder "p5-Perl-Unsafe-Signals" is empty. Is removing it and `portsnap fetch update` safe?


----------



## marino (Jan 28, 2016)

Okay, that's the problem.  Your tree is corrupt. 
portsnap(8) may not fix it.   It might be worth deleting the whole thing and reextracting it, but you have to be careful with /usr/ports/distfiles and /usr/ports/packages if those are being used (see why those are stupid default locations?)


----------



## marino (Jan 28, 2016)

Synth should probably be more tolerant of a bad tree, or at least say "hey!, the're no makefile here!"


----------



## talsamon (Jan 28, 2016)

Got all packages to install, update seems to work. Except lang/ghc - is either something with the port or the tmpfs/swap -  I don't know (it states "out of swap" - I don't believe it). For the moment I have marked it as broken (the only way to skip it with Synth).
By the way
sysutils/htop also installs, editors/abiword after I set option `MATHVIEW` to off (should make a bug report).


----------



## basbebe (Jan 28, 2016)

I downloaded a new ports tree to /tmp/ports, deleted anything but distfiles and packages in /usr/ports and moved the new tree back.
Now it seems to work, thanks a lot!


----------



## marino (Jan 28, 2016)

I don't think anything is wrong with lang/ghc as a port, because it builds fine for me in about 55 minutes.


----------



## marino (Jan 28, 2016)

basbebe said:


> I downloaded a new ports tree to /tmp/ports, deleted anything but distfiles and packages in /usr/ports and moved the new tree back.
> Now it seems to work, thanks a lot!



Great!  and I'll use the experience you had to make synth(1) handle cases like that better.


----------



## talsamon (Jan 28, 2016)

No, it works not. Last try seems all ok. I make a new test (no updated package from the server in the meantime):
59 packages to recompile - synth(8) is never to satisfy.
For the hell what does it want?


----------



## marino (Jan 28, 2016)

You absolutely have to identify which package that is always rebuilding.  It is most likely an error in the port itself, but if you don't tell us which one, how can we check?


----------



## talsamon (Jan 28, 2016)

How should I? I have nothing for diagnose. In the ports I have pkg_libchk(1), but with synth(1)?


----------



## marino (Jan 28, 2016)

Turn off ncurses mode using `# synth configure` and log the output before building starts.  probably the first package that is removed is the one.  It's outputted to the screen.


----------



## marino (Jan 28, 2016)

You understand the problem, right?  Synth sees a package that you think is up to date, and it says, "no, it's not up to date", and removes it, and then a bunch of packages that need it are also removed.  One package can remove hundreds or even thousands of ports.


----------



## marino (Jan 28, 2016)

basbebe said:


> I set the env variable manual to `/tmp`
> Querying system about current package installations.
> Stand by, comparing installed packages against the ports tree.
> Stand by, updating external repository catalogs ... pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
> ...



basbebe, this shouldn't happen.  can you run `pkg -vv` and paste everything below "REPOSITORIES {" ?


----------



## talsamon (Jan 28, 2016)

> One package can remove hundreds or even thousands of ports.


A ports-suicide tool!

I don't know how you imagine. Compile three hours a day or a whole day in the week, this will not work. This is not makeable if each update 60 or more package will removed.
What should I do, install the packages one for one. If it is move on in this tempo I am ready, when 11.0 is released.
Then I have a reason to install new. Synth only for someone who installs new (but never switch to it).
Not I have to find which package(s) is/are wrong, the system should tell me.


----------



## marino (Jan 28, 2016)

talsamon: I am having a hard time understanding you, but it seems to me that you don't know how builders work.  The port clusters do the same thing.  Extreme example: Perl 5 is version is changed.  Everything that directly uses lang/perl5 is removed, everything that uses the removed packages are removed, and that's recursive.  Thus one change can removed thousands of files.  Poudriere does the same thing.

As for "telling you", it does tell you.  It tells you which files are removed and the general reason for the removal (e.g. dependency check failure).


----------



## talsamon (Jan 28, 2016)

Now I have compiled two of the packages as test. Answer of synth it removes 10 further packages.

*Edit:* I can set it  back all to standard-option - but this is not the sense of the port-system.

*Serious*: Can you tell me a trick to find circular depencies ?. I think it is something like that.
I think there are two: one in the audio/multimedia/ffmpeg/gstreamer complex and one in docbook/texmf/and_so_on.


----------



## marino (Jan 28, 2016)

No, you don't need to change the options, but we do need to know WHICH PORTS.
Also, don't forget that your machine is not reliable, so we don't know if it's is a hardware problem or not.  It is possible.


----------



## talsamon (Jan 28, 2016)

Here it is:

```
These are the ports that would be built:
  => textproc/docbook-utils
  => multimedia/ffmpeg
  => graphics/graphviz
  => print/ghostscript9-x11
  => graphics/ImageMagick
  => graphics/gegl
  => multimedia/gstreamer1-libav
  => graphics/gimp-app
  => graphics/opencv
  => graphics/zbar
  => audio/alsa-plugins
  => audio/chromaprint
  => graphics/gstreamer1-plugins-opencv
  => graphics/gstreamer1-plugins-zbar
  => audio/gstreamer1-plugins-chromaprint
  => print/cups-pstoraster
  => www/youtube_dl
  => graphics/openimageio
  => print/gimp-gutenprint
  => print/gutenprint-cups
  => multimedia/gstreamer1-plugins-all
  => www/libxul
  => graphics/openshadinglanguage
  => graphics/py-gimp
  => audio/cmus
  => print/gutenprint
  => multimedia/mpv
  => multimedia/phonon-gstreamer
  => multimedia/vlc
  => deskutils/gucharmap
  => net/py-pyzmq
  => www/firefox
  => www/xombrero
  => graphics/blender
  => graphics/cuneiform
  => graphics/gimp
  => graphics/ocre
  => graphics/xournal
  => lang/ghc
  => sysutils/ck4up
  => sysutils/coreutils
  => sysutils/duplicity
  => misc/qt4-qtconfig
  => dns/dnstracer
  => java/icedtea-web
  => mail/thunderbird
  => multimedia/gtk-youtube-viewer
  => multimedia/mencoder
  => ports-mgmt/genplist
  => x11-fm/rodent
  => x11-wm/fvwm-crystal
Total packages that would be built: 51
```


----------



## marino (Jan 28, 2016)

You don't understand what I'm asking.

Theory: There is 1 package that get's rebuilt over and over and over, even though it should be current.  
TASK: Find that one (1) port that creates the package.

Not give a list of 51 ports.  How does that help?


----------



## talsamon (Jan 28, 2016)

I can't tell there are more. Maybe it is multimedia/ffmpeg and maybe it is graphics/ImageMagick, but never sure.
Why is lang/ghc in this list, I have marked them as broken?

Edit: graphics/graphviz is also a candidate.


----------



## marino (Jan 28, 2016)

I assume you are using "`# synth upgrade-system`" (rather than a file list of ports)
So lang/ghc is on the list because you have it installed.  It gets the list of ports from what you've installed on the system, remember?

And yes, you can do more.

Here's how:

`# synth just-build A B C D E` (where A,B,C,D,E are port origins)
then
`# synth just-build A B C D E` again
if 1 or more of them are removed and then build again, it's bad.

You could do that ten times for that list of 51.


----------



## talsamon (Jan 28, 2016)

Synth has overwritten the Makefile in lang/ghc. You can't do this. That means I can never change anything in any Makefile. (can't find the "broken" line).


----------



## marino (Jan 28, 2016)

Nice theory, but synth(1) doesn't do that.


----------



## talsamon (Jan 28, 2016)

But who? vi(1) wasn't => `virecover_enable="NO"`


----------



## marino (Jan 28, 2016)

I don't know.  It's out of scope.  Synth uses read-only null mounts.  It could not overwrite your port Makefile even it wanted to.


----------



## wblock@ (Jan 28, 2016)

marino@ said:


> I disagree it's semantics.  Making a package is a specific thing with specific meaning.  A package is not being made.  The installation occurs directly from stage directory.
> 
> Case in point: if a package is being made, where is it?


Counterpoint: if a package is not being made, how can pkg-list(8) and pkg-delete(8) work on it?  This is why I say it's semantics.

For the purposes of that doc, the files in the stage directory are equivalent to a package, in the same way that a coffee cup is topologically equivalent to a donut.

Here is what it says now:


> When a port is compiled, it actually creates a package. The package is then installed and managed by the package management system.



I can make it more pedantic:


> When a port is compiled, it creates the same files as would be contained in a single binary package file.  Rather than compress those files into a single .txz binary package and then re-extract them, it just installs them directly.  The package is then installed and managed by the package management system.



The second adds implementation detail which is irrelevant to the point and actually obscures it, that being that ports and packages are parts of the same system.

Again, I'm willing to reword this if there is a more technically accurate way to say it that does not muddy the point.


----------



## marino (Jan 28, 2016)

I think the distinction is important because pkgsrc does create an actual package, so anyone familiar with it would have been misled by original wording.

I think the new wording isn't great either.  This would be fine to me:



> The system compiles the port and installs it in a staging area, and pkg(8) then directly installs it on to the system, similar to how it would install a package.


something like that.


----------



## talsamon (Jan 28, 2016)

Ok, in the moment looks good (till the next update? )
Have removed rodent (sniff) - has a bug. Make PR's for editors/abiword and x11-fm/rodent.
With `just-build` no package was removed. I installed all 41 new (10 packages caused by me - changed audio options).
Changed options for graphics/blender to nearly standard (turned off cycles and openimageio - both broken)..
I don't know if graphics/blender caused the chaos. But synth installed before graphics/openimageio which is marked in the graphics/blender port as broken (synth - the tool for hobby-forensics).
After all I know as much before (except `just-build`).



> everything that uses the removed packages are removed


I don't think this is not necessairly. The highest level port should check the next "deeper" level, but not more. Except the next deeper level will be also upgradet. The recursive check is an endless circle. It will be too much to compile, as I mentioned already above.

Postponed lang/ghc till tomorrow and the audio/jack problem to any of the next days. The box needs cool down, and I had after four days for the moment enough from compiling.

But very thanks for your patience.


----------



## fernandel (Jan 28, 2016)

marino@ said:


> Did you update your ports tree?  If so, that would be normal.
> (or does your ports tree get updated with a cron job?)



Yes, I did . Thank you. Now I have a problem just with math/atlas (skipped ports).


----------



## talsamon (Jan 28, 2016)

math/atlas seems broken (there exists a newer version but till now, nobody has update it), if you need for math/py-numppy take the option `netlib` or `openblas`.


----------



## marino (Jan 28, 2016)

fernandel said:


> Yes, I did . Thank you. Now I have a problem just with math/atlas (skipped ports).


Try modifying /usr/ports/math/atlas/Makefile: Comment out the line starting with "MANUAL_PACKAGE_BUILD=" (put "#" in front of it).  I would think it would build then.


----------



## fernandel (Jan 28, 2016)

marino@ said:


> Try modifying /usr/ports/math/atlas/Makefile: Comment out the line starting with "MANUAL_PACKAGE_BUILD=" (put "#" in front of it).  I would think it would build then.


Yes, it works. It's building 'skipped" ports. Thanks for the help.
It is done.


----------



## PacketMan (Jan 29, 2016)

So far so good, its going through the list one by one, but I do have one issue:  if I roll my mouse wheel it causes a graceful shutdown.  Arrgggg did it a couple times already. Other than that so far so good.

For my own education would be nice on your webpage to have a thorough explanation of what each build status means, maybe you can put that in the man page.  Also would be nice if the Built counter was split into two: Built-C (for ports/packages compiled locally on the machine), and Built-F (for packages fetched and extracted). Something for you to consider if you find yourself bored.  

Its too early to tell but overall progress seems to be considerably faster than pure postmaster compile for all. I'm estimating 10 hours at this point in time.

Thanks again.


----------



## Chris_H (Jan 29, 2016)

PacketMan
Regarding the moune-wheel; is your box strained? Meaning, are the resources pretty maxed out, while SYNTH is building?
Also; are you running SYNTH from your Desktop? I'm not marino@ , and I didn't develop SYNTH, but I knowing what I do, I can't imagine how SYNTH could be at all responsable, aside from further taxing an already fairly taxed system.

Just my thoughts. Hoping they might help.

--Chris

EDIT:
I just realized I mis-read your statement regarding the mouse-wheel; I somehow saw the shutdown as _system_ shutdown -- my bad. Sorry.


----------



## talsamon (Jan 29, 2016)

lang/ghc: failed depency check

```
========================< phase : package  >========================
===>  Building package for ghc-7.10.2
actual-package-depends: dependency on /lib/libutil.so.9 not registered (normal if it belongs to base)
```
.

reminds me on:
x11-fm/rodent

```
========================< phase : package  >========================
===>  Building package for ghc-7.10.2
actual-package-depends: dependency on /lib/libmagic.so not registered (normal if it belongs to base)
```

but the rodent error also appears in the port, I don't if it is also with lang/ghc.
(By the way it compiles with poudriere - with the error message).

Both files exists.


----------



## Chris_H (Jan 29, 2016)

marino@ ,
My standard procedure for updating my production servers, has been to use traditional jails on the development box I just updated with ports-mgmt/synth. Given my experience, aside from a couple of (not unexpected) huckups with ports-mgmt/synth was so good. I got to thinking how I might use synth as an update strategy for updating my servers. A couple of thoughts come to mind. On one hand, I could simply point synth to a different set of Ports, PortsConf, and Packages folders, using copies of those I already use for them within the jails I currently use to update them. But given the servers are on different different versions, I'm thinking -- especially where Linux related ports are concerned, that I might run into issues. So I was thinking it might be a better idea, if I ran synth within a jail itself. Seems fiesable, but thought I'd like to get your thoughts on the matter. Are either of my ideas better than the other, or niether, or...?

Thanks!

--Chris


----------



## Chris_H (Jan 29, 2016)

talsamon


> (normal if it belongs to base)


As I read this; it is my understanding that if _either_ of those libraries (libutil.so.9, or libmagic.so) come from the "base install" of FreeBSD, than you should _expect_ to see those warnings. _Meaning_, that since synth builds the ports in a chroot(8) file system -- a jail(8) so to speak. Than the port builders do not have access to those libraries. TL;DR -- in short; it's OK. You do not have to be concerned_. 

All the best.

--Chris_


----------



## talsamon (Jan 29, 2016)

```
Than the port builders do not have access to those libraries
```

.. and it fails, and tries to compile again  and again.

I edit above: it compiles/installs with poudriere and in the port - with the error message (rodent).


----------



## Chris_H (Jan 29, 2016)

talsamon said:


> it compiles/installs with poudriere and in the port - with the error message (rodent).


If I'm not mistaken; ports-mgmt/poudriere _does_ have access to those libraries. It _may_ be that ports-mgmt/synth might need to better accomodate that situation. I'll defer to marino@ for all the technicalities, regarding that. 

--Chris


----------



## talsamon (Jan 29, 2016)

Yes, I understood. Maybe synth better detect the errors. My critc is that I could not exclude the failures, to prevent the tries to compile again and again. Mark broken is an ugly solution.


----------



## marino (Jan 29, 2016)

talsamon Chris_H: Those aren't even errors.  It's just information.  All it is saying is that those libraries aren't registered in pkg(8) database and the most likely reason is that they are not PORTS libraries but BASE libraries.  And actually that the case here.

To repeat : THOSE ARE NOT ERRORS!  and they come from pkg(8) , not synth.


----------



## marino (Jan 29, 2016)

PacketMan said:


> So far so good, its going through the list one by one, but I do have one issue:  if I roll my mouse wheel it causes a graceful shutdown.  Arrgggg did it a couple times already. Other than that so far so good.



Only the escape key causes a shutdown, so unless your mouse wheel is causes escape sequences to go to the screen, I doubt that's it.  Another way to think about it: Surely you aren't the only person moving a mouse?  Nobody else is reporting this.



> For my own education would be nice on your webpage to have a thorough explanation of what each build status means, maybe you can put that in the man page.


  Those are ports build targets.  There's pretty self explanatory IMO, unless you are talking about "idle" and "shutdown", which I think is also fairly obvious.  In any case, it's just extra information to give insight into the build.  Full understanding of exactly target the port is using isn't required.  I think explanating port targets is out of scope for what I wanted to achieve with either the tutorial or the man page.



> Also would be nice if the Built counter was split into two: Built-C (for ports/packages compiled locally on the machine), and Built-F (for packages fetched and extracted). Something for you to consider if you find yourself bored.



Hmm?  Example: 10 packages should be built, and 6 are available. remotely.  It should pull in those 6 and show "4" as the number to build.  Is that not what is happening?  For example, it's showing 10 left to build and completing after 4?  If so, that's not intended.  The "total" number is intended to be build-only, so that split you suggest doesn't make sense in that case.



> Its too early to tell but overall progress seems to be considerably faster than pure postmaster compile for all. I'm estimating 10 hours at this point in time.



No, it's not too early.  portmaster builds serially, one at a time.  Synth can build 4, 6, even 10 packages simulateously (we've got it set 10 builders x 5 jobs each).  There's no comparison here.  Even using one or two builders for fetching while another 2 builders builder is a huge win over portmaster.


----------



## marino (Jan 29, 2016)

Chris_H said:


> marino@ ,
> My standard procedure for updating my production servers, has been to use traditional jails on the development box I just updated with ports-mgmt/synth. Given my experience, aside from a couple of (not unexpected) huckups with ports-mgmt/synth was so good. I got to thinking how I might use synth as an update strategy for updating my servers. A couple of thoughts come to mind. On one hand, I could simply point synth to a different set of Ports, PortsConf, and Packages folders, using copies of those I already use for them within the jails I currently use to update them. But given the servers are on different different versions, I'm thinking -- especially where Linux related ports are concerned, that I might run into issues. So I was thinking it might be a better idea, if I ran synth within a jail itself. Seems fiesable, but thought I'd like to get your thoughts on the matter. Are either of my ideas better than the other, or niether, or...?



It's technically impossible to run synth in a jail because jails won't allow tmpfs mounts, so it's not an option.

Even if it was, I wouldn't recommend it.   All your servers that share a common ABI (e.g. FreeBSD 10 amd64) can use the same repository.  You only need to build it once and you can share it among many.  If the release number or architecture is different, you'll need a unique repository for those. 

A repository is just a menu.  You can have a 5000-package local repository and only have 20 packages installed on the system that uses it.  So having linux packages in a repository where you don't need them doesn't hurt anything.


----------



## marino (Jan 29, 2016)

PacketMan


> Only the escape key causes a shutdown, so unless your mouse wheel is causes escape sequences to go to the screen, I doubt that's it. Another way to think about it: Surely you aren't the only person moving a mouse? Nobody else is reporting this.



That being said, the choice of using ESCAPE key has already bitten, maybe something to do with ANSI codes for terminals.  That issue was bandaged but maybe we should change it to capital "Q" because this isn't the first report of inadvertent shutdown.  I chose it because it's an intuitive thing but it seems to have some undesired consequences.


----------



## Chris_H (Jan 29, 2016)

marino@ said:


> to repeat : THOSE ARE NOT ERRORS! and they come from pkg(8), not synth.


Indeed. That was my point exactly. I apparently didn't articulate that very well.  But I _did_ also believe that pkg/synth felt they needed access to them. But my point was that the so-called "errors" were _informative_ only. Oh well.


----------



## Chris_H (Jan 29, 2016)

marino@ said:


> It's technically impossible to run synth in a jail because jails won't allow tmpfs mounts, so it's not an option.
> 
> Even if it was, I wouldn't recommend it.   All your servers that share a common ABI (e.g. FreeBSD 10 amd64) can use the same repository.  You only need to build it once and you can share it among many.  If the release number or architecture is different, you'll need a unique repository for those.
> 
> A repository is just a menu.  You can have a 5000-package local repository and only have 20 packages installed on the system that uses it.  So having linux packages in a repository where you don't need them doesn't hurt anything.


Well, that was pretty much my assessment; save the _in_ability to run within a jail.
As to the ABI differential; My development box is tracking -CURRENT, while the servers are _currently_ on 9.3-STABLE. I'm aware of the repo hierarchy, and it's ability to harbor i386/amd64 branches, to an extent. But in the end I guess it's just not feasable to use synth for this. As to linux; My only concern in that regard was versioning (ABI) from say f10 9(.3) v. c6 (11) being an issue.

Thanks for the helpful reply, John.

--Chris


----------



## marino (Jan 29, 2016)

Chris_H said:


> But in the end I guess it's just not feasable to use synth for this. As to linux; My only concern in that regard was versioning (ABI) from say f10 9(.3) v. c6 (11) being an issue.



Why not?
Build a 9.3 world on current and install it with DESTDIR.  Create a new profile that defines the system root at that destdir.  voila, you have 11 building packages for earlier systems.  The only gotcha are ports that run conftests that pass with 11 kernel but fail with 9.3 kernel.  Then the packages produced would be nonfunctional on 9.3


----------



## PacketMan (Jan 29, 2016)

marino@ said:


> Only the escape key causes a shutdown, so unless your mouse wheel is causes escape sequences to go to the screen, I doubt that's it.  Another way to think about it: Surely you aren't the only person moving a mouse?  Nobody else is reporting this.





marino@ said:


> Packetman@
> That being said, the choice of using ESCAPE key has already bitten, maybe something to do with ANSI codes for terminals.  That issue was bandaged but maybe we should change it to capital "Q" because this isn't the first report of inadvertent shutdown.  I chose it because it's an intuitive thing but it seems to have some undesired consequences.



I'll do some testing but I'm pretty sure its happened twice: if I put my mouse arrow over the xterm session running synth and roll the wheel, synth pretty much instantly sticks that yellowish line in there "graceful shutdown in progress....".


----------



## marino (Jan 29, 2016)

somehow you mouse is emitting escape key code.  I think there's enough weird stuff happening to switch this to a capital "Q" which requires a two-key combination and will be hard to do accidently (unless some ansi code happens to trigger it)


----------



## PacketMan (Jan 29, 2016)

marino@ said:


> Hmm?  Example: 10 packages should be built, and 6 are available. remotely.  It should pull in those 6 and show "4" as the number to build.  Is that not what is happening?  For example, it's showing 10 left to build and completing after 4?  If so, that's not intended.  The "total" number is intended to be build-only, so that split you suggest doesn't make sense in that case.



When I started I had a number like 656 in the Total field. As each one is 'built', whether fetched prebuilt packages, or source code is downloaded an compiled on machine into a package, the Left counter decrements by 1, the Built counter increments by 1.  Coming back to my screen later, say after a nights sleep, I have no way of telling how many actually prebuilt packages were downloaded and used versus how many were source code compile on my machine.  See what I mean?  I'm just trying to get a sense of what percentage of ports used prebuilt packages instead of local compile.


----------



## marino (Jan 29, 2016)

No, I don't see what you mean because fetching happens before the ncurses screen comes up.
Every increment is built; none are fetched.


----------



## PacketMan (Jan 29, 2016)

marino@ said:


> somehow you mouse is emitting escape key code.  I think there's enough weird stuff happening to switch this to a capital "Q" which requires a two-key combination and will be hard to do accidently (unless some ansi code happens to trigger it)



Dunno,.....my mouse wheel works perfectly fine in all other apps and XTerm sessions.  How about CTRL-Q since its a control signal we are sending after all.

But at least you got the main guts of the synth program working really well it seems.  You have a beautiful mind.


----------



## marino (Jan 29, 2016)

Thanks!
Maybe control-q would work (or control-shift-q) if it's a single character.  I don't want to look for multiple consecutive keypresses; too much trouble.


----------



## PacketMan (Jan 29, 2016)

Sorry I am confused too I guess.  I am using the [N] option. I thought some sort of counter that increments for every prebuilt package downloaded and used (instead of source code being downloaded and compiled) could be made.

For example.
Built 500
Compile 400
PreBuilt Fetched 100

Anyway I didn't intend to muddy up a good thread so I will stop this topic... bigger problems in life to fix.


----------



## marino (Jan 29, 2016)

If it's working correctly:


starts with 875 (for example)

scans existing packages
finds 375 suitable packages, total down to 500
scans remote packages
finds 100 suitable remote packages
downloads the 100, total down to 400
starts building 400 ports from source.
If the totals reflect something else, it's not working as intended.


----------



## PacketMan (Jan 29, 2016)

Hmmm, I don't think I am seeing that, my Total counter does not decrement. My Left counter decrements, my Built counter increments, serially one port at a time. (Right now its an average of 18 per hour). In other words, in my opinion, the Built counter is an accumulation of ports compiled from source PLUS ports that used/fetched prebuilt packages.

But remember I am on my first run if that matters.


----------



## marino (Jan 29, 2016)

Total does not decrement; it's constant.
Remaining column starts as same as total, and decrements.
Same thing.


----------



## marino (Jan 29, 2016)

The build counter is 4 counters. All reflect the outcome of the build.
There is no package fetching during this phase at all. Only building.


----------



## PacketMan (Jan 29, 2016)

Let me ask it this way. These are my numbers currently:

Total 568
Left 385
Built 183

How many so far have used prebuilt packages?


----------



## marino (Jan 29, 2016)

Zero.
Any fetching would have been before that screen appears, and "TOTAL" reflects what is left to build (as demonstrated in my previous example).


----------



## marino (Jan 29, 2016)

You should have seen the fetching, it takes some time, it should have been obvious.


----------



## talsamon (Jan 29, 2016)

Rodent/GHC:
Ok, it is no error - but synth(1) states failure and delete the package.


----------



## PacketMan (Jan 29, 2016)

marino@ said:


> zero.
> any fetching would have been before that screen appears, and "TOTAL" reflects what is left to build (as demonstrated in my previous example)





marino@ said:


> you should have seen the fetching.  it takes some time.  it should have been obvious.



Okay I will look into this, but I'm gonna say no prefetching is happening. I'll restart using my mouse roll wheel, errr ahhh I mean ESC key.


----------



## PacketMan (Jan 29, 2016)

Clue?


```
# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Failed!
The external repository could not be updated.
No prebuilt packages will be used as a result.
Scanning existing packages.
```


----------



## marino (Jan 29, 2016)

PacketMan: it's the second time it's been reported (and I asked for the output of `pkg -vv` after "REPOSITORIES {" but the person didn't comply)

My guess is that it's finding it's own repository and trying to update itself and that's not supposed to happen.  If it finds the Synth repository, it's supposed to skip it.  That's why I need to see what pkg(8) thinks the repositories are.


----------



## PacketMan (Jan 29, 2016)

marino@ said:


> PacketMan: it's the second time it's been reported (and I asked for the output of `pkg -vv` after "REPOSITORIES {" but the person didn't comply)



This what you looking for?


```
REPOSITORIES {
}
VALID_URL_SCHEME [
  "pkg+http",
  "pkg+https",
  "https",
  "http",
  "ftp",
  "file",
  "ssh",
]


Repositories:
  FreeBSD: {
  url  : "pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/quarterly",
  enabled  : yes,
  priority  : 0,
  mirror_type  : "SRV",
  signature_type  : "FINGERPRINTS",
  fingerprints  : "/usr/share/keys/pkg"
  }
  Synth: {
  url  : "file:///var/synth/live_packages",
  enabled  : yes,
  priority  : 0
  }
$
```


----------



## marino (Jan 29, 2016)

Yes, it is.  The code is designed to pick "FreeBSD" repo (and then eject).  If "Synth" was encountered, it would just keep going.  However, it seems like the external repository check is happening on Synth, not FreeBSD.  Strange.  Seems that it should be working.  (It works for me, but I'd like to see it work on FreeBSD)


----------



## fernandel (Jan 29, 2016)

Hi!

I have a problem to build graphics/enblend (update). I sent email to maintainer with log file. It is problem just on my system or also on the others, please?
Thank you.


----------



## garry (Jan 29, 2016)

marino@ said:


> I don't think anything is wrong with lang/ghc as a port, because it builds fine for me in about 55 minutes.



I hit a small problem with lang/ghc which mystified me at first.  Synth reported that the options for that port were out of date and that I needed to `make -C /usr/ports/lang/ghc config`.  After running the `make config` I got the same error and so ran `make rmconfig && make config`.  Still the options seemed to be out of date according to Synth.  Looking at the Makefile for lang/ghc I found that it adds "hidden" options if /usr/local/bin/ghc or /usr/local/bin/HsColour exist:

```
.if exists(${LOCALBASE}/bin/ghc)
OPTIONS_DEFINE+=  BOOT
.endif

.if exists(${LOCALBASE}/bin/HsColour)
OPTIONS_DEFINE+=  BOOTH
.endif
```
allowing GHC to bootstrap using the installed ghc and hscolour.  I got around the impasse by removing lang/ghc, running `make rmconfig && make config` one more time, and then building my packages.   Maybe you want to consider this to be a (somewhat general) problem in how Synth handles the environment when analyzing options.  Or you've already fixed it?? [Personally I think the lang/ghc port does the wrong thing in checking the live environment that way.  I would prefer that it simply check for the availability of a local GHC package, but I may be wrong.  It's a bootstrap problem.  Always ugly.]

I used synth-0.99_2 to build a new repository, reproducing what I had previously built with ports-mgmt/poudriere using the same general setup (3 jails, 3 jobs, tmpfs for the workdir and data).  Synth built my 1800 packages in about 18 hours of wall time with no problems.  It had taken more than 24 hours of wall time to build that repository with poudriere(8).  It seems that Synth is very efficient in setting up and tearing down builders (chroot(8)s / jail(8)s). I noticed that small packages can be done in 20 seconds by Synth but take a minimum of two minutes with poudriere(8).

Synth is very pleasant to use.  Thanks again.


----------



## kpa (Jan 29, 2016)

garry said:


> Looking at the Makefile for ghc I found that it adds "hidden" options if /usr/local/bin/ghc or /usr/local/bin/HsColour exist:
> 
> ```
> .if exists(${LOCALBASE}/bin/ghc)
> ...



That is very hackish and is never going to work under a package builder that starts from a clean state for each port. The binaries mentioned are never going to exist in the jail/chroot unless they come from an explicitly declared build dependency that gets installed before building of the port starts.


----------



## marino (Jan 29, 2016)

kpa is correct.  Unfortunately it's not as uncommon as I would like.  When Synth highlights issue like that, I usually see if we can get the port fixed.
For something like this, we could probably wrap the conditional statements with a PACKAGE_BUILDING check. It's fixable but we need the maintainer to buy in.


----------



## talsamon (Jan 30, 2016)

Sorry, I don't understand the sense of this tool. Today one update (audio/lame). It compiles 28 packages (2 and a half hours), some of them indirect dependent,  and installs only the lame package. For what it compiles the other packages if it does not install it.
If I had 20 packages, I guess there were 120 packages or more to compile, or for once in a week it could be 400 or more.  Who had the time to compile this?


----------



## marino (Jan 30, 2016)

1) it compiles what is necessary.  If it says audio/lame is necessary for any reason, it will build it if has to.  It could be a build dependency.
2) It sounds like you have a very weak machine (on top of already being unreliable).  In that case you probably should only build what you customized and let the rest fetch from official binaries.  If it's too painful to build, then don't build.  Nobody is forcing you.


----------



## chrbr (Jan 30, 2016)

Dear talsamon,
you might want to try devel/ccache to reduce the build time.


----------



## talsamon (Jan 30, 2016)

No, if you say above you need 55 minutes for lang/ghc my machine need 68 minutes, not much slower.
And the system is running good since last February without bigger problems (only not properly plugged cable, in July destroyed the system).

Maybe ports-mgmt/portmaster don't recompile all dependencies that needed and don't check the portstree right, but this is too much.
Why must www/firefox recompiled? audio/alsa-lib sure, but Firefox? Why must www/libxul be recompiled, multimedia/gstreamer yes, but www/libxul? And why emulators/virtualbox-ose has only one audio option (PULSEAUDIO) it is off. Which option has any dependency to audio or multimedia=

During the compile you can't use the machine (compiler take all resources it could get). And if I limit it, it needs an endless time. If it's one or one and a half hour (or sometimes longer) it is ok, in summary, but nor for only one small package like audio/lame. And why it compiles all the other packages store it in the repo - but don't use (install) it?? But you can't wait e.g. three hours for them, and maybe each day (or as I mentioned above, for a day, if you update once in the week.
Don't misunderstand me, I honor your work - it seems a good tool for testing, but it seems not for updating And I wanted to have a better tool as ports-mgmt/portmaster for long time - but this is nearly "overkill".

kpa I see it with the options in lang/gcc (this was not synth, ok). I did not try to remove ghc. (and I see misc/compat9x has no libutil file, so I think we can remove this LIB_DEPENDS line in Makefile, I see this after the last try). Maybe that is the reason for synth rejected `config`, if I try to set the option boot to on. Something is wrong with the port. Next try I will remove ghc before I try it with synth.


----------



## marino (Jan 30, 2016)

talsamon, you've been making a lot of technical comments like "this is not necessary to do".  I've not been responding to them because they are not correct, but I don't want to get into the hows and why they aren't correct.    The failure cascades are absolutely required, not doing them would result in an unusable repository.  Your "one level deep" theory will fail.  If you give it a lot of thought you can probably deduce why.

I'm working on some polish problems (synth hasn't reached release yet) but one area that has been bulletproof is the build logic.  It's correct.

And yes, if you aren't using ccache, you absolutely should.  (And the reason I don't trust your machine is because you have several issues that cannot be reproduced on multiple machines and multiple OS, so I'm really leary of your machine)


----------



## fernandel (Jan 30, 2016)

talsamon said:


> No, if you say above you need 55 minutes for lang/ghc my machine need 68 minutes, not much slower.
> And the system is running good since last february without bigger problems (only not ptoprtrly plugged cable, in july
> destroyed the system).
> 
> ...


For example:

```
pkg info -r lame
lame-3.99.5_3:
   gstreamer-plugins-lame-0.10.19_1,3
   ffmpeg-2.8.5,1
   transcode-1.1.7_25
   gstreamer1-plugins-lame-1.6.3
   sox-14.4.2_1
   normalize-0.7.7_10
   aqualung-1.0
   mencoder-1.2.r20151219_2
```
and than:

```
pkg info -r ffmpeg
ffmpeg-2.8.5,1:
   youtube_dl-2016.01.15
   libxul-38.5.2
   gstreamer1-libav-1.6.3
   alsa-plugins-1.1.0
   vlc-2.2.1_6,4
   opal-3.10.10_9
   mediastreamer-2.12.1
   transcode-1.1.7_25
   mlt-0.9.6
   chromaprint-1.1
   libquicktime-1.2.4_11
   firefox-43.0.4_1,1
   gimp-gmic-plugin-1.6.0.0_4
   opencv-2.4.9_7
   aqualung-1.0
   blender-2.76b
```

It is what I think why synth rebuild more ports not just "lame" in this case.
Maybe I am wrong.


----------



## marino (Jan 30, 2016)

fernandel you are not wrong


----------



## PacketMan (Jan 30, 2016)

marino@ said:


> Yes, it is.  The code is designed to pick "FreeBSD" repo (and then eject).  If "Synth" was encountered, it would just keep going.  However, it seems like the external repository check is happening on Synth, not FreeBSD.  Strange.  Seems that it should be working.  (It works for me, but I'd like to see it work on FreeBSD)



Well I really hope you get this working, and I will be willing to work with you to provide any help I can.  I intend on using the [N] fetch prebuilt option heavily on my XFCE desktop machine, and maybe other desktop machines someday. I think for my NAS machines I will not use [N] option since number of ports installed is small.  That just said I will be at the mercy of waiting for ports to get updated before we can 'try again'. (All we need is to wait for a Firefox or Chromium update. )

Lastly, (maybe for a version 2.0 of synth) I would really appreciate seeing the fetching of pre-built packages, and associated counter, after the ncurses display, not before.  I know you got the smarts; you can do it! ......but I realize beggars can't be choosers.


----------



## PacketMan (Jan 30, 2016)

Hmmm, okay so my initial run is done.  Rebooted, updated the ports tree again, and ran synth again.  Does this mean the fetch packages option is working?


```
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
These are the ports that would be built:
  => audio/jack
  => audio/fluidsynth
  => editors/openoffice-4
  => multimedia/vlc
Total packages that would be built: 4
The complete build list can also be found at:
/tmp/synth_status_results.txt

# synth upgrade-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
```

That just said, for these 4 ports, manual compiling from source is being done, fetching a prebuilt package was not done.

I'll keep watching for it, but I did not see these lines this 2nd time around:

```
Stand by, updating external repository catalogs ... pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Failed!
The external repository could not be updated.
No prebuilt packages will be used as a result.
```


----------



## talsamon (Jan 30, 2016)

And no end:
I tested ccache with firefox.
First time it compiles firefox.
Second time it compiles firefox and 3 other packages (why the 3 others).
After this:

```
subversion-1.9.3_1.txz failed option check.
openjdk8-8.66.17_3.txz failed option check.
p5-subversion-1.9.3.txz failed dependency check.
py27-mako-1.0.3.txz failed dependency check.
portdowngrade-1.5.txz failed dependency check.
porttools-1.06.txz failed dependency check.
truthtable-1.2.2_2.txz failed dependency check.
freecol-0.11.6.txz failed dependency check.
tvbrowser-3.4.3.txz failed dependency check.
icedtea-web-1.5.2.txz failed dependency check.
jedit-5.3.0,1.txz failed dependency check.
deluge-1.3.12,1.txz failed dependency check.
```

what the hell, I have nothing changed in the ports.


----------



## talsamon (Jan 30, 2016)

P.S: I have removed the firefox-package. Synth does not want to compiles, as it was there. Causes the removing of the packages such things. The same package the same version, and what has firefox to do with portdowngrade  and so on.....?? That is irrational.


----------



## talsamon (Jan 30, 2016)

There is no way out. If I start with ccache, without swapfile, it runs out of space. If   I start with both, ccache does not work.

Edit: Is not true, both work in thier ports. synth only works one time with ccache. After reboot it works no more. I have changed nothing in the configuration.


----------



## fernandel (Jan 30, 2016)

talsamon said:


> There is no way out. If I start with ccache, without swapfile, it runs out of space. If   I start with both, ccache does not work.


What is size of your swapfile? What do you have in `Num, concurrent builders` and i `Max. jobs per builder`?


----------



## chrbr (Jan 30, 2016)

Dear All,
please start with a minimalistic collection of ports first. I have run my very first tests starting with ports-mgmt/poudriere and also with devel/ccache just with mail/abook and www/lynx with almost no dependencies. I am pretty sure that for ports-mgmt/synth the procedure makes sense, too.

If a tool is new to me it is very likely that the mistakes are on my side. Sorting that out is by far less time consuming using small ports. In 99.99% it is not sorting out but finding out that the issue is https://en.wikipedia.org/wiki/User_error#PEBKAC.


----------



## talsamon (Jan 30, 2016)

fernandel Num, concurrent builders: 2. Max. jobs per builder 3. Swapfile 12 GB. (8 GB RAM).


----------



## fernandel (Jan 30, 2016)

talsamon said:


> fernandel  Num, concurrent builders: 2. Max. jobs per builder 3. Swapfile 12 GB. (8 GB RAM).


Do you have swap partition or swap file?
I have 2 as you and 4 per jobs, swap partition is 3 GB and I never had a problem...usually is used about 1% to 4% and I have 8 GB of RAM too.


----------



## talsamon (Jan 30, 2016)

A swapfile. I know it is too big, the system only use 1200 MB. But experimented, cause I had a kernel-panic warning at shutdown  (But till now, it causes nothing - Last reboot it was no warning).


----------



## marino (Jan 30, 2016)

PacketMan said:


> Hmmm, okay so my initial run is done.  Rebooted, updated the ports tree again, and ran synth again.  Does this mean the fetch packages option is working?
> 
> I'll keep watching for it, but I did not see these lines this 2nd time around:



Maybe but not necessarily.  Your synth repo exists now, so it could be reading it (and thus never downloading anything).  You got the lines the first time because there was no repository generated yet, but now it has been.


----------



## marino (Jan 30, 2016)

fernandel said:


> Do you have swap partition or swap file?
> I have 2 as you and 4 per jobs, swap partition is 3 GB and I never had a problem...usually is used about 1% to 4% and I have 8 GB of RAM too.



I think you are on the right track.  Something is seriously suspicious about his machine.  He keeps talking about how "irrational" everything is, and how nothing is reproducible, but he's the only one this these issues.  Chances are there is a serious misconfiguration somewhere.  Maybe it's this swapfile thing.  If not, it's got to be something unorthordox.


----------



## PacketMan (Jan 31, 2016)

marino@ said:


> Maybe but not necessarily.



Just did another ports tree update. 26 updates available, and all are being built, meaning zero pre-built packages were fetched.  If you need me to post anything let me know, otherwise I'll just carrying on using synth and waiting for an update that says fetching of pre-built packages is fixed!


----------



## PacketMan (Jan 31, 2016)

marino@ said:


> Maybe but not necessarily.



This sure looks like it fetched (and then installed) some pre-built packages. But note this came after the ncurses screen, not before. NetSpades and libslang2 were built in the ncurses screen first.


```
# synth install games/netspades
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
The following packages will be fetched:

New packages to be FETCHED:
   glib12-1.2.10_15 (11.04% of 1 MiB: 145 KiB)
   gtk12-1.2.10_24 (88.96% of 1 MiB: 1 MiB)

The process will require 1 MiB more space.
1 MiB to be downloaded.
Fetching glib12-1.2.10_15.txz: 100%  145 KiB 148.4kB/s  00:01  
Fetching gtk12-1.2.10_24.txz: 100%  1 MiB  1.2MB/s  00:01  
Scanning entire ports tree.
Scanning existing packages.  
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  260 B  0.3kB/s  00:01  
Fetching packagesite.txz: 100%  150 KiB 153.4kB/s  00:01  
Processing entries: 100%
Synth repository update completed. 658 packages processed.
Checking integrity... done (0 conflicting)
The following 4 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   NetSpades: 4.2.0_9 [Synth]
   glib12: 1.2.10_15 [Synth]
   libslang2: 2.3.0 [Synth]
   gtk12: 1.2.10_24 [Synth]

The process will require 13 MiB more space.
[1/4] Installing glib12-1.2.10_15...
[1/4] Extracting glib12-1.2.10_15: 100%
[2/4] Installing libslang2-2.3.0...
[2/4] Extracting libslang2-2.3.0: 100%
[3/4] Installing gtk12-1.2.10_24...
[3/4] Extracting gtk12-1.2.10_24: 100%
[4/4] Installing NetSpades-4.2.0_9...
[4/4] Extracting NetSpades-4.2.0_9: 100%
```

But last night I did another ports tree update and Open Office was updated. But regretfully the pre-built package was not fetch, instead it downloaded the source, and took about 6 hours to compile. 

Thoughts? Comments?  And why does it scan existing packages twice?


----------



## marino (Jan 31, 2016)

I don't think it's technically possible that fetching came come after ncurses screen.
Those are difference package scans.  The first is limited to the build (very small number) and the second is every package that has been built (potentially a very big number).   That last scan only happens on installs.
As for skipping OpenOffice, it apparently decided it was unsuitable (did not meet requirements).


----------



## kpa (Jan 31, 2016)

Is there actually a problem with the pre-built packages? I think you are all talking past each other unless I have missed something.

Now, for build time dependencies it is of course beneficial to use existing packages as much as possible and as far as I have gathered it is working already? This is situation where a port has an update but the update will not affect other installed ports because none of them is depending on the updated port. The build time dependencies can be installed from existing packages before launching the port build.

The other thing is then what happens when there are dependent ports on the updated ports. I don't see how you could use pre-built packages in this case. The knowledge that a dependent port did not actually require a rebuild is not available until you actually have finished rebuilding of the dependent port. There is nothing you can do before the build to do any clever guessing if the rebuild will result in a changed package or not, that would require some very advanced code analysis tools that just not available for FreeBSD ports.


----------



## marino (Jan 31, 2016)

> as far as I have gathered it is working already?


It might be buggy.  There's at least one suspicious thing that needs investigation.  It is "experimental" given how new the feature is and how little testing it's had.

For the rest, synth queries several factors of the remote packages in order to determine if it can be used or not.  If it can't be used it just ignores it and rebuilds it.


----------



## PacketMan (Jan 31, 2016)

kpa said:


> Is there actually a problem with the pre-built packages? I think you are all talking past each other unless I have missed something.



Yeah I think you missed something.  Based on earlier log files we thought there was a problem, and a 2nd person had informed of similar complaint if I understood Marino right. I was posting in response to him.  Now this morning I am seeing behaviour that suggest to me that fetching of pre-built packages is working. Would you rather I not update him and leave him wasting his time?  I'd rather be courteous and let him know what I have found.  And since this is brand new software and this feature even newer with very little testing, I thought I would contribute. As simple as that. I would say unless Marino ask me of something, or I see something else, or whatever, my commentary in this thread will come to a stop very soon.


----------



## PacketMan (Jan 31, 2016)

marino@ said:


> i don't think it's techincally possible that fetching came come after ncurses screen.



I am sure it did. NetSpades and libslang2 were built in the ncurses screen first, and then I got the output shown above where the other pre-built packages were downloaded and installed.  I'll keep watching for that behaviour if you want me to.

Really liking this synth. Thank you so much Marino.


----------



## PacketMan (Jan 31, 2016)

marino@ said:


> It might be buggy.  There's at least one suspicious thing that needs investigation.  It is "experimental" given how new the feature is and how little testing it's had.



It might be buggy but it sure does seem to be working Marino. I did this just for fun.  Synth didn't even download the package. It just reused what was already in 'store'.  Sweet.


```
/usr/ports/x11/xeyes # make deinstall clean
/usr/ports/x11/xeyes # make rmconfig

# synth install x11/xeyes
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Scanning entire ports tree.
Scanning existing packages.   
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  260 B  0.3kB/s  00:01   
Fetching packagesite.txz: 100%  150 KiB 153.4kB/s  00:01   
Processing entries: 100%
Synth repository update completed. 658 packages processed.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
   xeyes: 1.1.1 [Synth]

The process will require 22 KiB more space.
[1/1] Installing xeyes-1.1.1...
[1/1] Extracting xeyes-1.1.1: 100%
```


----------



## marino (Jan 31, 2016)

PacketMan said:


> I am sure it did. NetSpades and libslang2 were built in the ncurses screen first, and then I got the output shown above where the other pre-built packages were downloaded and installed.  I'll keep watching for that behaviour if you want me to.



Here's what happened:
1) it fetched it before the screen
2) you missed it, it might have been fast
3) the ncurses screen came up
4) when it was done, it restored the terminal to what it was before it came up
5) you saw the old text (pre-build) and though it was new because it was the first itme you saw it.


----------



## Crivens (Feb 1, 2016)

I for one, would take the moment to thank you for some great software!
This thread is turning into rope, sort of thing. Hopefully it will turn into some chapter of the handbook.

Good work!


----------



## talsamon (Feb 1, 2016)

Once more, and if you hate me:

Update for ftp/curl and www/youtube_dl.

```
cmake-3.4.2.txz failed dependency check.
libquvi-0.4.1_4.txz failed dependency check.
ffmpeg-2.8.5,1.txz failed dependency check.
webkit-gtk2-2.4.9.txz failed dependency check.
raptor2-2.0.15_1.txz failed dependency check.
gstreamer1-libav-1.6.3.txz failed dependency check.
gimp-app-2.8.16_1,1.txz failed dependency check.
alsa-plugins-1.1.0.txz failed dependency check.
chromaprint-1.1.txz failed dependency check.
gstreamer1-plugins-chromaprint-1.6.3.txz failed dependency check.
gstreamer1-plugins-curl-1.6.3.txz failed dependency check.
foomatic-db-20150819.txz failed dependency check.
foomatic-db-engine-4.0.12,2.txz failed dependency check.
wx30-gtk2-3.0.2_4.txz failed dependency check.
openimageio-1.5.20_3.txz failed dependency check.
vorbis-tools-1.4.0_10,3.txz failed dependency check.
rasqal-0.9.33.txz failed dependency check.
gimp-gutenprint-5.2.10_3.txz failed dependency check.
gutenprint-foomatic-5.2.10_1.txz failed dependency check.
gstreamer1-plugins-all-1.4_3.txz failed dependency check.
freedink-dfarc-3.12.txz failed dependency check.
git-2.7.0.txz failed dependency check.
libcmis-0.5.0.txz failed dependency check.
libxul-38.6.0.txz failed dependency check.
opencv-2.4.9_7.txz failed dependency check.
py27-gimp-2.8.16.txz failed dependency check.
cmus-2.6.0_3.txz failed dependency check.
cdrtools-3.01.txz failed dependency check.
synergy-1.7.5.txz failed dependency check.
redland-1.0.17_4.txz failed dependency check.
php56-curl-5.6.17.txz failed dependency check.
gutenprint-5.2.10.txz failed dependency check.
mpv-0.15.0,1.txz failed dependency check.
phonon-gstreamer-4.8.2_1.txz failed dependency check.
vlc-2.2.1_6,4.txz failed dependency check.
transmission-cli-2.84_4.txz failed dependency check.
transmission-daemon-2.84_3.txz failed dependency check.
transmission-gtk-2.84_5.txz failed dependency check.
freeciv-2.5.2.txz failed dependency check.
freedink-108.4.txz failed dependency check.
git-review-1.24_2.txz failed dependency check.
osmo-0.2.14.txz failed dependency check.
firefox-44.0,1.txz failed dependency check.
midori-0.5.11.txz failed dependency check.
ap24-mod_security-2.9.0.txz failed dependency check.
nspluginwrapper-1.4.4_5.txz failed dependency check.
webkit-gtk3-2.4.9.txz failed dependency check.
clamav-0.99.txz failed dependency check.
blender-2.76b.txz failed dependency check.
feh-2.14.txz failed dependency check.
gimp-2.8.16,2.txz failed dependency check.
gstreamer1-plugins-opencv-1.6.3.txz failed dependency check.
mupdf-1.8,1.txz failed dependency check.
openshadinglanguage-1.6.6.txz failed dependency check.
virtualbox-ose-4.3.34_1.txz failed dependency check.
qt4-qtconfig-4.8.7.txz failed dependency check.
icedtea-web-1.5.2.txz failed dependency check.
thunderbird-38.5.0.txz failed dependency check.
codelite-9.0.txz failed dependency check.
libreoffice-5.0.4_1.txz failed dependency check.
cclive-0.7.16.txz failed dependency check.
gtk-youtube-viewer-3.2.0.txz failed dependency check.
redports-node-0.1.2.txz failed dependency check.
transmission-2.84.txz failed dependency check.
fvwm-crystal-3.0.6_8.txz failed dependency check
```
gimp-app, blender, thunderbird, virtualbox-ose, webkit-gtk3,  ibreoffice !!! .... libreoffice alone is 2 hours, all will be about between four and five hours ... that is impossible to do.
How I wrote above  three hours per day or a whole day in the week...

That's enough ...finished, forget it.


----------



## marino (Feb 1, 2016)

talsamon
If you don't get why this commit to CURL would do that, I don't know if you can be helped:
http://www.freshports.org/commit.ph...=201602011704.u11H4DSb054094@repo.freebsd.org

It's freaking CURL.  326 ports depended on it.  What do you possibly expect to happen?


----------



## marino (Feb 1, 2016)

If you've given up: maybe it's a good thing.  You seem to have some misunderstandings (or disagreements) how building should work and those conceptions seem to be a source of frustration for you.   I would advise not rebuilding daily, because stuff like bumping curl will drive you crazy.


----------



## talsamon (Feb 1, 2016)

Did you not understand,  that is impossible to do. The system is the half day busy with recompiling something. That is much too extreme. There must be an other way. I hope portmaster (I don't like It) will never deprecated.
If this the only way the concept of shared libraries failed.


----------



## marino (Feb 1, 2016)

1) you are free to build a tool that works like you think it should
2) portmaster has same constraints.  If you choose not to rebuild dependencies, you will break your system at some point, pretty much guaranteed.
3) Who said you have to build daily?  Why not once a week, at night?  Or once every 2 weeks, month?  Your pain is self-inflicted.
4) it's quite possible.  Have you noticed nobody else is agreeing with you?  What does that say?

Thanks for giving it a try, and I'm sorry Synth is not working out for you.  Portmaster's future is not bright and I recommend you at least use poudriere, but it acts like synth so you will probably hate it too.


----------



## talsamon (Feb 1, 2016)

```
Have you noticed nobody else is agreeing with you?
```
Yes and I wonder about. I  doubt everybody has the capacity to do this.

Ok, I try it once per week, but these will also bulk of packages and hours to compile.


----------



## marino (Feb 1, 2016)

do what, exactly?
1) you are on the ports head, not the quarterly.  The head moves fast.
2) just about every day, somebody makes a commit the breaks hundreds if not thousands of ports.  Libreoffice and KDE pretty much break every day
3) if you try to build every day, this is what you and everyone should expect

thus: don't build every day.

Remember what synth is doing: building an up-to-date repository for 1 or more systems to update themselves.  By it's very nature, it's compiler intensive.  You should be doing it periodically, not continuously.

Also note that you are not updating single packages, but the entire system.  Maybe `synth upgrade-system` is not for you.  Maybe you should identify specific packages you to update and just update those, and then stuff like curl won't bite you.  In other words, you have a very demanding concept of operations, but you are unwilling to accept the consequences.  You either need to change your operations or change your expections.  (dropping synth is a way of changing both)


----------



## kpa (Feb 1, 2016)

I think I've tried to explain this already many times but the crux of the matter is dynamic linking done by ld.so(1) and the way link library dependencies are so incredibly inflexible as done in almost every other UNIX/UNIX-like OS. We have no choice on the matter because FreeBSD doesn't provide its own methods/ways doing those, they were all taken as given from other systems where those features were developed and while alternatives do exist nobody seems to use them. Also the ELF file format that is behind all this is taken for granted as the standard file format, everyone expects that ld.so(1) works more or less the same on all UNIX/UNIX-like OSes.

In a nutshell you have port A that provides a shared library libfoo.so.1. That library file exports a set of symbols that makes the API/ABI of the module libfoo and its version this API is 1 as noted by the suffix. Now there's port B with an executable program named foobar that needs the library libfoo.so.1 so naturally it gets it from port A, B now depends on A. What is done with this program foobar is that at the linking stage of the build of the executable a reference is inserted to the shared library it depends on so that the dynamic linker will know to load the shared library, libfoobar.so.1, also references to the unresolved symbols to this shared library are left in the executable and those are resolved by the dynamic linker when the program is loaded.

The tricky parts is that if there is ever an update to the port A that causes the API/ABI provided by libfoo.so to change in such a way that the version number has to be bumped up to 2 (making the library libfoo.so.2) it causes a problem for the program foobar in port B because it's now hardwired to load libfoo.so.1 by the dynamic linker at startup.  So what is the solution you ask? Of course rebuilding of the executable foobar with the updated port B installed with the new library libfoo.so.2 so that the correct library is referenced and more importantly the executable foobar is made to use the correct API/ABI because of the change in the dynamic link library. Now the real kicker, this solution (rebuild of the dependent executable) is the only solution in existence if using direct dynamic linking with ld.so(1) or dlopen(3)/dlsym(3) methods. Now think about what this means when you have a port that is used by a huge number of other ports such as the mentioned ftp/curl and then the dependents of the ports that depend on ftp/curl.

There are other ways of dealing with the dependencies and API/ABI concerns that are very much superior from the application's and the developer's point of view, the most widely used is the COM/COM+ as you find it in MS Windows:

https://en.wikipedia.org/wiki/Component_Object_Model#COM.2B_and_DCOM


The downside of those methods is that they introduce extra baggage that is fine on user application level but may not be desirable on the system service/utility level.


----------



## marino (Feb 2, 2016)

I just pushed version 0.99_3.  Note a major change of action in the `synth build-repository` (it was renamed and new command took over the name).  The removal of "Escape" for graceful shutdown was a big change too.


```
build/synth: bug fixes, new features

WARNING: rebuild-repository command has changed action!  see below!

The follow changes have been made since v0.99_2:
  * Change the graceful shutdown key from "Escape" to Control-C.
    The former was easy to hit inadvertently (reported) and could be
    interfered with by terminal ANSI codes and/or mouse wheels.  The
    documentation has also been modified to reflect this change.
  * Fixed bug where installed packages that no longer had a port
    might cause the scan to fail rather than be ignored as advertised
  * New feature: SYNTHPROFILE environment variable
    When SYNTHPROFILE is set toTill be loaded rather than the default
    profile.  This is aimed for synth's use in scripts.
  * The "rebuild-repository" command has been renamed to "prepare-system".
    This is partly because the former command will be repurposed.
  * A new command assumed the name "rebuild-repository"; it performs a
    sanity check on all the built packages, removes the bad ones, and
    rebuilds the local Synth repository on command.  It is primarily for
    scripting use, but it has other legitimate uses.
  * Fix case where prefetching packages would try to update a non-existent
    local Synth directory.  As a consequence, prefetching is only done
    from a single external repository (the normal use case thought)
```


----------



## marino (Feb 2, 2016)

Oops, there's a bad typo in the commit message.  The new shutdown sequence is *Control-Q*, not Control-C !


----------



## protocelt (Feb 2, 2016)

marino@ said:


> Oops, there's a bad typo in the commit message.  The new shutdown sequence is *Control-Q*, not Control-C !



Seen that and was about to post a message about the parade of disgruntled synth users coming shortly to post it's broken. 

After using Synth for the last couple of weeks now, I'm really liking it. Much easier to setup than Poudriere and seems subjectively faster as well.


----------



## marino (Feb 2, 2016)

There's an earlier post saying a specific job takes poudriere 24 hours but Synth does the same job in 18 hours (on the given machine).  So it's more than "subjectively" faster.


----------



## protocelt (Feb 2, 2016)

I have 900+ ports installed on my workstation. Last repository rebuild I did, with the ccache(1) cache fully populated, Synth took around 3.5 hours to rebuild the entire repository. I don't know how long Poudriere took on the same machine but I know it was longer. I thought there was a mistake and ports were not getting rebuilt, but after checking, they were all in fact rebuilt. It is quite fast.

Again, I appreciate the tool and work your putting into it. Thanks!


----------



## fernandel (Feb 2, 2016)

I am also using Synth from the first day and after all this years which I "spent" with portmaster I am so happy and thankful that is Synth in the ports and for Marinos help too.
Thank you.
BTW: I start synth-upgrade system before bad time and in the morning is everything done .


----------



## PacketMan (Feb 3, 2016)

protocelt said:


> Last repository rebuild I did, with the ccache(1) cache fully populated, Synth took around 3.5 hours to rebuild the entire repository.
> ...............
> Again, I appreciate the tool and work your putting into it. Thanks!



I need to understand devel/ccache and its requirements. Memory, CPU, etc. I'd love to use this.  Feel free to PM me, anyone!

Yes its a great tool; I can't wait to roll it out to all my machines.


----------



## protocelt (Feb 3, 2016)

PacketMan said:


> I need to understand devel/ccache and its requirements. Memory, CPU, etc. I'd love to use this. Feel free to PM me, anyone!


Take a look at the ccache(1) man page and website for more information on what it is and how it works, then create a new thread to ask any more specific questions you might have.


----------



## kpa (Feb 3, 2016)

protocelt said:


> I have 900+ ports installed on my workstation. Last repository rebuild I did, with the ccache(1) cache fully populated, Synth took around 3.5 hours to rebuild the entire repository. I don't know how long Poudriere took on the same machine but I know it was longer. I thought there was a mistake and ports were not getting rebuilt, but after checking, they were all in fact rebuilt. It is quite fast.
> 
> Again, I appreciate the tool and work your putting into it. Thanks!



Keep in mind that ports-mgmt/poudriere goes to very great lengths to ensure an absolutely clean build environment and that includes providing a pristine base system for every single individual build, that is implemented by jail cloning which is quite slow on UFS and on ZFS is still a minor slowdown. Synth on the other hand makes use of the running live system by nullfs(5) mounting the needed directories over the chroot(8) environment which is very fast.


----------



## marino (Feb 3, 2016)

I would argue that synth chroots are just as clean and pristine as poudriere's.
The only thing the jails buy you over chroot are:
1) Jails can kill rogue processes efficiently where those may cause issues for synth (inability to dismount, etc).  I tried to implement watchdog to address that but it's only 99% effective.  that's still a work in progress
2) Jails can turn off network per phase (desirable for it's purpose) but I don't care about that

Other that #1, there's not much advantage to the jail over chroot for a typical user.


----------



## marino (Feb 3, 2016)

I pushed a relative minor update to synth today:


```
ports-mgmt/synth: new feature, minor bug fixes and improvements

This is a minor update to synth, which includes:
  * Support for the WHYFAIL environment variable.  If this variable
    is defined (to any value) in the environment, Synth will turn on
    the "debug" mode for dependency and option sanity checks.  This
    mode will provide exact details on how the package failed the check.
  * README.md: editorial corrections, 3 images replaced to reflect current
    version of Synth
  * Man page: editorial correction, WHYFAIL documented, and the "Impulse"
    indicator was documented (in NOTES section)
  * Significantly improve ports scan error messages.  In particular,
    eliminate the 'bad value ""' messages that are caused by ports that
    are partially or completely missing.  Also propagate exception
    messages when helping.
  * Log 03 (ignored ports) did not list the actual ports, only the reason
    the port was ignored.  Fix bug to show category/port too.

Erratum on previous commit message: The "Graceful Shutdown" is initiated
with Control-Q, not Control-C!  The typo is doubly unfortunate because
Control-C will exit Synth without cleaning up the mounts.
```


----------



## fernandel (Feb 3, 2016)

One question more, please:
I have all packages built and rebuilt to the last version. Is it good idea to reinstall everything?
Thank you.


----------



## kpa (Feb 3, 2016)

fernandel said:


> One question more, please:
> I have all packages built and rebuilt to the last version. Is it good idea to re install everything?
> Thank you.



It's not really needed but `# pkg upgrade -f` is safe to run (if it wasn't we'd have big problems) and usually doesn't take too long. It's needed when you switch from one repository to another or there is a major change in the repository you're using such as a change in the default version of perl.


----------



## marino (Feb 3, 2016)

I don't think that is necessary.  On the other hand, it doesn't hurt anything to do that either.
I do this "bulletproof" update technique form time to time to remove unwanted packages.  You could the same perhaps:
https://www.dragonflybsd.org/docs/howtos/HowToDPorts/#index4h1 

basically you'd remove everything and reinstall the key packages from Synth repository and it would pull in the dependencies as a result, so there would be no extra packages installed at the end.


----------



## marino (Feb 4, 2016)

I pushed 0.99_5 to fix a minor regression that nobody told me about. 


```
ports-mgmt/synth: Fix end-run regression

A fairly recent change caused a regression after a build was complete.
Previously a "tally" or summary of the build would appear after the
ncurses screen was restored to the regular terminal mode.  It would
list how many ports were built, failed, etc.  After the regressin, it
just ended abruptly.

This commit restores the tally to show as it did previously.
```


----------



## Crivens (Feb 4, 2016)

Minor Feature Request:
aaaa deeefffaaaulllt niiicceee vvaaaallluuuee please


----------



## marino (Feb 4, 2016)

??


----------



## kpa (Feb 4, 2016)

Perhaps he means nice(1)?


----------



## marino (Feb 4, 2016)

But I still don't get it.  by it's nature, building is greedy.  (jobs/builders can be modified with different profiles, so launch a "nicer" profile instead, right?)


----------



## protocelt (Feb 4, 2016)

Crivens said:


> Minor Feature Request:
> aaaa deeefffaaaulllt niiicceee vvaaaallluuuee please


And this^ is why we can't have nice(1) things. 

I just run a `# synth configure` quick to change the number of builders/jobs before building if I need to use the system at the same time.

Edit: oops, already mentioned by marino@


----------



## Crivens (Feb 4, 2016)

I'll stick to nice(1) synth itself. But the default config just hogs all resources, which is good when you want to have it done faster. If you want to have anything else done while building, this is not so good.


----------



## marino (Feb 4, 2016)

protocelt
Try this: create a second profile, call it, say, "Nice"
Set the builders and jobs low.

Then launch synth with `env SYNTHPROFILE=Nice synth upgrade-system` or something like that.

Crivens can do the same.


----------



## protocelt (Feb 4, 2016)

I bring up the configure screen to check that things are in order after each minor update you push out before building so I just change it from there since I'm already at that screen. I won't likely do that after Synth is tagged feature complete/bug free. SYNTHPROFILE is mentioned in the ENVIRONMENT section of the man page so I was aware of that, but thanks for the pointer. 

I do have a second profile and have tried that already. No problems.


----------



## marino (Feb 4, 2016)

it's been feature complete since 0.99 --- at least as far as the features I think 1.00 should have that is.
I'm not aware of any bugs right now.
The only thing preventing 1.00 from being tagged is time.  If a few days goes by and nobody finds any new problems, I'll do that.
In any case, it's damn close.


----------



## protocelt (Feb 4, 2016)

marino@ said:


> it's been feature complete since 0.99 --- at least as far as the features I think 1.00 should have that is.
> I'm not aware of any bugs right now.
> The only thing preventing 1.00 from being tagged is time.  If a few days goes by and nobody finds any new problems, I'll do that.
> In any case, it's damn close.



Out of curiosity, what are some of the future features you're considering adding?


----------



## marino (Feb 4, 2016)

On the short list is caching tree scans (only scan tree after it's changed) because those take minutes.  4.5 minutes for me, but I just heard somebody quote 20 minutes to 60 minutes.  Maybe ability to delete profiles.  Then we'll see what other people request (that can't be handled in other ways)


----------



## protocelt (Feb 4, 2016)

I didn't realize the scan was taking so long for some people. It takes about 45 seconds or so on my main machine and I'd say about 3 minutes max on the other one so I never gave it a second thought. Anyway, I'm sure a speed increase of any kind would be appreciated.

One thing that might be nice is adding a way to add/remove make(1) options for ports in the configure screen instead of manually editing Livesystem-make.conf(or whatever profile your using). Just a thought.


----------



## Crivens (Feb 4, 2016)

`/usr/bin/time synth status`
       82.11 real        58.41 user        42.57 sys

It's scanning 930 installed packages on a pretty old core2duo laptop. I'd say ZFS on SSD can make up for some old age.


----------



## Crivens (Feb 5, 2016)

One thing that came to my attention might be to block the different builders from staging/building ports in parallel which are huge and have a big set of dependencies. Especially when you use tmpfs for this. When for example libreoffice and firefox try to get staged at the same time, this will hit the swap pretty hard and performance will tank completely. Maybe as a rule of tumb you can count all the dependencies and allow only for a maximum to be active at any time, then schedule the builds accordingly? But that might be something for maybe 3.0


----------



## marino (Feb 5, 2016)

it was in the original design but I did not implement it because I never hit it myself.  If that is happening the builders/jobs settings are wrong.  You should always increase jobs over builders when memory is in play.  My suggestion is reduce number of builders and increase number of jobs (e.g. 4 builders x 3 jobs => 3 builders x 4 jobs)


----------



## marino (Feb 5, 2016)

in other words, synth won't prevent somebody from shooting their foot off.  
Did you keep the suggested builders/jobs from the initial configuration, or did you modify those settings?


----------



## talsamon (Feb 5, 2016)

Again, it is not clear. security/gnutls update. multimedia/ffmpeg is depend of security/gnutls, compiles and installs - ok. X11-wm/fvwm-crystal is depend of multimedia/ffmpeg. Fvwm-crystal compiles *and does not install*. What sense it makes to  compile and does not install it? If anything is changed in fvwm-crystal, itself  and the system will nothing "know"  about, and if nothing changed it is not necessarily.


----------



## marino (Feb 5, 2016)

talsamon what are you implying?  Are you trying to say this is a bug?  Why do you feel qualified to discuss what is "necessary" and "not necessary"?  You've made several inaccurate technical statements already.  If you believe those statements, fine, but I know Synth build logic has no flaws in it.


----------



## talsamon (Feb 5, 2016)

Maybe, I am wrong. But that is not logical. Any explanation? (but I fear I will get none).


----------



## marino (Feb 5, 2016)

half of this 16 page thread has been dedicated to these observations by you and I'd just as soon not continue doing that (it's noise that can confuse others and give them the wrong idea).  If you aren't confident in synth logic or if you prefer another approach, then just don't use the tool.


----------



## Crivens (Feb 5, 2016)

marino@ said:


> in other words, synth won't prevent somebody from shooting their foot off.
> Did you keep the suggested builders/jobs from the initial configuration, or did you modify those settings?


I kept the initial settings. That was two builders with two jobs each, for a core2duo with 4GBytes of memory. I switched it to "tmpfs" where possible to keep the wear of the SSD down a bit. Maybe that was a mistake as the swapping will have done some wear on it's own. And I added devel/ccache to the settings.

Maybe it is that ZFS is holding on to it's memory with more strength than anticipated? This might mess heuristics up somewhat.


----------



## marino (Feb 5, 2016)

If you can run tmpfs(5), then do that.  Only turn if off if you have no choice.
Personally I've not seen loads higher than 10 or swap higher than 5% on my machine.
2x2 is pretty low.  maybe 2x1 would be better?  I recommend at least 2 so at least you can fetch and build at the same time (unless you have the bad luck of both builders fetching).  
Santa Claus didn't bring a modern machine this year? 
Maybe building and doing something other than building isn't going to work well on that machine, no matter what.


----------



## Crivens (Feb 6, 2016)

marino@ said:


> Santa Claus didn't bring a modern machine this year?


Not this year, I have been a bad boy. For some time.  Maybe this year, or next. I usually use any hardware untill it breaks down completely, for environmental reasons among others. Also, there is currently no laptop which is affordable, worth it's money and trustworthy (sorry Lenovo, you are out).


----------



## kpa (Feb 6, 2016)

I noticed that the log files under /var/log/synth could use some more organization, poudriere for example uses subdirectories with timestamps in their names to separate logs from each build run to their own subdirectories.


----------



## marino (Feb 6, 2016)

kpa said:


> I noticed that the log files under /var/log/synth could use some more organization, poudriere for example uses subdirectories with timestamps in their names to separate logs from each build run to their own subdirectories.



And poudriere's choice causes a ton a problems.  each full bulk run costs 2Gb of disk space.  Poudriere has no inbuilt way to clean logs and boom: suddently you have 20G of logs.

I made a conscious and informed decision to handle logs differentfy and intentionally only retain the last log and not every log that's every been attempted.  It's also a conscious decision to base log names on port origins and not package names.  So this is not an oversight.  Poudriere has a different purpose and timestamped logs make sense for that purpose.  I'm not interesting in following poudriere lead on logs as its been working quite well in my opinion (with the only real criticism that apache doesn't like to display 20,000 logs in one virtual directory, but that's an apache problem)


----------



## fernandel (Feb 6, 2016)

kpa said:


> It's not really needed but `# pkg upgrade -f` is safe to run (if it wasn't we'd have big problems) and usually doesn't take too long. It's needed when you switch from one repository to another or there is a major change in the repository you're using such as a change in the default version of perl.


`# pkg upgrade -f` and everything was fast and no errors (1689 packages).


----------



## gkontos (Feb 6, 2016)

marino@ It is difficult to follow up this thread. However, I wanted to congratulate you for your excellent job. I really hope that more developers can get inspired by your patience and innovation.


----------



## marino (Feb 6, 2016)

gkontos, thanks!
Don't worry, when 1.00 is released, I'll start a new thread and hopefully it won't get muddied up.
I was hoping 0.99_5 was the last release before 1.00, but it looks like there will be one more.  (As a bonus, a new dev feature that I was saving for the next release could make it into 1.00)


----------



## marino (Feb 7, 2016)

```
ports-mgmt/synth: bug fixes and feature for port developers

I had hoped that 0.99_5 would be bug-free and the basis for the first
release (1.00), but couple were found.  If use of 0.99_6 reveals no
further issues after a week or so, I'll re-release it as v1.00.

bugs fixed:
  * if the origin started with a directory separator, an exception would
    occur.  Now it properly labels it as an invalid origin.
  * The "extract" stage was labelled as "checksum".  Internally everything
    was fine, but on the display, the order was checksum, extract-depends,
    checksum instead of checksum, extract-depends, extract.
  * During one phase (build), the DEVELOPER flag was set unconditionally.
    This was a regression as it wasn't always the case.  This code was
    tweaked several times since 0.99_5 and now the DEVELOPER setting has
    been moved the builder's make.conf to ensure it's consistently
    present or absent (as needed).
  * It turns out that the ports tree scan is affected by the DEVELOPER
    flag.  It turns out the setting can affect the dependencies list so
    it needs to be set (or absent) appropriately to match how it will be
    on the builders.  The make.conf solution above solves this too.
  * There was "NO_BACKUP" set in the builders make.conf.  This line is
    for the DragonFly src builder and it's presence caused no harm, but
    it's been removed now.
  * Make ports makefile respect CFLAGS
```


```
new feature:
  * Provide ability to break into a build at a specific point and
    interact with it.
      -  Only available on "test" command
      -  Only active when one (1) port origin is given to "test" command
      -  Only active when ENTERAFTER is defined in environment as:
           > extract
           > patch
           > configure
           > build
           > stage
           > install
           > deinstall
      -  All dependencies are built first with typical display and
         DEVELOPER=1 set.  Afterwards, Synth converts to text mode and
         builds the specified port up to and including the phase specified
         by ENTERAFTER.  Then it launches a tcsh shell and gives control
         to the user at the builder's root directory
      -  The user ends the interactive session with the shell cmd "exit"
      -  Synth will clean up and exit (it will not try to continue the
         build due to possible corruption from the users)
   * This is a port developers tool.  The average user does not need it.
   * The average user might use "test" command to generate a log to submit
     as a FreeBSD Bugzilla PR attachment.
```


----------



## basbebe (Feb 7, 2016)

Thank you so much marino@ for this great new tool!

I have two questions that I hope someone can help me with or point me in the right direction:

To use ccache and tmpfs, do I need to do anything besides installing ccache? Do I need to set specific settings in ccache or establish specific mounts for tmpfs?

If I build ports from a list to make them available to other systems (jails) via null mounts: How to make sure all dependencies for e.g. mariadb are considered correctly in the target system? Would I just add something in the LiveSystem-make.conf like WANT_MYSQL_VER=    100m and add all the packages from the target system that could need mysql to the list?


----------



## marino (Feb 7, 2016)

Unless your machine is critically low on memory, you should already have tmpfs as the default (check with `synth command`).  If synth defaults to tmpfs=off then you probably don't have much RAM so building with tmpfs may not be a good idea.

for ccache, you install it separately and configure it (per man page or type `ccache` for a list of commands).  When it's configured, type `ccache -s` and use the value for *cache directory *(which you may have configured already) and put that directory in option [H] of `synth configure`.  That's it, synth will use ccache after that.



> If I build ports from a list to make them available to other systems (jails) via null mounts: How to make sure all dependencies for e.g. mariadb are considered correctly in the target system? Would I just add something in the LiveSystem-make.conf like WANT_MYSQL_VER= 100m and add all the packages from the target system that could need mysql to the list?



No, you only need one port origin (databases/mariadb100-server) on your list.  Synth will build the dependencies and put everything in the local repository when you decide to build it.  You only need to specify the leaf ports.


----------



## basbebe (Feb 7, 2016)

marino@ said:


> for ccache, you install it separately and configure it (per man page or type `ccache` for a list of commands).  When it's configured, type `ccache -s` and use the value for *cache directory *(which you may have configured already) and put that directory in option [H] of `synth configure`.  That's it, synth will use ccache after that.



So I won't have to add anything to make.conf or LiveSystem-make.conf?



marino@ said:


> No, you only need one port origin (databases/mariadb100-server) on your list.  Synth will build the dependencies and put everything in the local repository when you decide to build it.  You only need to specify the leaf ports.


Will this also work if the host system doesn't have mariadb100 installed but it's only in the list? I know that it won't be linked correctly for the host system but will it still work for the targets?
And (hypothetically, just out of curiousity) what would happen if I had mysql56 and mariadb100 both on my list?


----------



## marino (Feb 7, 2016)

basbebe said:


> So I won't have to add anything to make.conf or LiveSystem-make.conf?



no, you don't add anything to those files.



> Will this also work if the host system doesn't have mariadb100 installed but it's only in the list? I know that it won't be linked correctly for the host system but will it still work for the targets?
> And (hypothetically, just out of curiousity) what would happen if I had mysql56 and mariadb100 both on my list?



You can add whatever you want, and yes it always builds the dependencies.
A repository is a collection of packages.  The host system (or any system) that uses the respository only installs a subset of the packages.

Extreme example:  I can tell synth to build every port in the tree, and afterwards when I run the `synth upgrade-system` command, only a few packages are replaced.  There's no relationship about what is installed on system and what's in the local repository.  All the "prepare-system" and "system-upgrade" commands really do is give a port build list to synth so it can update / add those to the local repository.

more clear?


----------



## basbebe (Feb 7, 2016)

marino@ said:


> You can add whatever you want, and yes it always builds the dependencies.
> A repository is a collection of packages.  The host system (or any system) that uses the respository only installs a subset of the packages.
> 
> Extreme example:  I can tell synth to build every port in the tree, and afterwards when I run the `synth upgrade-system` command, only a few packages are replaced.  There's no relationship about what is installed on system and what's in the local repository.  All the "prepare-system" and "system-upgrade" commands really do is give a port build list to synth so it can update / add those to the local repository.
> ...



Thanks, yes.
I'm still not entirely sure as to how I would replace mysql56-client with mariadb100-client.
If I had a list with e.g. dovecot2, postfix, php56-mysql and mariadb100-client, would all these packages built with mariadb100-dependencies?


----------



## marino (Feb 7, 2016)

Unless you specify otherwise, they build with the default ports options.
There are two ways to customize options:
1) go to port and run `make config` and save the results
2) put in global options in the LiveSystem-make.conf file that you create. 
You won't be able to support two different mariadb defaults in the same repository.  All the systems that share a repository have the same package configurations (should be obvious).


----------



## marino (Feb 7, 2016)

by the way, dovecot has no dependency on mariadb.  php56-mysql probably doesn't either.  So the answer is "no", just putting them in the same portlist does not influence dependencies in any way.


----------



## basbebe (Feb 7, 2016)

marino@ said:


> by the way, dovecot has no dependency on mariadb.  php56-mysql probably doesn't either.  So the answer is "no", just putting them in the same portlist does not influence dependencies in any way.


Thanks for the clarification!

I put WANT_MYSQL_VER= 100m into LiveSystem-make.conf and now `synth status /usr/local/etc/synth/jail_ports.txt` shows:


```
These are the ports that would be built:
  => mail/dovecot2
  => databases/p5-DBD-mysql
  => databases/py-MySQLdb
  => mail/dovecot2-antispam-plugin
  => mail/dovecot2-pigeonhole
  => mail/opendmarc
  => mail/postfix-current
```

Thanks!

Looking forward to a great synth github wiki


----------



## fernandel (Feb 7, 2016)

I have one ignored port, sysutils/fusefs-expat:

```
sysutils/fusefs-exfat: License Microsoft-exFAT needs confirmation, but BATCH is defined
```
Should I modify Makefile, please?
Thank you.


----------



## hukadan (Feb 7, 2016)

Try adding the following line to make.conf.

```
DISABLE_LICENSES=yes
```
You can have a look to the /usr/ports/Mk/bsd.licenses.mk file for more details (and options).


----------



## marino (Feb 8, 2016)

yes, I should probably add DISABLE_LICENSES to the main make.conf as well for release 1.00


----------



## marino (Feb 8, 2016)

although maybe I shouldn't and just let people add "DISABLE_LICENSES" as needed.  The current situation might be the best.


----------



## marino (Feb 8, 2016)

yes, there's a LICENSES_ACCEPTED and LICENSES_GROUPS_ACCEPTED that would be appropriate to set.


----------



## Uniballer (Feb 8, 2016)

I built ports-mgmt/synth and used it to build a set of ports I use on certain headless systems (amd64).  I was very impressed at how smoothly it went.  Nice work.

I discovered that I can clean up the mounts and directories from an interrupted build by running `synth status`.  I didn't see that in the man page, did I just miss it?

Is there a way to tell synth to get rid of everything and start over?  I mean, get rid of all of its live packages, logs, etc.  I know I can delete them from /var/synth/live_packages/All and /var/log/synth, but is that really everything?


----------



## marino (Feb 8, 2016)

any major command (read: not help or version, but everything else) will clean up dirty mounts if detected.  Do you know about graceful exits?  (Control-Q).  Just breaking the build with control-C is not recommended.  I don't remember if auto-cleanup is on the man page or not.

`rm -rf /var/synth/live_packages/*` should do it.  Do you really care about the logs?  They'll get overwritten.   You could also remove /usr/local/etc/synth/* but that will blow away your configuration, so you probably don't want to do that.


----------



## marino (Feb 8, 2016)

Uniballer, yes it is in the man page.  See "NOTES" section, "Graceful Exit".  There's a comment about cleaning up dirty mounts if the build was hard-interrupted.


----------



## PacketMan (Feb 8, 2016)

marino@ said:


> You can add whatever you want, and yes it always builds the dependencies.
> A repository is a collection of packages.  The host system (or any system) that uses the respository only installs a subset of the packages.
> 
> Extreme example:  I can tell synth to build every port in the tree, and afterwards when I run the `synth upgrade-system` command, only a few packages are replaced.  There's no relationship about what is installed on system and what's in the local repository.  All the "prepare-system" and "system-upgrade" commands really do is give a port build list to synth so it can update / add those to the local repository.



Sorry if this seems redundant:  So can I

have my fastest machine build all the packages using the `synth just-build` command?
then copy required packages (or all of em) to the other machines repositories? (with a `portsnap fetch update` on each one too)

then issue `synth upgrade-system` and it will use those freshly built packages instead of building locally from scratch, to upgrade only what is installed?
in the event one machine has different make options configured, it will not use that package and create its own by building from scratch?

And suppose I want my 'master machine' to update (rebuild) all the packages using the new updates?  Do I do a `portsnap fetch update` followed by a `synth rebuild-repository` then copy the packages over to the other machines, and then issue `synth upgrade-system`?

Its still not clear to me when I would use `synth rebuild-repository` versus `synth prepare-system`.


----------



## marino (Feb 8, 2016)

> have my fastest machine build all the packages using the synth just-build command?


Not normally.  Normally you would only issue this command for ports that haven't been added to the system yet.  Once they are on the host system (in pkg database) synth will rebuild *if necessary* with prepare-system or upgrade-system commands


> then copy required packages (or all of em) to the other machines repositories (with a portsnap fetch update on each one too)


no.  Why would you?  Just tell synth the URL (file:///, http:///, etc)
it's possible I don't understand what you mean but it sounds very very wrong.


let me stand back and start again.  It soulds like you want a common repo with multiple consumers.  So in that case, you want a fixed list of port origins.  That list represents what is required to be in the target repository.
You could script it so synth does a "just-build" with that list and it will rebuild incrementally.
Then the script would follow with a "rebuild-repository" command.

"system-prepare" and "upgrade-system" are commands only for the host system.  In this scenario you wouldn't use them.  You would just manually issue the `pkg upgrade -a` command on each repository consumer.

Did I get us both closer?


----------



## PacketMan (Feb 8, 2016)

marino@ said:


> Did I get us both closer?


Not sure.  

Compiling ports from scratch drives me bonkers. It is very time consuming, especially on my older machines, one port alone can take hours.  So I was thinking I would have a fast server do all the building/compiling work. It would contain all the packages for whats installed on all the other machines (and including itself).  Then when ports tree updates are available, I want to do any and all rebuilding on the fast server, then push the packages to the other machines, and then tell them to simply 'upgrade' to the new packages, with no local build/compiling required, unless of course a package can't be reused because make options are different.

I don't want the machines to share a single common repository. I want each to have its own, and I want to manually copy packages from the master fast server to the slower machines.

See what I mean?


----------



## marino (Feb 8, 2016)

PacketMan
I was with you until you said "I want them all to have their own repo".
The normal scenario is have your fast machine build one repo and all the other machines would connect to it somehow.
You _can_ duplicate it X times, but why would you?  what's the benefit?  you have to rsync everything... for a gain I don't see (unless the machines aren't connected by the same network, but then how do you copy gigabytes of data ...?)


----------



## PacketMan (Feb 8, 2016)

marino@ said:


> PacketMan
> The normal scenario is have your fast machine build one repo and all the other machines would connect to it somehow.
> You _can_ duplicate it X times, but why would you?  what's the benefit?  you have to rsync everything... for a gain I don't see (unless the machines aren't connected by the same network, but then how do you copy gigabytes of data ...?)



Bingo! My machines are not on the same LAN.  I intend on using Bittorrent SYNC to do this.  I can do my building on one machine one day, and then launch the upgrades on other machines another day. And in the event my main builder dies, I can use my next fastest machine to take over, or replace it with a brand new machine, and repeat the whole thing, and let Bittorent Sync to its magic!

So is there anything bad with what I want to do? I don't see it.


----------



## marino (Feb 8, 2016)

it's inefficient, but it will work.  The ideal is to have one common repository (and when the repo is rebuilt, the new packages are immediately available everywhere)


----------



## PacketMan (Feb 8, 2016)

marino@ said:


> it's inefficient, but it will work.  The ideal is to have one common repository (and when the repo is rebuilt, the new packages are immediately available everywhere)



Remember this phrase my freind, I learned it a long time ago and it has served me very well.  "_The best technical solution is not necessarily the best business solution._"

I will test and post later on the forum, likely in a new discussion topic.  Thanks again Marino for an awesome product. You have give me new hope and joy.


----------



## fernandel (Feb 9, 2016)

marino@ said:


> yes, there's a LICENSES_ACCEPTED and LICENSES_GROUPS_ACCEPTED that would be appropriate to set.



I put both in /etc/make.conf

```
LICENSES_ACCEPTED=yes
LICENSES_GROUPS_ACCEPTED=yes
```


----------



## marino (Feb 9, 2016)

Those are not "yes/no" questions.  You need to read that file to get the licence name.

It's like asking "What are the names of your friends" and you answering "yes"


----------



## talsamon (Feb 9, 2016)

Crivens says:


> When for example libreoffice and firefox try to get staged at the same time, this will hit the swap pretty hard and performance will tank completely...


On my box it freezes for seconds. It does not help to set 3 builders and 4 jobs or reduce it to 2 builders and 3 jobs.
And I have no old box (4-core). I have stopped all daemons and programs who could interfer. How can I limit synths "resourcehunger" ?? (had never such issue before and I had it not on the old i586 with FreeBSD).


----------



## marino (Feb 9, 2016)

You need to understand why it's freezing.  Even when builders/jobs are set too high, things don't "freeze"; they just happen slowly".  It sounds like a different issue.  It's probably just extracting or linking.  I don't know what you can do if the machine just isn't very new.


----------



## basbebe (Feb 9, 2016)

I'm using sysutils/tmux.
I have the feeling that every time synth builds and I quit the session, it freezes.
When I return I see the building screen again but it is frozen and `top` doesn't show a lot of load.


----------



## marino (Feb 9, 2016)

What does top(1) say?  Does `ps -auxww` change?
"not a lot of load" could easily be extracting a large file (or linking?)
figure out what it's doing. (or confirm it's not doing anything) on another tty


----------



## marino (Feb 9, 2016)

Oh, basbebe is a different person.
Tmux should work.  I use Screen with Synth just fine.


----------



## marino (Feb 9, 2016)

tux == tmux i assume?


----------



## basbebe (Feb 9, 2016)

tux = tmux
I edited the typo.

The screen didn't change, sometimes, it didn't even render completely.

I let it run for several minutes and nothing changed (either on the synth nor on the top side).
I'll check this the next time it happens.


----------



## marino (Feb 9, 2016)

Check the 00_* log to see what it was working on, to see if it's a specific port (and that it's repeatable if you try to build the same port)


----------



## marino (Feb 9, 2016)

I got a second report about audio/jack.  I had to use FreeBSD to find the problem (it only appears there) but it's been fixed now.  The port won't rebuild over and over anymore.


----------



## PacketMan (Feb 9, 2016)

talsamon said:


> PacketMan says:When for example libreoffice and firefox try to get staged at the same time, this will hit the swap pretty hard and performance will tank completely...




What the heck, I didn't say that! Why are you quoting me for saying something I didn't? Please don't even reply. Just delete that post.  PBCAC I say. I'm not even gonna reply.


----------



## marino (Feb 9, 2016)

Crivens said it I think, referencing an old machine.


----------



## talsamon (Feb 9, 2016)

Sorry for that PacketMan. Wrong address. marino@ is right, it was Crivens.


----------



## dave_overton (Feb 9, 2016)

Bug just found, might be (probably is) my fault, but reporting cause its got me stuck.  10.3-PRERELEASE FreeBSD 10.3-PRERELEASE #0 updated last with freebsd-update(8)

CORE DUMP FIXED, it was some leftover cruft from an old archiver install.  Apparently it was being triggered by synth instead of the newer version.  Found and killed it off, the core dump went away.

All good now, thanks for making me look further.

Core dump like this:  (can be any command, this one is just faster)

```
# synth status
Builder mounts detected; attempting to remove them automatically ...
Dismounting successful!
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Illegal instruction (core dumped)
```

gdb core:


```
Core was generated by `synth'.
Program terminated with signal 4, Illegal instruction.
#0  0x000000000047f374 in ?? ()
(gdb) bt
#0  0x000000000047f374 in ?? ()
#1  0x0000000000818c50 in ?? ()
#2  0x000000000045eb49 in ?? ()
#3  0x0000000000818b80 in ?? ()
#4  0x00007fffdffff120 in ?? ()
#5  0x00007fffdffff160 in ?? ()
#6  0x00000000000002c3 in ?? ()
#7  0x0000000000000034 in ?? ()
#8  0x0000000000476a0c in ?? ()
#9  0xf370bb4102c302c3 in ?? ()
#10 0xdffff040ba490047 in ?? ()
#11 0x90e3ff4900007fff in ?? ()
#12 0xcd90bb410045eb49 in ?? ()
#13 0xdffff040ba490046 in ?? ()
#14 0x90e3ff4900007fff in ?? ()
#15 0x00007fffdffff160 in ?? ()
#16 0x00007fffdffff090 in ?? ()
#17 0x00000000000002c3 in ?? ()
#18 0x000000000047f3e7 in ?? ()
#19 0x0000000000000000 in ?? ()
```

And pkg(8) no longer works on this system


```
# pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
Updating Synth repository catalogue...
pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
repository Synth has no meta file, using default settings
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Unable to update repository Synth
```

Suggestions on how to fix pkg(8)?  -->  Never mind, fixed that easy enough.


----------



## marino (Feb 9, 2016)

Remove the /usr/local/etc/pkg/repos/00_synth.conf file.
But pkg(8) error is just a warning.  Why do you think it's broken?  It's just failing to update a repo that doesn't exist.  I think pkg(8) still works.


----------



## dave_overton (Feb 9, 2016)

marino@ said:


> remove the /usr/local/etc/pkg/repos/00_synth.conf file.
> But pkg error is just a warning.  Why do you think it's broken?  it's just failing to update a repo that doesn't exist.  I think pkg still works.



Yep, figured that out, although it was still broke.  `pkg xxx` would just spit out those few lines you see, and return to the command line.  Deleting the 00_synth.conf killed the error of course.

Synth still won't run of course, core dumps every single time.


----------



## marino (Feb 9, 2016)

I assumed because you said it's "(probably) my fault" that you knew of something you did that might explain it.  I've never seen any reports on core dumps.  Did you build it with crazy settings in make.conf?  Why would pkg not work anymore?  Related?  System FUBAR?


----------



## marino (Feb 9, 2016)

By the way, if you get the first few lines from pkg with `pkg update`, it looks like it's working to me.


----------



## xtaz (Feb 9, 2016)

I seem to be stuck at the first hurdle. I've just installed it to give it a go but when I try and run it I just get this.

```
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Encountered issue with ports-mgmt/pkg or its dependencies
  => bad input for 'Value: ""
Unexpected pkg(8) scan failure!
Unfortunately, the system upgrade failed.
```

There is no log file generated, and I have a full ports tree with no missing files. Any ideas?


----------



## marino (Feb 9, 2016)

What version of synth do you have installed?  The latest is 0.99_6.  (you can check with pkg info synth)

The error suggests your ports tree is corrupt.


----------



## fernandel (Feb 9, 2016)

marino@ said:


> Those are not "yes/no" questions.  You need to read that file to get the licence name.
> 
> It's like asking "What are the names of your friends" and you answering "yes"


Thank you. I will try.


----------



## fernandel (Feb 9, 2016)

hukadan said:


> Try adding the following line to make.conf.
> 
> ```
> DISABLE_LICENSES=yes
> ...



No, it doesn't work for sysutils/fusefs-ex*f*at on my system.


----------



## xtaz (Feb 9, 2016)

marino@ said:


> What version of synth do you have installed?  The latest is 0.99_6.  (you can check with pkg info synth)
> 
> The error suggests your ports tree is corrupt.



It's synth-0.99_6, just freshly installed from the ports tree itself as I'm not currently using packages for anything at all. I've just ran `svn status -u /usr/ports` and it doesn't show that any files are wrong or missing. I could delete and fetch the tree via `svn` again if you think it might help?


----------



## marino (Feb 9, 2016)

it doesn't seem to be able to read /usr/ports/ports-mgmt/pkg.
Do basic queries like `make -C /usr/ports/ports-mgmt/pkg -V PKGNAME` work?


----------



## hukadan (Feb 9, 2016)

fernandel said:


> No, it doesn't work for sysutils/fusefs-ex*f*at on my system.



On my system without the option :

```
====>> Ignoring sysutils/fusefs-exfat: License Microsoft-exFAT needs confirmation, but BATCH is defined
build of sysutils/fusefs-exfat ended at Tue Feb  9 23:47:47 CET 2016
build time: 00:00:01
```
With the option:

```
===========================================================================
=======================<phase: package  >============================
===>  Building package for fusefs-exfat-1.0.1
===========================================================================
====>> Cleaning up wrkdir
===>  Cleaning for fusefs-exfat-1.0.1
build of sysutils/fusefs-exfat ended at Tue Feb  9 23:48:50 CET 2016
build time: 00:00:04
```
So I am sure this option should work. IMHO, if the option is written properly then you put it in the wrong file. I do not use ports-mgmt/synth but from what I understood, if you kept the default profile, you should put this option in the /usr/local/etc/synth/LiveSystem-make.conf file.


----------



## xtaz (Feb 9, 2016)

marino@ said:


> it doesn't seem to be able to read /usr/ports/ports-mgmt/pkg
> Do basic queries like `make -C /usr/ports/ports-mgmt/pkg -V PKGNAME` work?



I have just completely deleted and checked out the ports tree so I'm sure it isn't corrupt now. Running the command you said:


```
# make -C /usr/ports/ports-mgmt/pkg -V PKGNAME
pkg-1.6.3
```

Despite deleting and recreating the ports tree the error with synth is exactly the same where it mentions pkg.


----------



## marino (Feb 9, 2016)

i guess you'll have to show the contents of `synth configure` screen and tell me if there is anything unconventional about any of the directories.  I've never seen an error like that before.  the bad value errors shouldn't even happen anymore.  Something is seriously different about your configuration.


----------



## Wapcaplet (Feb 10, 2016)

marino@ said:


> i guess you'll have to show the contents of `synth configure` screen and tell me if there is anything unconventional about any of the directories.  I've never seen an error like that before.  the bad value errors shouldn't even happen anymore.  Something is seriously different about your configuration.


I have the same issue.  I'm running 10.2 on i386:


```
root@wintermute-pts/0(9:28pm)~/:synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Encountered issue with ports-mgmt/pkg or its dependencies
  => bad input for 'Value: ""
Unexpected pkg(8) scan failure!
Unfortunately, the system upgrade failed.
```
Just re-extracted the ports tree with `portsnap extract`, so it shouldn't be corrupted.

Here is my `synth configure` screen -- nothing unconventional:


```
Synth configuration profile: LiveSystem
===============================================================================
  [A] Ports directory  /usr/ports
  [noparse][B][/noparse] Packages directory  /var/synth/live_packages
  [C] Distfiles directory  /usr/ports/distfiles
  [D] Port options directory  /var/db/ports
  [E] Build logs directory  /var/log/synth
  [F] Build base directory  /usr/obj/synth-live
  [G] System root directory  /
  [H] Compiler cache directory  disabled
  [noparse][I][/noparse] Num. concurrent builders  2
  [J] Max. jobs per builder  2
  [K] Use tmpfs for work area  true
  [L] Use tmpfs for /usr/local  true
  [M] Display using ncurses  true
  [N] Fetch prebuilt packages  false

  [>]  Switch/create profiles
  [RET] Exit

Press key of selection:
```

And this command works fine:

```
root@wintermute-pts/0(9:35pm)~/:make -C /usr/ports/ports-mgmt/pkg -V PKGNAME
pkg-1.6.3
```


----------



## marino (Feb 10, 2016)

From that output, I know exactly which routine fails but not why.  There are only two possible causes:
1) the builder (jail area) didn't get constructed so the internal `chroot` command fails
2) There's something wrong with the Makefile, at least in that enviroment, that causes bmake to return with an error

I would think 1) would show an error earlier.

this is the command it's trying to run on the chroot:
	
	



```
/usr/bin/make -C /usr/ports/ports-mgmt/pkg \
-VPKGVERSION -VPKGFILE:T -VMAKE_JOBS_NUMBER -VIGNORE \
-VFETCH_DEPENDS -VEXTRACT_DEPENDS -VPATCH_DEPENDS \
-VBUILD_DEPENDS -VLIB_DEPENDS -VRUN_DEPENDS \
-VSELECTED_OPTIONS -VDESELECTED_OPTIONS \
-V_INCLUDE_USES_SCONS_MK -VUSE_LINUX
```

I'd ask if you can see mounts on /usr/obj/synth-live but it would construct and deconstruct so quickly you'd probably miss it.

FWIW, I can't reproduce on FreeBSD 11 i386 VM


----------



## marino (Feb 10, 2016)

If one of you two want to try this, it could help understand what's going on.  You'll need gcc6-aux installed (`pkg ins gcc6-aux`).

`cd /usr/ports/ports-mgmt/synth`
`make clean ; make patch`
`cd <WRKDIR>/synth-4417017/src/`
edit portscan.adb
go to line 562 and insert "TIO.Put_Line (content);".  it's right before "for k in result_range loop"
`cd /usr/ports/ports-mgmt/synth`

`make deinstall` (if you already have synth installed)
`make install`
run `synth status` command again and show output


----------



## xtaz (Feb 10, 2016)

My configuration is as follows:


```
Synth configuration profile: LiveSystem
===============================================================================
  [ A ] Ports directory  /usr/ports
  [ B ] Packages directory  /var/synth/live_packages
  [ C ] Distfiles directory  /usr/ports/distfiles
  [ D ] Port options directory  /var/db/ports
  [ E ] Build logs directory  /var/log/synth
  [ F ] Build base directory  /usr/obj/synth-live
  [ G ] System root directory  /
  [ H ] Compiler cache directory  /var/db/ccache
  [ I ] Num. concurrent builders  3
  [ J ] Max. jobs per builder  3
  [ K ] Use tmpfs for work area  true
  [ L ] Use tmpfs for /usr/local  true
  [ M ] Display using ncurses  true
  [ N ] Fetch prebuilt packages  true
```

I have some things in /etc/make.conf that could maybe affect it?


```
KERNCONF=TAO
SVN_UPDATE=yes
SVN=/usr/local/bin/svn
BATCH_DELETE_OLD_FILES=yes
WRKDIRPREFIX=/usr/obj
FORCE_PKG_REGISTER=yes
DISABLE_VULNERABILITIES=yes
OPTIONS_UNSET+=X11
WITH_OPENSSL_PORT=yes

DEFAULT_VERSIONS=apache=2.4 gcc=5 perl5=5.22 pgsql=9.5 php=5.6 python=2.7 python2=2.7 python3=3.5
WITH_BDB6_PERMITTED=yes

FORCE_MAKE_JOBS=yes
MAKE_JOBS_NUMBER=4

.if defined(WITH_GCC)
CC=gcc52
CXX=g++52
CPP=cpp52
.endif

WITH_CCACHE_BUILD=yes

.if (!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*))
.if !defined(NOCCACHE) && exists(/usr/local/libexec/ccache/world/cc)
CC:=${CC:C,^cc,/usr/local/libexec/ccache/world/cc,1}
CXX:=${CXX:C,^c\+\+,/usr/local/libexec/ccache/world/c++,1}
.endif
.endif
```

I'm using 10.3-PRERELEASE amd64 dated from Jan 30th taken from the stable/10 branch. I'll try what you suggested in the post above, but it will take me a few hours to reinstall gcc6-aux again. Unfortunately it bootstraps itself so isn't cached with ccache. It's only an Intel Atom 1.8GHz so isn't massively powerful and for some reason if I try and install it using pkg it wants to install postgresql 9.4 despite the fact I have 9.5 already so that would break and I have no idea what gcc6-aux is linked to postgresql in the first place.


----------



## marino (Feb 10, 2016)

/etc/make.conf is not read at all.

gcc6-aux is not connected to pgsql in any way.  Check again.


----------



## xtaz (Feb 10, 2016)

```
]# pkg install gcc6-aux
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Updating database digests format: 100%
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  gcc6-aux: 20160124 [FreeBSD]
  postgresql94-client: 9.4.5_1 [FreeBSD]

The process will require 261 MiB more space.
53 MiB to be downloaded.

Proceed with this action? [y/N]:
```

Go figure? Also in steps 1 and 6 above, did you mean /synth rather than /pkg ?


----------



## marino (Feb 10, 2016)

I can't explain why pkg wants to upgrade pgsql but I swear to you it's not related to gcc6-aux. 

yes, 1 and 6 should be synth.  I will edit.


----------



## xtaz (Feb 10, 2016)

OK. This time I'll leave gcc6-aux installed! However I can't get it to compile. Pasting exactly what you said with the " marks around it gives this:


```
ada -c -g -O2 -gnat12 -gnatyaAbBcdefhiklnOprsSxt -I- -gnatA /usr/obj/usr/ports/ports-mgmt/synth/work/synth-4417017/src/portscan.adb
portscan.adb:562:06: statement expected
gnatmake: "/usr/obj/usr/ports/ports-mgmt/synth/work/synth-4417017/src/portscan.adb" compilation error
*** Error code 4
```
If I remove the quotes then it gives this:


```
ada -c -g -O2 -gnat12 -gnatyaAbBcdefhiklnOprsSxt -I- -gnatA /usr/obj/usr/ports/ports-mgmt/synth/work/synth-4417017/src/portscan.adb
portscan.adb:562:10: no candidate interpretations match the actuals:
portscan.adb:562:10: missing argument for parameter "Item" in call to "PUT_LINE" declared at a-textio.ads:259
portscan.adb:562:21: expected type "Standard.String"
portscan.adb:562:21: found private type "Ada.Strings.Unbounded.Unbounded_String"
portscan.adb:562:21:  ==> in call to "Put_Line" at a-textio.ads:263
gnatmake: "/usr/obj/usr/ports/ports-mgmt/synth/work/synth-4417017/src/portscan.adb" compilation error
*** Error code 4
```
You'll have to excuse me in that I have never seen ada before so don't know if the quotes were you quoting what needs to be added or part of the code!


----------



## marino (Feb 10, 2016)

it's my fault, I didn't test that line.
It should be 
	
	



```
TIO.Put_Line (JT.USS (content));
```
I think.  (I didn't test this either.  )


----------



## xtaz (Feb 10, 2016)

Getting somewhere now.


```
# synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
1.6.3
pkg-1.6.3.txz
(huge amount of whitespace here)
Encountered issue with ports-mgmt/pkg or its dependencies
  => bad input for 'Value: ""
Unexpected pkg(8) scan failure!
Unfortunately, the system upgrade failed.
```


----------



## marino (Feb 10, 2016)

Strange.  There's no problem where you modified the file.  That's the expected output.  Let me think about this a minute.


----------



## marino (Feb 10, 2016)

xtaz: is this a uniprocessor machine?


----------



## xtaz (Feb 10, 2016)

```
CPU: Intel(R) Atom(TM) CPU D525  @ 1.80GHz (1795.74-MHz K8-class CPU)
avail memory = 4103008256 (3912 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
```


----------



## marino (Feb 10, 2016)

xtaz you should have had a third line of output with "4" indicating 4 cpus.  It's not there.  My guess is that it's a parsing error because it's not expecting a blank value.

e.g.
`make -C /usr/ports/ports-mgmt/pkg -VMAKE_JOBS_NUMBER`
should return "4", not nothing.


----------



## Crivens (Feb 10, 2016)

marino@ said:


> signal-handling: Use the "Escape" key.  I had to abandon SIGINT capture because it propagates to the builders and breaks when conftests segfaults.  You should almost never use control-C.  However, if you do, run any major synth command, like `synth configure` and it will clean up the dirty mounts automatically.


There may be a way around this. When you fork a new process, this is the child of a parent process. Signals may propagate to that one. Now, if you do not fork a builder directly but a trampoline process, which then in turn forks a builder, passes that builders PID to its parent (the main process) and then exits, your builder has no living parent process but may still be controlled by the main process via the PID. Could this solve the Ctrl-C problem?


----------



## xtaz (Feb 10, 2016)

marino@ said:


> xtaz@ you should have had a third line of output with "4" indicating 4 cpus.  It's not there.  My guess is that it's a parsing error because it's not expecting a blank value.
> 
> e.g.
> 
> ...



OK, so the problem *was* /etc/make.conf then. If you look at where I pasted that file in my previous post you can see I set these two variables:


```
FORCE_MAKE_JOBS=yes
MAKE_JOBS_NUMBER=4
```

Commenting those out solves the problem and synth is working now. I guess that is overriding something important then?

Although interestingly it shows 3, not 4:


```
Stand by, comparing installed packages against the ports tree.
1.6.3
pkg-1.6.3.txz
3
Stand by, building pkg(8) first ...
```


----------



## marino (Feb 10, 2016)

Crivens, unfortunately no.  I spent 2 solid days on it.  There's no workaround.  I could fork 3 generations, the parent always retains control of the signal.


----------



## marino (Feb 10, 2016)

xtaz if you're correct and /etc/make.conf is considered, that's a problem.  It might be a case where I intentionally scan the pkg(8) package (only) using the host ports tree and /etc/make.conf gets sucked in (and I forgot about consequences).  I'll have to think about this because that's two reports that stem from that decision.


----------



## marino (Feb 10, 2016)

xtaz by the way, you can just delete both lines because they don't do anything anyway.


----------



## marino (Feb 10, 2016)

xtaz

```
[ J ] Max. jobs per builder  3
```


----------



## marino (Feb 10, 2016)

xtaz I was able to reproduce and I've added a handler for this specific case:

https://github.com/jrmarino/synth/commit/dd8117fac2e1ba4aa714212ae2ee16447de72a84


----------



## Crivens (Feb 10, 2016)

marino@ said:


> Crivens, unfortunately no.  I spent 2 solid days on it.  There's no workaround.  I could fork 3 generations, the parent always retains control of the signal.


Maybe  setsid(2) is the key then? Place the builders in their own sessions?


----------



## marino (Feb 10, 2016)

xtaz there's only one place where host /etc/make.conf is read, and that is where the jail's makeconf is being created.  Some variables are cached so the tree scans faster, and by defining MAKE_JOBS_NUMBER it caused _SMP_CPUS variable to remain undefined, which eventually defined the make jobs.  Obscure, but something I need to handle.


----------



## marino (Feb 10, 2016)

Crivens so you don't believe that over the course of 40 straight hours, I didn't try everything?


----------



## xtaz (Feb 10, 2016)

Thanks for your help there marino@ much appreciated! I've been using FreeBSD for about 15 years, always with ports and world/kernel from source code. Packages and binary updates are new to me. It's clear that this is the way the project is going which is why I'm taking a good look at this nice looking software you've written as it looks like it could be good for transitioning. As a result I'm now looking into cleaning up my /etc/make.conf and switching to a GENERIC kernel with loaded kernel modules etc. so that I can try to embrace this new world. Starting with removing those lines from that file!


----------



## marino (Feb 10, 2016)

The MAKE_JOBS_NUMBER line is legitimate (and thus causing me to change synth) but in your case you don't need it because you are defining it to the default value of 4.
But this case revealed an issue so thanks for helping me discover and diagnose it.


----------



## marino (Feb 10, 2016)

xtaz to close this issue, here's the final fix:
https://github.com/jrmarino/synth/commit/e175360547261d706a64dc06056c68a932f46157

It will be pushed with the upcoming version 1.00.


----------



## Crivens (Feb 10, 2016)

marino@ said:


> Crivens@ so you don't believe that over the course of 40 straight hours, I didn't try everything?


When you work 40 hours in two days, I would belive that you tried everything coming to your mind - which will politely avoid to speak about the state of mind one is in after such a ride . Been there, done that, got the eyesore and when coming back after some days, wondered  how in hades I had missed that obvious thing to do.  No offense, I just stumbled over that.


----------



## xtaz (Feb 10, 2016)

I've just had a failure when building one of the packages. I can see from previous posts in this forum that you've been gradually fixing various ports which fail so I guess you are probably interested in this.

lang/python27 seems to do this:


```
========================< phase : package  >========================
===>  Building package for python27-2.7.11_1
pkg-static: Unable to access file /construction/xports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/_ssl.so: No such file or directory
*** Error code 1

Stop.
make: stopped in /xports/lang/python27
===========================================================================
```

The /xports looks weird as that directory doesn't exist. My build directory is /usr/obj/synth-live


----------



## marino (Feb 10, 2016)

xtaz said:


> I've just had a failure when building one of the packages. I can see from previous posts in this forum that you've been gradually fixing various ports which fail so I guess you are probably interested in this.



I can't reproduce, not even with synth test command.  Obviously lang/python27 is a popular port so if it didn't build we would have heard about it by now.  Do you have a LiveSystem-make.conf defined?  If so, what's in it?


----------



## xtaz (Feb 10, 2016)

It's definitely going to be something I've overridden again isn't it!


```
$ cat LiveSystem-make.conf
DEFAULT_VERSIONS=apache=2.4 gcc=5 perl5=5.22 pgsql=9.5 php=5.6 python=2.7 python2=2.7 python3=3.5
OPTIONS_UNSET+=X11
WITH_BDB6_PERMITTED=yes
WITH_OPENSSL_PORT=yes
```


----------



## marino (Feb 10, 2016)

The last line of LiveSystem-make.conf looks suspicious.


----------



## xtaz (Feb 10, 2016)

I'll let it finish the remaining package build and then I'll experiment with that file to see if I can get it to work. That option looks like a valid thing to use though from /usr/ports/Mk/bsd.openssl.mk. "WITH_OPENSSL_PORT=yes - Use the OpenSSL port, even if base is up to date"


----------



## marino (Feb 10, 2016)

It might very well be valid, but that doesn't mean there's not a bug in lang/python27.  It's probably worth opening a PR against it.  You can modify pkg-plist locally but it should be fixed everywhere.

My feeling is that there is a good chance that lang/python27 doesn't account for this and it needs to.


----------



## protocelt (Feb 10, 2016)

I'm not sure WITH_OPENSSL_PORT=yes is an issue. If it is, it's at least not universal as I use it for both 10-STABLE and 11-CURRENT with Synth without problems.


----------



## kpa (Feb 10, 2016)

It hasn't been a problem for me, lang/python27 has built fine with both Synth and Poudriere with that setting.


----------



## xtaz (Feb 10, 2016)

Put it this way, all of those entries in that file are identical to ones that I have been using for years in /etc/make.conf and lang/python27 has never had that issue with a standard port compile. But I will experiment with it later and try different things (including deleting the file) to see if I can get it to work. So watch this space.

Saying that... I've never tried to `make package` before.


----------



## marino (Feb 10, 2016)

I think synth has proven many times to detect errors that have gone unnoticed for a long time.


----------



## marino (Feb 10, 2016)

That being said, I just tried it and like kpa says, python27 builds fine with ports openssl.  It's something else.


----------



## Wapcaplet (Feb 10, 2016)

marino@ said:


> xtaz@ there's only one place where host /etc/make.conf is read, and that is where the jail's makeconf is being created.  Some variables are cached so the tree scans faster, and by defining MAKE_JOBS_NUMBER it caused _SMP_CPUS variable to remain undefined, which eventually defined the make jobs.  Obscure, but something I need to handle.


Just wanted to let you know this solved my issue too -- I commented out MAKE_JOBS_NUMBER in my /etc/make.conf, and synth now works.  Thanks!


----------



## marino (Feb 10, 2016)

Wapcaplet, great!  When 1.00 is released you can re-enable that setting.  I've fixed the problem permanently.


----------



## kpa (Feb 10, 2016)

xtaz said:


> Saying that... I've never tried to `make package` before.



Doing `make package` and `make install` on a port are almost identical. Both will check the pkg-plist for omitted entries after the build has finished and do other consistency checks. The only difference is that in the former the result is a package file made from the contents of the stage directory, in the latter the files are installed on the system from the stage directory and registered as an installed package.


----------



## Wapcaplet (Feb 10, 2016)

I just finished my first run of `synth upgrade-system`, and it looks like the line WITH_OPENSSL_PORT=yes inside my /etc/make.conf was ignored.

After `synth upgrade-system` completed, I ran `pkg autoremove`, and security/openssl was offered for deletion.  After deleting it, I went back and checked on my installed packages, and none of them depended on security/openssl anymore.

Does WITH_OPENSSL_PORT=yes need to be explicitly included in /usr/local/etc/synth/LiveSystem-make.conf for this to work?  Right now I have no such file.  As I understand it, <profile>-make.conf should be appended to /etc/make.conf and not replace it.


----------



## marino (Feb 10, 2016)

Yes.  /etc/make.conf is not used by synth.  Check the man page, FILES section, second entry, <profile>-make.conf


----------



## Wapcaplet (Feb 10, 2016)

marino@ said:


> yes.  /etc/make.conf is not used by synth.  Check the man page, FILES section, second entry, <profile>-make.conf


After rereading the man page, as I understand it, it looks like /etc/make.conf is pulled in only if <profile>-make.conf exists:


```
This is an optional, user-provided file. If it exists, the builder's [file]/etc/make.conf[/file] will be appended with the contents of this file.
```
So /etc/make.conf is not included at all, even if <profile>-make.conf exists?  I guess the word "appended" is confusing me.


----------



## marino (Feb 10, 2016)

Look at any build log, you'll see a make.conf.  It's not yours, it's a standard one.   <profile>-make.conf is appended to that.  The host /etc/make.conf is not used at all, under any circumstance.


----------



## kpa (Feb 10, 2016)

Yes, the host's /etc/make.conf is not used in any way. This is intentional I assume because it often contains settings that can mess up the builder badly. Other package builders operate with the same principle and only allow make.conf(5) customization trough the dedicated customization files.


----------



## mirco (Feb 10, 2016)

I get the following error message. What to do?


```
# synth status
Builder mounts detected; attempting to remove them automatically ...
Dismounting successful!
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.

raised CONSTRAINT_ERROR : bad input for 'Value: ""
```


----------



## Wapcaplet (Feb 10, 2016)

OK, I get it.  The builder's /etc/make.conf is being referenced in the man page, not the system's /etc/make.conf.  So I just created LiveSystem-make.conf with the contents of the system's /etc/make.conf and reran synth upgrade-system, and it looks like it's including the lines I copied from the system's /etc/make.conf as the packages are rebuilt.  Thanks!


----------



## marino (Feb 10, 2016)

mirco  do you have MAKE_JOBS_NUMBER defined in /etc/make.conf?  If so, comment it out.  It's messing synth up (will be fixed in next release)


----------



## marino (Feb 10, 2016)

Wapcaplet said:


> OK, I get it.  The builder's /etc/make.conf is being referenced in the man page, not the system's /etc/make.conf.  So I just created LiveSystem-make.conf with the contents of the system's /etc/make.conf and reran synth upgrade-system, and it looks like it's including the lines I copied from the system's /etc/make.conf as the packages are rebuilt.  Thanks!



Careful.  We don't use /etc/make.conf because of its contents.  If you indiscriminately dump /etc/make.conf into the customized make.conf you could be doing yourself a big disservice.  If you want, paste it here so we can check you aren't doing something wrong.

In other words, you should be able to justify each and every line in the custom make.conf.  When in doubt, remove it.


----------



## Wapcaplet (Feb 10, 2016)

I think it should be OK.  Here's what the file contains:


```
CPUTYPE?=core2
WITH_OPENSSL_PORT=yes
OPTIONS_UNSET= X11 NLS
```


----------



## marino (Feb 10, 2016)

Yeah, that's pretty innocent (CPUTYPE scares me but I'm finding people like it).


----------



## mirco (Feb 10, 2016)

marino@ said:


> mirco  do you have MAKE_JOBS_NUMBER defined in /etc/make.conf?  If so, comment it out.  It's messing synth up (will be fixed in next release)


Yes, that was it. Thank you!


----------



## marino (Feb 10, 2016)

FYI, you don't need "?=" operator on CPUTYPE.  Use "=".  Nothing else will override it.


----------



## mirco (Feb 10, 2016)

Previously I  deinstalled audio/pulsaudio and set /etc/make.conf to OPTIONS_UNSET+=PULSEAUDIO.

But `synth status` tells, that I _will_ build audio/pulsaudio.

How's that  ?


----------



## marino (Feb 10, 2016)

i guess there's a port in there that requires pulseaudio unconditionally.


----------



## marino (Feb 10, 2016)

And read previous comments from today: /etc/make.conf *IS NOT USED.*


----------



## Wapcaplet (Feb 10, 2016)

I'm now running `synth upgrade-system` on my other server, which has comms/hylafax on it.  The compilation of that port is interactive -- you have to make selections during the build process.  How does `synth` handle those kinds of ports?

I guess I'll find out in about 30 minutes or so...


----------



## marino (Feb 10, 2016)

There are no more interactive ports in the entire tree.  The answer is no, synth (and poudriere) only build non-interactive ports.


----------



## xtaz (Feb 10, 2016)

Hi again. I've found the problem that I had with lang/python27. I noticed that it also failed on mail/spamassassin except that complained about missing SSLv3 symbols before crashing out. This made me remember that I had switched off the options for SSLv2/SSLv3 support in security/openssl. As you had pointed out the line about openssl as being a potential issue this made me think maybe I should try and switch those options back on. Guess what? Now both python and spamassassin compiled without any issue.

I have one further question though. I have the option turned on which should fetch packages from the external repo if they haven't been changed. This doesn't appear to have worked for anything so far. I'm wondering is that maybe because I have /var/db/ports entries for *every* port, even if I haven't actually changed anything from the defaults, because that's just how the port/portmaster behaves. I'm wondering if it might work better if I ran `make rmconfig` on a few of them?


----------



## Wapcaplet (Feb 10, 2016)

marino@ said:


> There are no more interactive ports in the entire tree.  The answer is no, synth (and poudriere) only build non-interactive ports.


Yes, you're right.  The log shows no interactive prompts.  That goes to show how long it's been since I had to rebuild that port!


----------



## marino (Feb 10, 2016)

It's my opinion that you should only have options set on ports that differ from the defaults.
That being said, it's hard to being that no ports are suitable for pre-fetching.
Maybe your external report (freeBSD repository) is set to quarterly packages?


----------



## marino (Feb 10, 2016)

Wapcaplet said:


> Yes, you're right.  The log shows no interactive prompts.  That goes to show how long it's been since I had to rebuild that port!


It's both.  If it's inside a builder, it builds in batch mode.  If you build it live with make, it's interactive (according to commit log)


----------



## PacketMan (Feb 11, 2016)

marino@ said:


> That being said, it's hard to being that no ports are suitable for pre-fetching.
> Maybe your external report (freeBSD repository) is set to quarterly packages?



The jury is still out, but to me its looking like very little fetching of pre-built packages is happening.  I'll know better in a week or two. How do I determine again what my machines are set to use?  I have not changed any defaults. Stock FreeBSD 10.2-RELEASE and stock synth.


----------



## marino (Feb 11, 2016)

You know FreeBSD 10.2 STOCK defaults to quarterly branch right?
Check `pkg -vv` and see the repository url.  If it says "quarterly" then I guess you can expect very little fetching.
(ports are equivalent to "latest")


----------



## PacketMan (Feb 11, 2016)

marino@ said:


> You know FreeBSD 10.2 STOCK defaults to quarterly branch right?



Yeah I didn't know that. Still learning. I'm a internetworking routing & switching guy, not a full time server guy.  

So yeah its set to quarterly. I thought synth kinda override that and make it own decisions, but I know yer gonna tell me not now.  So should I change that to some sort of 'latest' branch value? Do I even want to? Should I leave well enough alone?


----------



## marino (Feb 11, 2016)

If you leave it at quarterly, then you might as well turn off prefetching option.
For some reason, FreeBSD 10.2 is set to quarterly and the previous FreeBSD releases are set to latest.

Synth will not "override" pkg(8).  It would have to invasively rewrite a non-synth pkg(8) conf file and it philosophically will not (and should not) do that.

What it perhaps could do is warn the user if quarterly is detected though (in the URL), but see below, quarterly could be valid.

Summary: your FreeBSD repository should match your ports!
If your ports tree is on "latest" (e.g. portsnap) then you need FreeBSD repo to be on latest.
If your ports tree is on quarterly (e.g. svn branch) then you need FreeBSD repo to be on quarterly


----------



## marino (Feb 11, 2016)

As I can see this being a common problem (thanks FreeBSD 10.2), I added this to the FAQ:
https://github.com/jrmarino/synth/b...-very-few-are-actually-retrieved--whats-wrong


----------



## xtaz (Feb 11, 2016)

I've gone through /var/db/ports and removed everything except the bare minimum now for which I want to keep custom options. I've deleted the whole repository and let it rebuild it again and still as far as I can tell it has compiled its own custom packages for everything rather than just pre-fetch ones from the FreeBSD repo. I am using the latest repo rather than quarterly because I am using the stable branch rather than a release branch and the defaults are different for stable branches. So my repo URL is "pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest". Oh well, it's not a huge problem, just I'm curious. Unless maybe amd64 packages are built less often than i386 ones?

Other than that, I am now succesfully using synth and it works very well. So I would just like to say thank you for such a great tool.


----------



## Crivens (Feb 11, 2016)

Could someone please testbuild deskutils/calibre and check if it waits for godot in staging?

The "build deinstall reinstall clean" takes only a minute or so when done using make.


----------



## marino (Feb 11, 2016)

xtaz: new packages are built weekly for both amd64 and i386, so that's not the issue.
It might help to turn off ncurses mode and capture the text when it tries to connect to external repository.  confirm that it's doing that successfully first.

from what you have describe, it should be prefetching.


----------



## talsamon (Feb 11, 2016)

> Could someone please testbuild deskutils/calibre


Crivens confirm, hangs on

```
Installing bash completion to: /ram/usr/ports/deskutils/calibre/work/stage/usr/local/share/bash-completion/completions/calibre
Setting up desktop integration...
```

but hangs also in the port on my system - maybe, it is something with `ccache`?


----------



## xtaz (Feb 11, 2016)

Just tried that. It doesn't log anything at all to suggest it is even trying to?


```
Stand by, building pkg(8) first ... done!
Stand by, updating external repository catalogs ... done.
pkg: Repository Synth missing. 'pkg update' requiredpkg: Repository Synth missing. 'pkg update' required

pkg: Repository Synth missing. 'pkg update' required
pkg: Repository Synth missing. 'pkg update' required
etc etc etc....
Scanning existing packages.
00:00:04 => [03] Builder launched
00:00:04 => [02] Builder launched
00:00:04 => [01] Builder launched
00:00:14 => [02] 00:00:06 Success devel/autoconf-wrapper
00:00:14 => [03] 00:00:06 Success devel/automake-wrapper
00:00:25 => [01] 00:00:17 Success devel/ccache
00:00:26 => [03] 00:00:09 Success misc/mime-support
00:00:35 => [03] 00:00:07 Success print/indexinfo
00:00:56 => [03] 00:00:17 Success textproc/expat2
00:01:04 => [02] 00:00:47 Success devel/cmake-modules
```

And so on..... Would you expect to see something else there then?


----------



## xtaz (Feb 11, 2016)

Ahhhh I see the problem. If I delete /usr/local/etc/pkg/repos/00_synth.conf then it fetches them all before launching builders. So it appears to be looking at itself first, and finding nothing, because that errors because the repo hasn't been built yet. Chicken and Egg situation?

Both repos appear to have priority 0.


----------



## marcpearsonti (Feb 11, 2016)

Hello marino@

Excuse my English, I'm a French speaker.

First, I would like to thank you for Synth, is a great tool

But I have so many questions ...

1. How Synth scan the port tree ? I get the port with SVN from 2016Q1 branch. But it's missing ports-mgmt/synth in it. So I put Synth manually in the port tree with a specific checkout from the head but I got the error when I execute `synth status`: 
	
	



```
Scan of ports-mgmt/synth failed, it will not be considered.
```

2. Why Synth need to build so many ports when a package like ftp/curl change. I have used portmaster(8) since many years and it doesn't do this kind of rebuild. My FreeBSD server never broke. Maybe the way Synth works is the good one but I would like to understand. Maybe I was lucky with portmaster(8) 

3. `synth status` shows that 3 ports need to be rebuilt but after the build/install process is complete, `synth status` show the same 3 ports with failed dependency check. How can I debug this situation ?

4. When Synth rebuilds the package repository, does it need to repackage all of them or only the ones who changed?  On my old FreeBSD server this operation take approx. 30 minutes for 312 packages

5. Is the message "<portname> failed dependency check" how Synth report that a package needs to be rebuilt or is it an error message?


I would like to make suggestions for improvements:

1. It's possible when running `synth status` to show from old version to new version each ports will be upgrade. Example: databases/rrdtool (4.8.9_1 to 5.0.1)
2. If "failed dependency check" is a normal message, is't a little bit scary. You should use a message like "<portname> is no more in sync with dependencies" (You are better in english then me!)
3. Could Synth keep old versions of a package (Configure could have a yes/no option and how many old versions to keep ). So if a user need to rollback a package it will be very easy.


----------



## marino (Feb 11, 2016)

xtaz said:


> hem all before launching builders. So it appears to be looking at itself first, and finding nothing, because that errors because the repo hasn'





xtaz said:


> Ahhhh I see the problem. If I delete /usr/local/etc/pkg/repos/00_synth.conf then it fetches them all before launching builders. So it appears to be looking at itself first, and finding nothing, because that errors because the repo hasn't been built yet. Chicken and Egg situation?
> 
> Both repos appear to have priority 0.



Don't know.  IIRC it was told to only look at the external repo (which appears to update successfully).
It's not looking for itself, but that strange error might be preventing it from working.


----------



## xtaz (Feb 11, 2016)

The strange errors are because the 00_synth.conf file is being read but because it is synths very first build run none of the metadata files exist inside the repo directory so it's not (yet) a valid repo to use. I guess in that situation it isn't falling back to using the FreeBSD repo?


----------



## marino (Feb 11, 2016)

marcpearsonti said:


> But I have so many questions ...



It's probably better if you asking them one at a time, this is a bit much. 



> 1. How synth scan the port tree ? I get the port with SVN from 2016Q1 branch. But it's missing ports-mgmt/synth in it. So i put synth manually in the port tree with a specific checkout from the head but i got the error when i execute synth status :_ Scan of ports-mgmt/synth failed, it will not be considered._


_
You've taken a known good tree and added something to it, and now you got an error. You should probably expect that when you transplant ports from one tree to another .  (Don't do it).  I could guess it failed because gcc6 isn't in the Quarterly tree or something.  Don't corrupt trees anymore is my advice.
_


> 2. Why synth need to build so many ports when a package like ftp/curl change. I have used portmaster since many years and it doesn't do this kind of rebuild. My FreeBSD server never broke. Maybe the way synth works is the good one but il would like to understand. Maybe i was lucky with portmaster



Yes, you were lucky.  (And I kind of doubt you never had a problem).  hundreds of packages depend on Curl so you should expect major fallout.  This very port has been discussed earlier in the thread.



> 3. `synth status` shows that 3 ports need to be rebuild but after the build/install process is complete, `synth status` shows the same 3 ports with failed dependency check. How can I debug this situation ?



Define WHYFAIL in the environment with synth(1) in non-curses mode.  It will tell you exactly why.  It could be a bug in the port.



> 4. When synth rebuild the package repository, does it need to repackage all of them or only the ones who changed?  On my old FreeBSD server this operation take approx. 30 minutes for 312 packages


This question doesn't make sense.  "pkg repo" packages *ALL* of them, by definition.  You can't do it any other way.



> 5. Does the message "<portname> failed dependency check" is how synth report that a package need to be rebuild or is a error message ?


it will delete the package (unless it's a status command).
This isn't an "error" message, it's normal.



> I would like to make suggestions for improvements
> 1. It's possible when running `synth status` to show from old version to new version each ports will be upgrade. Example: databases/rrdtool (4.8.9_1 to 5.0.1)


No, it's not possible (nor do I see the value)



> 2. If "failed dependency check" is a normal message, is't a little bit scary. You should use a message like "<portname> is no more in sync with dependencies" (You are better in English then me !)


Why is that scary?  It's normal.



> 3. Could synth keep old versions of a package (Configure could have a yes/no option and how many old versions to keep ). So if a user need to rollback a package it will be very easy.


This also makes no technical sense, it's not possible.
You're creating a local repository, everything is integral, you can't just roll some random port back.
Let's put this another way: Have you seen any tool that does that?


----------



## marino (Feb 11, 2016)

xtaz said:


> The strange errors are because the 00_synth.conf file is being read but because it is synths very first build run none of the metadata files exist inside the repo directory so it's not (yet) a valid repo to use. I guess in that situation it isn't falling back to using the FreeBSD repo?



It's not there before "first run", it gets installed on the first install.  But the question is why is it being read.  Synth tells it to read external repository (meaning don't read anything else).   I have to research what could be happening.


----------



## marcpearsonti (Feb 11, 2016)

Poudriere can keep old versions of package. Take a look at the https://github.com/freebsd/poudriere/blob/master/src/etc/poudriere.conf.sample


```
# Keep older package repositories. This can be used to rollback a system
# or to bisect issues by changing the repository to one of the older
# versions and reinstalling everything with `pkg upgrade -f`
# ATOMIC_PACKAGE_REPOSITORY is required for this.
# Default: no
#KEEP_OLD_PACKAGES=no
# How many old package repositories to keep with KEEP_OLD_PACKAGES
# Default: 5
#KEEP_OLD_PACKAGES_COUNT=5
```


----------



## marcpearsonti (Feb 11, 2016)

If Portmaster can show the versions when showing the "status", Synth should be able to too . Each ports has a version and revision in the Makefile. For the current installed ports, pkg(8) version could be great ! But I'm not a pro !


----------



## marcpearsonti (Feb 11, 2016)

Why repackage ALL ports/packages if only some ports (including dependencies of course !) has change. All the others unaltered ports are the same. Does synth skip all of these ?


----------



## marino (Feb 11, 2016)

marcpearsonti said:


> Poudriere can keep old versions of package. Take a look at the https://github.com/freebsd/poudriere/blob/master/src/etc/poudriere.conf.sample



You are not understanding what this does.

It's ALL or NOTHING.  You can't roll back "one" package.

Poudriere is making several copies of repositories.  If you wanted to roll one thing back, you'd have to roll everything back.

You can do this yourself with net/rsync/sysutils/cpdup.  You'd just duplicate the entire repository directory and save it somewhere.


----------



## marino (Feb 11, 2016)

marcpearsonti said:


> If portmaster can show the versions when showing the "status", Synth should be able too . Each ports has a version and revision in the Makefile. For the current installed ports, pkg version could be great ! But i'm not a pro !



And what would you do with that information?


----------



## marino (Feb 11, 2016)

marcpearsonti said:


> Why repackage ALL ports/packages if only some ports (including dependencies of course !) has change. All the others unaltered ports are the same. Does synth skip all of these ?


You are using the wrong terminology.
You asked about packages in a repository.  If you change one package and then rebuild the repository, every single package is scanned again and put in the digest.  It's the same work for 1 package or 1000 packages.

But now you use words like "packaging" and I don't know what you mean.

Are you talking about packages or repositories?  be specific.


----------



## kpa (Feb 11, 2016)

marcpearsonti said:


> Why repackage ALL ports/packages if only some ports (including dependencies of course !) has change. All the others unaltered ports are the same. Does synth skip all of these ?



This has been beaten to death already many times. There is no way to avoid the recursive pruning and rebuilding of dependent ports even if the end result of the build causes no apparent changes. FreeBSD ports and the package building tools have choice in the matter, it's not a design "feature" but an absolute necessity to make sure all of the potential changes are propagated properly where they have to be propagated. FYI Poudriere does the exact same thing when there are updates to ports.


----------



## xtaz (Feb 11, 2016)

The "problem" with people like myself who were used to just using ports-mgt/portmaster is that we only ever recompiled the exact ports which had changed and never had a need to redo the dependancies. We trusted that the ABI had not changed at all. The exception is when an entry appeared in /usr/ports/UPDATING telling us that we had to recompile all dependancies, usually when a shared library had its version bumped etc. In general this worked fine and never really caused any issues. But obviously when it comes to a repository you want it to be completely consistent and there's no shortcuts to that. People just need to realise that. One thing I will say is try and do what I have done. Enable package pre-fetching, try and prune the list of ports that have custom options to the bare minimum in /var/db/ports, and properly configure devel/ccache so that it caches object files. These things will greatly speed up the rebuild process.


----------



## chrbr (Feb 11, 2016)

kpa said:


> FreeBSD ports and the package building tools have choice in the matter, it's not a design "feature" but an absolute necessity to make sure all of the potential changes are propagated properly where they have to be propagated.


I have seen that one time since I use devel/poudriere which is just a few weeks. There was a message about an ABI change, too. Therefore it is good to know that the tools can detect this kind of stuff. Otherwise things can creep in and strange effects appear later. I do not understand why it is necessary to rebuild ports to find that out. But I accept that. If rebuild would not be required I am sure that better methods would have been applied in the tool chains by people with higher knowledge than mine.


----------



## garry (Feb 11, 2016)

marino@ said:


> xtaz@ if you're correct and /etc/make.conf is considered, that's a problem.  It might be a case where I intentionally scan the pkg(8) package (only) using the host ports tree and /etc/make.conf gets sucked in (and I forgot about consequences).  I'll have to think about this because that's two reports that stem from that decision.



Three reports actually.  That empty string error almost stopped me from continuing when I first started using synth; I fixed it by...

For the life of me I can't remember what I did.  If I can retrieve any info I'll add it later.  I *am* an ada programmer! <wink>  I wrote "hello world" in ada. I've been reading Feldman and Koffman's "Ada 95: Problem Solving and Program Design".

By the way I hope marino that you can at some point find time to consider the problem with lang/ghc -- Synth always rebuilds it and all of its dependents.  After completing a `synth just-build ghc` and friends look fine but `synth rebuild-repository` claims "missing dependencies" and throws the freshly built packages away.  I've edited the lang/ghc Makefile to add BOOT and BOOTH as normal options along with all other options (rather than options only if bin/ghc and bin/HsColour exist).  (and I keep those options turned off because bin/ghc and bin/HsColour will not exist in the chroot(8)).  That gets around the problem of Synth complaining always that the options are out of date.  My options on lang/ghc are

```
Options  :
  BCLANG  : on
  BOOT : off
  BOOTH : off
  DOCS  : on
  DYNAMIC  : on
  GCC  : off
  LLVM  : on
  PCLANG  : off
  PROFILE  : on
```
and it's dependencies are:

```
> pkg info -d ghc
ghc-7.10.2:
   libiconv-1.14_9
   llvm35-3.5.2_1
   gmp-5.1.3_2
   libffi-3.2.1
```

Synth builds all 2800 of the ports needed in my repository _except_ lang/ghc.


----------



## marino (Feb 11, 2016)

garry, if you want to build overnight, this could help:

Turn off ncurses mode from configure
Exec command `env WHYFAIL=1 synth just-build lang/ghc |& tee /tmp/ghc.log` (or use script or adjust for whatever shell you have)

If it really does think GHC is always bad, at least we'll know which dependency it doesn't like.


----------



## fernandel (Feb 11, 2016)

hukadan said:


> On my system without the option :
> 
> ```
> ====>> Ignoring sysutils/fusefs-exfat: License Microsoft-exFAT needs confirmation, but BATCH is defined
> ...



Thank you very much. Now I have two line in /usr/local/etc/synth/LiveSystem-make.conf:

```
DEFAULT_VERSIONS+= perl5=5.20 
DISABLE_LICENSES=yes
```
and it works .


----------



## marcpearsonti (Feb 11, 2016)

marino@ said:


> And what would you do with that information?



When managing a production server it's important to know the versions that will be installed. With the exact version change, I can read the changelog (BSD or official site) or refuse to upgrade the ports or prepare some configuration files.


----------



## marcpearsonti (Feb 11, 2016)

Running synth with WHYFAIL I see these errors about my stalled packages:

```
rrdtool-1.4.8_9.txz DOCS is OFF but port says it must be ON
cherokee-1.2.104.txz DOCS is ON but port says it must be OFF
databases/rrdtool package has more dependencies than the port requires (8)
```
I try to delete the ports options with `make rmconfig` but same errors.


----------



## marino (Feb 11, 2016)

@marcpearsonti, quarterly or latest?  I think databases/rrdtool was already fixed in latest.


----------



## marcpearsonti (Feb 11, 2016)

marino@ said:


> marcpearsonti@ quarterly or latest?  I think rrdtool was already fixed in latest.



Quarterly ports from 2016Q1 branch


----------



## marino (Feb 11, 2016)

marcpearsonti said:


> When managing a production server it's important to know the versions that will be installed. With the exact version change, i can read the changelog (BSD or official site) or refuse to upgrade the ports or prepare some configuration files



You've got a list of changed ports and you've got the port itself (which has the version number).  But I'll think about it.


----------



## marcpearsonti (Feb 11, 2016)

marino@ said:


> You've got a list of changed ports and you've got the port itself (which has the version number).  But I'll think about it.



I know ... but i need to take a look to each Makefile to see the version/revision. Thanks to think about it !


----------



## marcpearsonti (Feb 11, 2016)

marino@, is it really better to use the quarterly branch. I use this because i was thinking that ports receive only security updates but after looking to the svnweb site some ports has been upgraded too. So what's the real reason FreeBSD has created these ports branches ?


----------



## Wapcaplet (Feb 12, 2016)

Is there a way to delete a package file from the repository directory (other than `rm <package>.txz`)?

My repository has packages for MySQL client & server for both v5.5 and v5.6 -- one of my ports had been compiled against 5.5, and Synth successfully rebuilt it against 5.6, but 5.5 was still built as part of the initial population of the repository.  So nothing uses 5.5, and I don't see a need to keep it in the repository (especially since the default in ports is 5.6 anyway).

I can imagine there may be other package files in the repository directory that will be "orphaned" in the future if they are no longer a build or run dependency following some future upgrade that changes the dependent port's dependency lists.

The only sort-of-analogous command I found in Poudriere would be `poudriere pkgclean`, but that allows you to list package files you want to keep, instead of files you want to delete.


----------



## marino (Feb 12, 2016)

That's the only way, but it doesn't really matter if you have extra packages in a repo that aren't installed on the system.


----------



## xtaz (Feb 12, 2016)

I've found an odd one this morning. I have two ports which have a local patch file in the files/ directory that I have written. If I install the port manually then the patch is applied and works. If I look at the package that synth made then the patch has not been applied. Not sure what could cause this?


----------



## marino (Feb 12, 2016)

I don't see how it's possible.  The ports directory is mounted read-only.  The patch would be available to the port.
You can use `synth test` with the ENTERAFTER env var (see man page) to jump into the build and verify for sure whether or not the patch was applied.


----------



## xtaz (Feb 12, 2016)

Yep. Ignore me. I've obviously done something silly. The patch is in fact *not* being applied if I install the port manually. It was working last week so I have no idea why it's not working now as the port hasn't been updated at all. I'll investigate.


----------



## garry (Feb 12, 2016)

marino@ said:


> garry, this could help:
> 1) turn off ncurses mode from configure
> 2) exec command `env WHYFAIL=1 synth .....`
> 
> If it really does think GHC is always bad, at least we'll know which dependency it doesn't like.



I built an up-to-date repository with Synth, built lang/ghc, verified that the package was in the packages directory, and then tried to rebuild the repository with Synth.  It detected a "missing dependency" in lang/ghc and removed the "bad" package.  The result is that lang/ghc can not be built and added to the repository.  The details are mystifying:

```
synth prepare-system
env WHYFAIL=1 synth rebuild-repository
```
The result is that lang/ghc fails the dependency check:

```
Scanning entire ports tree.
lang/ghc package has less dependencies than the port requires (5)
Query: converters/libiconv:libiconv-1.14_9
devel/llvm35:llvm35-3.5.2_1
math/gmp:gmp-5.1.3_3
devel/libffi:libffi-3.2.1
                                                                                    <<<blank line?>>>
ghc-7.10.2.txz failed dependency check.
Local repository successfully rebuilt
```
But libiconv-1.14_9, llvm35-3.5.2_1, gmp-5.1.3_3, and libffi-3.2.1 are right there in the repository.  No dependency appears to be missing.  I took a look at the +COMPACT-MANIFEST for the GHC package that Synth built (before Synth threw it away) and didn't see anything of note.  Is the blank line in the output significant?  Could that be a "dependency" on non-printing chars?  There are not 5 dependencies listed. 

I'm at a loss.  Please let me know how further to debug this.


----------



## marino (Feb 12, 2016)

You are not interpreting the error message correctly.
The port Makefile says there are 5 dependencies, but the package only lists 4 dependencies.

This is almost always a problem in the port Makefile.


----------



## marino (Feb 12, 2016)

It would help to see the portion of the Synth log that says what options were set before the build.


----------



## xtaz (Feb 12, 2016)

Do you know why pkg(8) installs some things from the FreeBSD repo and some things from the Synth repo? If I do a `pkg upgrade -f` then it lists things like this:


```
gpgme-1.6.0 [FreeBSD]
  gnupg-2.1.8 [Synth]
  fdk-aac-0.1.4 [FreeBSD]
  expat-2.1.0_3 [FreeBSD]
  en-aspell-7.1.0_1 [Synth]
  duplicity-0.6.25 [Synth]
```

The packages for the FreeBSD ones are all present and correct in /var/synth/live_packages/All. I guess if the packages are identical in both repos it maybe just picks one at random?


----------



## marino (Feb 12, 2016)

Synth should have priority.  If I had to guess, pkg(8) goes to the repo where the last package is from.
You can probably force everything to synth if you "pkg prime-list > mylist", the delete all packages, then use the prime-list to reinstall (all from Synth)


----------



## hukadan (Feb 12, 2016)

marino@ said:


> If I had to guess, pkg goes to the repo where the last package is from.


I think this is related to the *CONSERVATIVE_UPGRADE* option. From pkg.conf(5) :


> CONSERVATIVE_UPGRADE: boolean
> Ensure in multi repository mode that the priority is given
> as much as possible to the repository where a package was
> first installed from.  Default: YES.


There was also this discussion on GitHub.


----------



## xtaz (Feb 12, 2016)

Thanks guys. I disabled the FreeBSD repo and then ran the upgrade again. After then reenabling the repo and running the command once more it shows Synth for everything. Makes sense now after reading about CONSERVATIVE_UPGRADE.


----------



## marino (Feb 12, 2016)

Let's try this again.  If no new problems reveal with version 0.99_7, I'll make the v1.00 release.

```
ports-mgmt/synth: Yet another release candidate

Unfortunately, there's been a bit too much change since 0.99_6 to
confidently release version 1.00, so another release candidate is
necessary.  Both new features and bug fixes were added.

New features:
  * Provide ability to define environment variables in a profile
    (/usr/local/etc/synth/<profile>-environment)
  * Support fetching by proxy using these environment variables
  * Add zsh and bash completion scripts
  * Accept port origins with trailing file separators (so people
    using completion scripts don't have to backtrack to remove them)
  * In text (non-curses) mode, output the current package build
    tally every 200 seconds (approximately)

Bug fixes:
  * Fix support for system roots that don't match host (e.g.
    ARCH, OSRELEASE, OSVERSION, etc
  * Fix ABI check for system roots that don't match host
  * Remove effect of system /etc/make.conf (originally seen when
    MAKE_JOBS_NUMBER was defined there and disabled synth)
```


----------



## marino (Feb 12, 2016)

garry, if I had to bet money, I think Line 83 is the culprit,
	
	



```
LIB_DEPENDS+=           libutil.so.9:${PORTSDIR}/misc/compat9x
```
My guess is you can comment that out and it won't rebuild anymore.


----------



## tankist02 (Feb 13, 2016)

On my system Synth keeps rebuilding devel/libc++ because it fails dependency. What does it mean? I don't have libc++ package installed:


```
root@bclinton:libc++# pkg info libc++
pkg: No package(s) matching libc++
```

My system:


```
root@bclinton:libc++# uname -a
FreeBSD bclinton 10.3-BETA2 FreeBSD 10.3-BETA2 #0 r295581: Fri Feb 12 13:54:41 PST 2016     toor@bclinton:/usr/obj/usr/src/sys/GENERIC  amd64
```


```
root@bclinton:libc++# pkg info synth
synth-0.99_7
...
```


```
root@bclinton:synth# tail devel___libc++.log
========================< phase : package         >========================
===>  Building package for libc++-208080
actual-package-depends: dependency on /usr/lib/libcxxrt.so not registered (normal if it belongs to base)
file sizes/checksums   [237]: ... done
packing files          [237]: ... done
packing directories      [0]: . done
===========================================================================

Finished: Friday, 12 FEB 2016 at 00:00:39 UTC
Duration: 00:00:10
```

Thanks a lot for creating such a great program!


----------



## garry (Feb 13, 2016)

marino@ said:


> garry@, if I had to bet money, I think Line 83 is the culprit,
> 
> 
> 
> ...



You would not lose your money.  After writing my last post and seeing your reply I took a closer look in the Makefile for the "missing" depends.  I did just as you suggest; I removed the compat9x depends line (libutil.so.9 is provided by the system -- I'm on FreeBSD 11 head).  That solved the problem.  Thank you, now it's time to cook myself a Friday night fried fish dinner and enjoy a movie!  After the movie I'll let Synth build my _entire_ repository and I'll stop using poudriere.

One question:  is the misc/compat9x dependency statement a bug in the lang/ghc Makefile?  (portlint(1) gives many other complaints about that port).  Should I report a bug against lang/ghc?  Or is this just a mismatch between how Synth scans Makefiles and what this one port does?  I checked another port that may require misc/compat9x, archivers/rar, but that works ok because the misc/compat9x port *is* pulled in by that package.


----------



## marino (Feb 13, 2016)

garry there is already a PR filed against that port for the same reason.  I found that out after I contacted the maintainers.  Since libutil doesn't appear to even be provided by misc/compat9x, I think it's a bug in any sense of the word.  I predict it will be removed once my finding is confirmed.


----------



## marino (Feb 13, 2016)

tankist02 said:


> On my system Synth keeps rebuilding devel/libc++ because it fails dependency. What does it mean? I don't have libc++ package installed:



You're not understanding what's it doing.  You would *never* check the host system in response to a synth error.  It brings in it's own packages that it already built, it would never rely on what's installed in the host.

It's probably rebuilding for the same reason lang/ghc (discussed above) and others before that rebuild: An issue with the Makefile that causes the dependency specifications to be different than what the built package actually has.  In the case of GHC, the Makefile said there were 5 dependencies but the scanned package only had 4 so Synth declared that it failed the dependency check.  I'm guessing the same thing is happening here.

Synth has been finding bugs in the port system due to this strictness.


----------



## talsamon (Feb 13, 2016)

> Synth has been finding bugs in the port system due to this strictness


That seems right and I had to excuse for some statements above.


----------



## PacketMan (Feb 13, 2016)

@marino, I am wondering if I am having an issue.  I've decided to try synth on a FreeBSD 10.2-RELEASE machine running GNOME3. The short story is Synth is continuing having to build a pile of ports.  1st time after the initial go-around it was 500+, then 300+ now 200+.  Also its skipping some and ignoring 1.  I watch it finish up last night and it look like it actually only installed 6 or so of the newly built packages meaning only 6 or ports were actually upgraded.  With ports like www/webkit-gtk3 being rebuilt every time I have to ask why.  NOTE: pretty sure all the ports being rebuilt are failing dependency check. When I used Portmaster I only ever had one missing library issue, never anything like this. These rebuilds are over 48 hours long. So by that time the ports tree has been updated, so I do my `portsnap fetch update` and presto - Synth says 200+ need to be rebuilt.  So my question is why?  What log can I look at to determine why?  Please note ports like www/webkit-gtk3, www/firefox and editors/openoffice-4 are usually always rebuilt.  OpenOffice was last updated Jan 29th, and Firefox was updated 9 Feb. Both of these were rebuilt a few days ago, and yet synth is telling me its going to do them again.

I am gonna do more testing; including issuing the upgrade command immediately after Synth finishes, and after a reboot, but both before I do another `portsnap fetch update`.

I know you have been finding issues with lots of ports and thus maybe that is the root cause on this GNOME3 machine.  Thoughts?


----------



## marino (Feb 13, 2016)

> ... newly built packages meaning only 6 or ports were actually upgraded.


This is not bad news.  pkg(8) only replaces what clearly is obsolete.  Synth (and Poudriere) rebuild even when a LIB/RUN dependency fails, no matter how low.  When they rebuild, pkg(8) might not see a difference.  You could argue, "but then why did I have to rebuild?"  Because you can't know beforehand and even pkg(8) perhaps needs to update but it can't detect that it does.

I've said it before, but LibreOffice and KDE4 are basically broken every single day. They have so many dependencies that each have dependencies which in turn has dependencies (and so on) that the chance of missing a cascading failure any given day is low, and that's PER DAY.  You should basically expect it.

Now that doesn't mean there isn't a bug in a port Makefile.
You can only tell if you build everything and then without updating the ports tree, build everything again.  Ideally Synth will say "nothing to do".  If it doesn't, that means it deleted a port a started a cascade.  You need to log what it's doing (perhaps in text mode) to determine which port gets deleted first.  Having WHYFAIL defined in environment will help diagnose as well (see GHC above)


----------



## marino (Feb 13, 2016)

Note that quarterly branches are much more stable so things are less likely to need rebuilding between runs.


----------



## marino (Feb 13, 2016)

One more thing on pkg(8).  This is why it's important for ports devs to bump the PORTREVISION.  If that pkg name (including version/port revision) don't change, then pkg will not reinstall the package, even if it's different, because it doesn't know that it's different.


----------



## PacketMan (Feb 13, 2016)

Okay, I'll keep an eye on it.  And thanks for the thought about using quarterly instead of latest.


----------



## marino (Feb 13, 2016)

PacketMan I wouldn't recommend Quarterly until the next version comes out in April.  I think the ports bugs that synth finds won't generally be fixed in Quarterly.


----------



## marino (Feb 13, 2016)

(but you can of course petition for those fixes to get backported and that should happen in general)


----------



## garry (Feb 13, 2016)

PacketMan said:


> ...synth is continuing having to build a pile of ports..... Thoughts?



This is a good thing, in my view a _very_ good thing and characteristic of continuous integration of such a huge collection of disparate sources.  As marino points out the totality of sources required by a big program, e.g. webkit2-gtk3, change every day, sometimes every hour!  FreeBSD is unique amongst the linuxes and bsd's in continuously integrating the operating system and ports.  There are FreeBSD build servers continuously building each branch of the system such as 10-STABLE, and the head of the operating system, 11-CURRENT.  There are also FreeBSD build servers continuously taking those latest builds of the operating system and building the latest releases of all 28,000 ports, and also the latest quarterly stable ports.  On a FreeBSD system `freebsd-update` and `pkg upgrade` can pull from that immense body of work and quickly update local machines.

You can see the status of the latest builds on the FreeBSD package build server at Portsmon  Check for all sorts of port changes and info at Freshports.

If you want to conduct your own local continuous integration you at least want to replicate the quality of FreeBSD in maintaining self-consistent repositories -- _everything_ in your local repository should be built against the other things that are present.  You don't want the repository to contain a package that was built on top of something that you've changed in the meantime.  It has been proved by a lot of experience (e.g. with Arch Linux) that that inevitably leads to mysterious failures.  Programs lock up, crash, behave bizarrely and eventually someone figures out that you "just need to rebuild X".  The only trustworthy path of development is to build self-consistent repositories in the first place.

Consider what self-consistent building means by looking at the list of libraries used by webkit2-gkk3:

```
Shared Libs required:
   libgtk-x11-2.0.so.0
   libxslt.so.1
   libsoup-2.4.so.1
   libpango-1.0.so.0
   libjpeg.so.8
   libicuuc.so.55
   libsqlite3.so.0
   libgstapp-1.0.so.0
   libcairo.so.2
   libgtk-3.so.0
   libxml2.so.2
   libwebp.so.5
   libfreetype.so.6
   libcairo-gobject.so.2
   libsecret-1.so.0
   libatk-1.0.so.0
   libgstfft-1.0.so.0
   libgstreamer-1.0.so.0
   libicui18n.so.55
   libharfbuzz-icu.so.0
   libgmodule-2.0.so.0
   libgdk_pixbuf-2.0.so.0
   libenchant.so.1
   libgio-2.0.so.0
   libXcomposite.so.1
   libXt.so.6
   libpangoft2-1.0.so.0
   libglib-2.0.so.0
   libgobject-2.0.so.0
   libXrender.so.1
   libGL.so.1
   libharfbuzz.so.0
   libEGL.so.1
   libX11.so.6
   libgstpbutils-1.0.so.0
   libXdamage.so.1
   libgstvideo-1.0.so.0
   libgsttag-1.0.so.0
   libgdk-x11-2.0.so.0
   libpng16.so.16
   libgstaudio-1.0.so.0
   libintl.so.8
   libfontconfig.so.1
   libgstbase-1.0.so.0
   libgnutls.so.28
   libgdk-3.so.0
   libpangocairo-1.0.so.0
```
If any of those libraries is changed webkit2-gtk3 should be rebuilt.

Now *pkg* doesn't always re-install every package that a port depends on, it tries to figure out from library sonames if that is necessary.  But sometimes programmers make significant changes in a library and neglect to renumber the library to reflect the "abi change".  Maybe the library itself is ok but some ancilary script changes.  It isn't always clear when it's necessary to force rebuilds.  We're dealing with a huge pile of complicated interdependencies created by tens of thousands of programmers some of whom are unqualified hacks and the result is what I call not "layers" of software but a "Pile of Software" or sometimes a "Pile of Shit".  So in the (correct) FreeBSD approach of continous integration even if *pkg* doesn't re-install everything, *everything* is in the repository -- correctly built against everything else.  On FreeBSD if you do get into bizarre failures after updates you can get a _completely_ consistent instatllation, whether from FreeBSD binary packages or from your local *synth* repository, by executing `pkg upgrade -f`. 

tldr:  The goal of *synth* or *poudriere* is to build a complete, self-consistent repository.  It's a challenging goal.  As everyone seems to be discovering in this thread *synth* does an especially good job of it.


----------



## PacketMan (Feb 13, 2016)

garry said:


> if any of those libraries is changed webkit2-gtk3 should be rebuilt............dealing with a huge pile of complicated interdependencies......



If rebuilding ports to ensure they all link with all the other ports is so darn necessary because we dealing with a huge pile of complicated interdependencies why are prebuilt packages even available for downloading? I've got a linux machine that only takes minutes to update, not 50 something hours just because 6 ports actually need to be upgraded!   I'm not sure I am further ahead with synth.  At least with portmaster what was needed was what got rebuilt.  On a GNOME 3 machine synth is basically saying lets rebuild 300+ ports only to have 6 or so ports actually upgraded.

Over the next few days I will likely start doing upgrades months apart to (a) hopefully take advantage of pre-built packages, and (b) simply have less 'downtime'.  That just said if I can get one 'master' machine doing all the heavy lifting (rebuilding all the ports that are in use over all my machines) and then use Bittorrent Sync to distribute the packages to the other machines, and then issue one single command `synth upgrade-system` to simply install those packages that were built by the master, with near zero ports being rebuilt then I will likely continue using synth.  If synth says yeah sorry still gonna rebuild those ports on these smaller machines because something is not pristine perfect then I would say I will likely stop using synth and go back to portmaster.

The best technical solution is not necessarily the best practical solution.


----------



## marino (Feb 13, 2016)

PacketMan@ you don't need synth on the other machines, only pkg(8).
You're not getting what a repository is, otherwise you wouldn't have even mentioned running synth on the smaller machines.


Going back to portmaster because it doesn't make you do stuff is like changing doctors to one that won't tell you that you have cancer.

The linux comparison is terrible.  There are no options, so there's no reason to build from source.  The equivalent is using stock FreeBSD packages which take exactly the same time as the linux case.

Also, why don't you take a look at the recent gnome3 thread in this forum and see what a smashing job portmaster did with it.

Plus, what is this "downtime" you speak of?
synth packages are not installed as you go (unlike portmaster).  Even with "synth prepare-system" nothing is upgraded.  When pkg(8) finally updates the system it takes seconds or at most a few minutes for hundreds of packages.

Nowhere above are you comparing apples to apples.


----------



## PacketMan (Feb 13, 2016)

The downtime is simply this: a machine is pretty much unusable when the cpu is 100% doing all this rebuilding.

Doctors are a good comparison considering how often they get you to do stuff that is unnecessary. Which I why fired mine and got a new one that simply instead helped me work with a certified asthma specialist and now I enjoy breathing again.

Referencing linux:  since my machines are 99% port stock (and I can make them 100% stock if I want to) again why is all the rebuilding necessary?

You are right, I do not (fully) understand what a repository is, I'm a newbie, but it was you that previously said I could/should point synth on the other machines to one single repository on one machine.  I now (think I) know I can copy the packages to the other machines and use pkg to install them; no synth at all. But again I think that supports what I am saying? -- why do we have to do all this rebuilding to guarantee all these complicated interdependencies are correct (and when most of that output is not even used) when I can simply bypass all of that by just installing packages that were prebuilt by another machine?

I have not had any issues with my GNOME3 machine and using portmaster to update it.  I read the usr/ports/UPDATING and plow through it. Takes a restart or two to fix an issue I missed in a new update, but at least I am compiling only 30 ports and not 300+.

Thanks for the feedback Marino, I am gonna carry on here and I will report back in a few days or in a week or two. (Depends on how much snow I get....got a nice snowmobile just purring to get out.)


----------



## garry (Feb 13, 2016)

PacketMan said:


> If rebuilding ports to ensure they all link with all the other ports is so darn necessary because we dealing with a huge pile of complicated interdependencies why are prebuilt packages even available for downloading? .....
> 
> If synth says yeah sorry still gonna rebuild those ports on these smaller machines because something is not pristine perfect then I would say I will likely stop using synth and go back to portmaster.



The FreeBSD prebuilt packages _are_ built correctly.  Their build server has done all of the work I described.  Those build servers are VERY busy machines, doing all that work so that we don't have to.  They use *poudriere* and they rebuild packages greedily.  The difference in behavior between FreeBSD and say Arch Linux (which is also a rolling system) is this:

In Arch Linux you can update your binary packages every day with `pacman -Syu` much as we do `pkg upgrade` in FreeBSD.  The Arch packages are built "lazily" with each developer throwing updated binaries into the pile as he sees the need to upgrade to a newer version or rebuild to accomodate changes in dependencies.  Because most dependencies are through dynamic libraries and in the perfect case they can be updated independently of their dependents stuff mostly updates day-by-day with relatively few packages being downloaded.  But because of the unconstrained chaos of complexity and undisciplined programming that I described it also often happens that an update _does_ break something.  Then users are expected to file bug reports, the developers track down the problem, maybe find that "Oh!  Y will need to be rebuilt now that we've updated X" and they bump the revision on Y, rebuild it and push out a new e.g. webkit2-gtk3 package and everyone is happy, after having a broken system for a day or two.

The same scenario happens with portmaster.  Search these forums to find numerous instances of people breaking some software by doing lazy upgrades with *portmaster*.

With FreeBSD binary updates it is also possible for your local machine to end up with a broken epiphany web browser because a cairo update broke webkit2-gtk3 BUT you can fix the problem instantly by re-installing your packages -- they are being drawn from a repository that itself is never broken in this way.  It is always clean and self-consistent.

You're not wrong of course.  The whole repository-building approach requires a lot of duplication of builds and is frustrating.  You may be happier with *portmaster*.  We're dealing with the problem with how to put together a reliable system from a large number of independently developed sources and doing it properly IS AN UNSOLVED PROBLEM.  Nix is an attempt to solve the problem from a better foundation.  They may have cracked the problem of reliable atomic upgrades!  You can learn a lot about The Problem starting from Eelco's PhD thesis.


----------



## marino (Feb 13, 2016)

1. That's not the definition of downtime.
2. If you have or want 100% stock then don't build at all.  Use packages.
3. Who said you have to build daily?  Can't you build weekly or monthly?
4. You're on the bleeding edge ports.  You have to expect this.  If you don't want change, get static branches.
5. The primary guys that wrote pkg, poudriere, and synth and all the experienced ports developers all say there's only one correct way to build and they all say portmaster isn't doing it correctly.  So we have a stack of resumes on one side yet the portmaster defenders have nothing like that except "it works for me".  Again, take a look at the gnome3 removal thread to debunk that.  Surely you don't think that *ALL* of us are wrong but guys with zero credibility are right?
6. I have no idea what you are asking about the respository.  We told you to have one common repository, you said that wasn't "practical" and you wanted to copy it.  Fine.  What does that have to do with rebuilding everywhere?  You have one machine for building, the remaining machines consume the packages.  You only need one copy of synth to produce the repository.
7. 300 ports is NOTHING.  My home machine can hit 1000 ports an hour, our blade machine can hit 2800 ports per hour.  If you use ccache, it will rebuild huge ports in minutes.  And again, how often is this happening?  A few hours every 2 weeks?

Maybe you just need to adjust you concept of operations.  (use stable, don't build daily, etc, etc)


----------



## PacketMan (Feb 13, 2016)

garry said:


> You may be happier with portupgrade.



Thanks Gary, I think I will re-read your last 2 posts a few times....is that a dim light I see coming on in the back of my head.  

No - I don't use portupgrade, I use portmaster on a couple machines (which I thought was different than portupgrade), and synth on others.  I will be happier when I buy a Dodge Hemi 440 powered 92 core 82000Ghz liquid nitrogen cooled processor!   Actually eyeing some sales on a 6-core cpu and motherboard.


----------



## PacketMan (Feb 13, 2016)

marino@ said:


> 1. That's not the definition of downtime.
> 2. If you have or want 100% stock then don't build at all.  Use packages.
> 3. Who said you have to build daily?  Can't you build weekly or monthly?
> 4. You're on the bleeding edge ports.  You have to expect this.  If you don't want change, get static branches.
> ...



With all due respect I take offence to this post. An OS and its tools successfully making it in the market accommodates a variety of thinking and user groups. I may have zero creditability but I am telling you I have had zero issues maintaining a GNOME3 machine with portmaster, and I don't have selective memory thank you.  I am sorry I don't have the resume you have, or a 28000 port per hour blade machine either.  Maybe your release notes should say something like "synth is only for people with fancy resumes and big budget servers so they can tolerate rebuilding 300+ ports EACH AND EVERY TIME only 6 or so ports actually need to be upgraded. No one said I had to build daily thank you, I was doing it to provide valuable feedback to you, but since you are clearly right and I am clearly wrong (and apparently in a lesser class too) I will take a break in reporting my findings.  Oh, I am using ports because guys in this forum who I put a lot of weight and trust in their experience advised me to stick with ports.

Bye! ........Come on snow.


----------



## marino (Feb 13, 2016)

I'm not sure what you are taking offense at.
Either you believe the logic is correct or you don't.  If you don't, and your belief against what all the experts are saying, I'm curious how that's reconciled.
You didn't respond to any of the implied questions, the primary ones being: How often are you rebuilding these things?

and why are you so sure other methods would have upgraded those 6 ports?

The "each and every time" line is a crock though.  If you don't portsnap/svn update the ports tree, it won't rebuild at all.  It's rebuilding because you are changing the tree.  As all the stuff about beefy machines and blades (missed the point).  I'm saying 300 ports is nothing, even on wimpy machines, especially when the machines use ccache.   But 300 ports per day every 2 weeks is only 600 ports a month.  Even the freebsd servers are only building once every 7 days.


----------



## marino (Feb 14, 2016)

I got an interesting request to make Synth detect cached options files that are identical to the default options and delete them.  Synth can't do that because the options are mounted read-only, but I came up with a good alternative.  I added the script directly to ports.  Here is the commit message:


```
Add new tool script: redundant-opt-files.sh

I got a request to make Synth identify "redundant" cached option files,
where "redundant" means the saved port options are identical to the
default options.  For Synth (and portmaster?) which use the port's
cache options, these redundant files are somewhat of a liability.  At
best they do nothing (Synth assumes default options) and at worst they
will cause a future build to stop if the maintainer changes the port
options later.

This situation is avoidable.  Rather than build detection into Synth,
I decided to write a generic shell script for ports.  When run, it
will display the full path to the port's options directory if the
cached options are the same as the defaults.  This output is suitable
to pipe to "xargs rm -rf" to remove all the redundant options in a
single command.
```

https://svnweb.freebsd.org/ports?view=revision&revision=408849

Once somebody decides to switch completely to Synth, it's a good idea to run this script.
In fact, it's a good idea to run the script *before* you run Synth the first time.
I found 122 identical cached option files on my machine.


----------



## xtaz (Feb 14, 2016)

Doh! I wish that script was there a few days ago before I went through every port in that directory manually! Yeah I think I deleted well over 100.


----------



## fernandel (Feb 14, 2016)

I decided to try devel/ccache. I just made a directory in /var/cache/ccache and I have as I read in /etc/make.conf:

```
WITH_CCACHE_BUILD=yes
```
Is this enough or I need to do something more, please?


----------



## marino (Feb 14, 2016)

fernanel@ no, that's not correct.  Remove that line from /etc/make.conf (if you put it there for synth).

All you need to do is `ccache -s` and copy the value of "cache directory" to `synth configure` screen.
Make sure you actually set up ccache first, see the man page (or ccache help screen) to size it correct and define the location if you don't like the default.


----------



## xtaz (Feb 15, 2016)

Some other weirdness that I'm trying to work out. I've seen `pkg` want to install math/gmp and lang/lua52 for no apparent reason quite a lot on apparently unrelated packages. For example:


```
# pkg install devel/gmake
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
Updating Synth repository catalogue...
Synth repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 3 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  gmake: 4.1_2 [Synth]
  gmp: 5.1.3_3 [Synth]
  lua52: 5.2.4 [Synth]
```

If I just use synth to do this:


```
# synth install devel/gmake
Stand by, updating external repository catalogs ... done.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Scanning entire ports tree.
Scanning existing packages.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%  264 B  0.3kB/s  00:01
Fetching packagesite.txz: 100%  66 KiB  67.6kB/s  00:01
Processing entries: 100%
Synth repository update completed. 220 packages processed.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  gmake: 4.1_2 [Synth]
```

Here is another example:


```
# pkg upgrade -f dns/ldns
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
Updating Synth repository catalogue...
Synth repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 3 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
  lua52: 5.2.4 [Synth]
  gmp: 5.1.3_3 [Synth]

Installed packages to be REINSTALLED:
  ldns-1.6.17_5 [Synth]
```

I'm wondering why there is different behavior between synth and pkg, but also why math/gmp and lang/lua52 are related at all to gmake or ldns in the first place as they are not dependancies of either?

Thinking my pkg database might be screwed in some way I actually tried doing `pkg prime-list > list`, `pkg delete -ay`, `rm /var/db/pkg/local.sqlite`, and `pkg install `cat list`` but this hasn't changed anything.


----------



## marino (Feb 15, 2016)

i don't know.  Isn't that implying that synth is doing the correct thing and it's pkg that is acting weirdly?


----------



## xtaz (Feb 15, 2016)

Yeah, sorry, I know it's not actually a synth issue, more a pkg one. Actually.... Just tried disabling the FreeBSD repository again and only leaving the synth one. Then it only installs gmake. So I guess it's the FreeBSD repo being consulted and having different build options or something..... Strange though, because both devel/gmake and dns/ldns do not have any options files in /var/db/ports.


----------



## Crivens (Feb 15, 2016)

Is that a build- or runtime dependency? pkg will not install build dependencies whereas synth will have to build them.


----------



## xtaz (Feb 15, 2016)

It's neither build or runtime. That's why it's weird.


----------



## marino (Feb 15, 2016)

xtaz try setting CONSERVATIVE_UPGRADE to false in pkg.conf file

maybe pkg(8) will behave after that.


----------



## xtaz (Feb 15, 2016)

Nope. That setting doesn't seem to make any difference. Only setting enabled to no for the FreeBSD repo stops it.

I have worked around it though, it appears I can run `pkg install -r Synth gmake`. It works then!


----------



## marino (Feb 15, 2016)

Is your system a mix of FreeBSD and synth packages?  You could always try flushing them and replacing with all synth packages (e.g. https://www.dragonflybsd.org/docs/howtos/HowToDPorts/#index4h1 )


----------



## marino (Feb 15, 2016)

(i'm grasping at straws and I don't know what the negative of just leaving it is)


----------



## xtaz (Feb 15, 2016)

It's 100% Synth packages. I even deleted all packages and reinstalled them all from the Synth repo to make sure everything was consistent. Oh well, it's OK now I have learnt about the -r switch to pkg install. That will do the trick


----------



## marino (Feb 15, 2016)

i just discovered that at least on dragonfly, the graceful shutdown no longer works on `synth rebuild-repository`.  It appears that when synth is in text mode (as opposed to curses mode), the keypress detection is not triggered by control characters.  Can anyone confirm the same from FreeBSD?  I might have to change this shutdown key to capital-Q or something.  really obnoxious.


----------



## marino (Feb 15, 2016)

the same thing seems to happen on FreeBSD as well.  Hopefully I can figure out how to make control codes always work, but I may have to switch the key again.


----------



## xtaz (Feb 16, 2016)

FYI. From reading the man pages for `pkg` I think I worked out why I was having issues with it looking in the wrong repo. It says:



> Install a package from a remote package repository.  If a package is found in more than one remote repository, then installation happens from the first one.  Downloading a package is tried from each package repository in turn, until the package is successfully fetched.



And if I run `pkg -vv` it shows FreeBSD before Synth:


```
Repositories:
  FreeBSD: {
  url  : "pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest",
  enabled  : yes,
  priority  : 0,
  mirror_type  : "SRV",
  signature_type  : "FINGERPRINTS",
  fingerprints  : "/usr/share/keys/pkg"
  }
  Synth: {
  url  : "file:///var/synth/live_packages",
  enabled  : yes,
  priority  : 0
  }
```

So I guess it maybe sees the FreeBSD one first and then installs/upgrades from that. Although just using -r Synth on the install/upgrade commands solves the problem as it only uses Synth then.


----------



## kpa (Feb 16, 2016)

xtaz, it's very much recommended that you don't mix the official repo with your self built packages. It works only to an extent that you can replace leaf ports (ports that are not required by other ports) from the other repository, anything else and you might cause a dependency conflict that can not be solved by the dependency solver.


----------



## xtaz (Feb 16, 2016)

Indeed. I'm just mentioning this because other people might have the same confusion as me. The instructions for synth don't say to disable the FreeBSD repo, and in fact it's designed to work with it to fetch packages from an external repo if it doesn't need to build them itself. Comments in this thread also state that because synth's config starts with 00 that it takes priority over the FreeBSD repo because they are supposedly read in alphabetical order. Clearly the way pkg works means that this isn't always the case. So either you have to disable the external repo completely or you have to remember to use the -r switch to point it to a specific repo. This basically started off because I wondered why `pkg install gmake` wanted to install lua52 and gmp, which I found was because pkg was not in fact using Synth as a priority at all. So just mentioning my findings in case it helps others


----------



## marino (Feb 16, 2016)

xtaz said:


> FYI. From reading the man pages for `pkg` I think I worked out why I was having issues with it looking in the wrong repo. It says:
> 
> _Install a package from a remote package repository.  If a package is found in more than one remote repository, then installation happens from the first one.  Downloading a package is tried from each package repository in turn, until the package is successfully fetched._
> 
> ...



The display order doesn't matter.
I think actually pkg(8) is broken.
The "first" one is the one that appears first in an alphabetical sort.  This is why the synth conf file starts with "00_".  The synth repo is the first one.

However, with -VV, I'm seeing synth first.  You didn't define your own synth.conf did you?


----------



## marino (Feb 16, 2016)

kpa said:


> xtaz, it's very much recommended that you don't mix the official repo with your self built packages. It works only to an extent that you can replace leaf ports (ports that are not required by other ports) from the other repository, anything else and you might cause a dependency conflict that can not be solved by the dependency solver.



this isn't exactly true.  When synth has the prefetch option, it fetchs packages from the official repo and saves it in the synth repo and then when the repo is created it looks like synth created.  So anyone with prefetch option enable is mixing packages (but the packages were vetted)


----------



## marino (Feb 16, 2016)

xtaz said:


> Clearly the way pkg works means that this isn't always the case. So either you have to disable the external repo completely or you have to remember to use the -r switch to point it to a specific repo. This basically started off because I wondered why `pkg install gmake` wanted to install lua52 and gmp, which I found was because pkg was not in fact using Synth as a priority at all. So just mentioning my findings in case it helps others



okay, so you already knew about alphabetical.  I don't think the "priority" setting works at all.  
I don't know if the order of -VV is significant or not (I tend to think it is but I'm not sure).
I still think the conf files priority is alphabetical.


----------



## marino (Feb 16, 2016)

I released synth version 1.00 and started a new thread: Thread 55144


----------



## kpa (Feb 17, 2016)

I'm curious about how the "foreign environment" building is going to work. There must be ports that make assumptions about what is available in terms of binaries based on the uname(1) output. Let's say the host is FreeBSD 10 amd64 and you're building for FreeBSD 9 amd64. Just as an example FreeBSD 9 still has /usr/bin/dig file because it still has BIND in base and some ports might test for the existence of this file or could try to use some other FreeBSD 9 only binaries. I don't see how Synth could build such ports successfully if it can't provide the "real thing" for the build environment.


----------



## marino (Feb 17, 2016)

check any log produced by 1.00, specifically the ENV variables set.
you will see definitions for UNAME components.  The uname(1) program and the uname(3) libc function both use these values when they are defined instead of sysctl results.  That's how it works.


----------



## kpa (Feb 17, 2016)

Yes I know that it sets the environment correctly and that's what Poudriere does as well when building for foreign environments. I'm more asking about the actual build environment because you have chosen not to go the Poudriere route which is providing pristine environments using separately installed base systems. I can see that your approach would work in most cases but as I said there must be ports that (correctly) assume that a FreeBSD 9 build environment would have all the FreeBSD 9 tools installed.


----------



## marino (Feb 17, 2016)

Nothing is different.
if you want to build FreeBSD 9 packages, you need a FreeBSD 9 system root -- just like poudriere.
The quickest way is download and extract.
(this is advanced stuff, but I expect anyone trying this already gets the basics)


----------



## marino (Feb 17, 2016)

this might help:

Poudriere:

sysroot installed via command (either built or installed or declared)
ports install via command (portsnap, svn, declared)
sandbox is constructed using freebds jails and poudriere refers to the sandbox as a jail
Synth:

sysroot is declared ("/" is default but user is responsible for providing it)
ports are declare (PORTSDIR is scanned to get the default)
sandbox is constructed by chroots but all based on sysroot.  Synth refers to definition as a profile.


----------



## marino (Feb 17, 2016)

pristine is a frame of mind.  If one extracts a FreeBSD release and doesn't touch it, and creates a synth profile that points to it, that is just as "pristine" as anything poudriere will use.  The only risk is when the live system "/" is used.  It's very clean but not guaranteed.


----------



## kpa (Feb 17, 2016)

Ignore me, I misunderstood how the foreign environments are used in Synth.


----------



## fernandel (Feb 17, 2016)

Hi!

I did run `synth upgrade-system`and I got:


> Scanning existing packages.
> libc++-208080.txz failed dependency check.
> 
> 
> ...



I quit Synth with ctrl-q because nevwr fetch the file. And devel/compiler-rt-devel is marked broken.

Thank you.
BTW: I have updated Synth.


----------



## fernandel (Feb 18, 2016)

There are also a problem with `synth status`


> Querying system about current package installations.
> Stand by, comparing installed packages against the ports tree.
> Scanning existing packages.
> libc++-208080.txz failed dependency check.
> ...


In the results.txt are the same two files.


----------



## marino (Feb 18, 2016)

fernandel the maintainer broke the port

```
> make -V BUILD_DEPENDS
clang-devel: not found
make: "/usr/ports/Mk/Uses/compiler.mk" line 69: warning: "clang-devel --version" returned non-zero status
make: "/usr/ports/Mk/Uses/compiler.mk" line 120: warning: "clang++-devel -### /dev/null 2>&1" returned non-zero status
llvm-config-devel:/usr/ports/devel/llvm-devel /usr/local/bin/cmake:/usr/ports/devel/cmake ninja:/usr/ports/devel/ninja
```
It's not Synth's fault.

I didn't understand your comment about /tmp/synth_status_results.txt.  It's supposed to be the same as what is on the screen.


----------



## fernandel (Feb 18, 2016)

marino@ said:


> fernandel the maintainer broke the port
> 
> ```
> > make -V BUILD_DEPENDS
> ...



Before it did show me all my built ports also.


----------



## marino (Feb 18, 2016)

i don't think so.  see the description in the man page.


----------



## fernandel (Feb 18, 2016)

marino@ said:


> i don't think so.  see the description in the man page.





> It will list all the ports that will be rebuild
> * along* with a total, and it also logs the same information to
> /tmp/synth_status_results.txt since the full list is often longer than
> the terminal height.


Thanks.


----------



## marino (Feb 18, 2016)

i'm not sure why you quoted it and underlined it.  Synth did exactly what the man page said it would do.  I am very confused about what you are trying to tell me.


----------



## vejnovic (Feb 18, 2016)

I like your synth. I try it and it works as expected.

I have prepare some ports which are not yet part of the ports tree and I want to test them.
When manually add port to the ports tree I got message before the build start:

```
Scan of devel/wrapper failed, it will not be considered.
```

What I have to done that synth build this port?


----------



## talsamon (Feb 18, 2016)

You have add it to the devel/Makefile to silence this message.
(Better make a own category for your ports. The category (here devel)-Makefile could be overweitten next update. You have then write your category in /usr/ports/Makefile. This will not overwritten.).


----------



## marino (Feb 18, 2016)

talsamon is correct; that's the problem (on FreeBSD, each category has a makefile and for the port to be considered eligible, it has to be listed in that makefile.  On DragonFly, there is no category makefile, by definition all ports are considered eligible)


----------



## talsamon (Feb 18, 2016)

Another question: Is in `synth` something similar to `portmster -o neworigin oldorigin.`


----------



## marino (Feb 18, 2016)

No, because it's not applicable
Look for examples of that command in /usr/ports/UPDATING.  Now look at the pkg(8) equivalent.  Most of the time a regular upgrade fixes it.  In any case, it would be the pkg(8) command that would be run.  

portmaster installs ports.
synth does not install ports.  It tells pkg(8) to install packages from the local repository it builds (and then only on the localhost).  So you can see it's not applicable.


----------



## kpa (Feb 19, 2016)

Both do install ports in a way because built port ends up as an installed package either way. The real difference is that portmaster builds and installs one port at a time taking no notice of the previously built ones, this effectively disables the dependency solver in pkg(8). Synth on the other hand uses a set of built ports/packages and uses pkg-upgrade(8) to perform the upgrade and that allows pkg to use its dependency solver to figure out if the requested upgrade is possible and "walk" the dependency tree to figure out all packages that need to be replaced to fulfill the dependency conditions.


----------



## marino (Feb 19, 2016)

i don't really see it that way.  If synth had no "install" command or "upgrade-system" command and simply stopped at building the repository, then the user would have to run something like `pkg upgrade -r Synth` manually and then it's clear Synth is not involved with installing ports.  The install/upgrade commands are just for convenience; they just invoke pkg.


----------



## fernandel (Feb 19, 2016)

marino@ said:


> i'm not sure why you quoted it and underlined it.  Synth did exactly what the man page said it would do.  I am very confused about what you are trying to tell me.


I am sorry. It was mistake. I sent quoted text to someone and it happened


----------



## talsamon (Feb 20, 2016)

Would anybody stop this incredible nonsense. Nobody can tell me that the update of gettext causes 608 packages to rebuild and that is technically correct (including e.g. compilers, never need gettext).

If this is true, why could anybody live only one day with portmaster, and have no unresponsive system?

To rebuild all is the most minimal and simplest idea. This can never be the truth.


----------



## protocelt (Feb 20, 2016)

talsamon, it is technically correct and I really just don't think you understand how a package builder works. Poudriere works the same way in that respect. It's already been explained how this works earlier in this monster thread. 

If you don't like the way Synth works, you're certainly free to use Portmaster or any other tool you like if that works better for you, but continuing to make posts complaining about Synth rebuilding too many packages all the time isn't adding anything constructive to the discussion here.


----------



## talsamon (Feb 20, 2016)

With the compiler I was wrong (depend via indexinfo from gettext-runtime), but


```
pkg info -r gettext-runtime | wc -l
  231
```
that's the truth.
I stop the (senseless) experiment with synth an leave the thread.


----------



## kpa (Feb 20, 2016)

I think this illustrates what the whole issue is about. The mail/mutt port requires the following ports (at runtime and also at build time):


```
% pkg info -d mutt         
mutt-1.5.24_4:
        openssl-1.0.2_8
        urlview-0.9.20131021
        mime-support-3.58
        db5-5.3.28_3
        libiconv-1.14_9
        libidn-1.31
        gettext-runtime-0.19.6
```

Now, devel/gettext-runtime in turn requires:


```
% pkg info -d gettext-runtime-0.19.6
gettext-runtime-0.19.6:
        indexinfo-0.2.4
```

However, print/indexinfo is not directly required by mail/mutt but indirectly.


```
% pkg info -r indexinfo-0.2.4     
indexinfo-0.2.4:
        gettext-tools-0.19.6
        libffi-3.2.1
        libidn-1.31
        gettext-runtime-0.19.6
```

Even so when print/indexinfo has an update you must rebuild mail/mutt, absolutely and no way to avoid it. This is because the change in print/indexinfo triggers the deletion of the package for the dependent devel/gettext-runtime. Now if we first build print/indexinfo and then the dependent devel/gettext-runtime we have to ask a question, "did the recompilation of devel/gettext-runtime with the updated print/indexinfo cause any changes to it that could affect mail/mutt?". Unfortunately that's not something you can test by automated programmed means. You could manually inspect everything in devel/gettext-runtime and decide it didn't change but requires human ingenuity and that's something a computer program does not have. So the only way to answer the question is to play it safe and delete the package for mail/mutt and rebuild the port.


----------



## marcpearsonti (Feb 20, 2016)

What am I suppose to do with a situation like this one


```
Scanning entire ports tree.
progress: 33.12%             
culprit: games/libretro-cores
Scan aborted because dependency is malformed.
FETCH: 0.20151110 (games/libretro-cores)
```

Synth can't build my repository anymore. How I can ignore this ports ?


----------



## talsamon (Feb 20, 2016)

Write in port Makefile

```
BROKEN=<TAB><TAB>some_text
```
, so synth ignores it.

(marked this thread  as unwatch, but still got a mail...).


----------



## free-and-bsd (Feb 21, 2016)

marino@ said:


> 1. That's not the definition of downtime.
> ...
> 7. 300 ports is NOTHING.  My home machine can hit 1000 ports an hour, our blade machine can hit 2800 ports per hour.  If you use ccache, it will rebuild huge ports in minutes.


Hi, marino@, first of all let me thank you greatly for this port.
Would you please share how you've set up your machine to rebuild packages with such speeds?

Once again, thank you for this port, I like it very much and I also like the idea after having read all these discussions and complaints etc. I HAVE had problems upgrading with portmaster created packages, so I can see now why this happened.


----------



## free-and-bsd (Feb 22, 2016)

Yes, I definitely like this ports-mgmt/synth. I've never used poudriere, only portmaster. But the latter doesn't make parallel builds, does it? And many other things.

But here I have a question. I usually build editors/libreoffice with -j6 for speed. How could I define the same with synth?

I also want to confirm that the time synth takes to build all my packages seems to be shorter than with portmaster. Neither does it make my system (Intel Celeron G2030 3.00GHz RAM 16G) "unresponsive" with 4 builders x 2 jobs per builder. Except for that unfortunate webkit-gtk package which takes forever to build either way, but still it's only _time_ and not an "unresponsive system".

BTW, which is the best configuration for speed with synth? Is it to have more builders, or have less builders but more jobs per builder?


----------



## marino (Feb 22, 2016)

free-and-bsd said:


> But here I have a question. I usually build editors/libreoffice with -j6 for speed. How could I define the same with synth?



The jobs number are global; they affect every port that hasn't specifically disabled multijob builds.  The only setting is shown on `synth configure`



> BTW, which is the best configuration for speed with synth? Is it to have more builders, or have less builders but more jobs per builder?



It is usually limited by memory.  When memory is the limiting factor, it's better to have less builders with more jobs per builder.  If the machine has a ridiculous amount of memory, you could get away with more builders with more jobs per builder.


----------



## free-and-bsd (Feb 22, 2016)

talsamon said:


> ...
> Better make a own category for your ports. ... You have then write your category in /usr/ports/Makefile. This will not overwritten.).


Right, but the port will not be built for the reason that "category XXX is not in list of valid categories". And these are defined in /usr/ports/Mk/bsd.port.mk, so I guess it must be defined there as well?

But I have run into another problem here. I'm trying to build a customized port which builds well with both portmaster and manual `make install` command. But with synth this problem appears (from synth log):

```
===>  Configuring for grub2-dev-2.02.b2.d
Importing unicode...
autogen.sh: python: not found
*** Error code 127

Stop.
```
Looks like /usr/local/bin/python symlink is not installed in the build environment? How can that be fixed?


----------



## marino (Feb 22, 2016)

free-and-bsd said:


> Right, but the port will not be built for the reason that "category XXX is not in list of valid categories". And these are defined in /usr/ports/Mk/bsd.port.mk, so I guess it must be defined there as well?



No.  Just add "VALID_CATEGORIES=mycategory" to /etc/make.conf (or the synth profile version of make.conf)



> But I have run into another problem here. I'm trying to build a customized port which builds well with both portmaster and manual `make install` command.


Forget "manual".  It has to build in `poudriere testport` or `synth test` and pass.  if it does, it works in manual, and it's required to work in the former.



> But with synth this problem appears (from synth log):
> 
> ```
> ===>  Configuring for grub2-dev-2.02.b2.d
> ...



It looks like grub2-dev has a problem.  It should be reproducible in poudriere using the same exact options.


----------



## free-and-bsd (Feb 22, 2016)

marino@ said:


> ...
> It looks like grub2-dev has a problem.  It should be reproducible in poudriere using the same exact options.


 Thank you very much for your answers.
Now this one is my custom port based on GIT version of GRUB2 (not submitted officially) and I'm not using poudriere. Having fixed the python problem (replaced python with /usr/local/bin/python2.7 in autogen.sh), I've run into another similar one: /usr/local/bin/autoreconf symlink is not installed in build environment, so it can't find autoreconf in the same way it failed to find python.OK, that symlink belong to package autoconf-wrapper, but /usr/local/bin/python symlink seems to be part of python package.


----------



## marino (Feb 22, 2016)

well, that's kind of my point.  You pretty much *have* to check new ports with either poudriere or synth.  If you don't, the person that tests it will and as soon as it fails, they'll kick it back to you.  so doing those checks aren't optional (and I recommend it so so you can save at least one kickback).

autoreconf is another setting.  it's probably something like "USES+= autoreconf".  check /usr/ports/Mk/Uses/autoreconf.mk or grep other ports to see how it's used.


----------



## free-and-bsd (Feb 22, 2016)

marino@ said:


> well, that's kind of my point.  You pretty much *have* to check new ports with either poudriere or synth.  If you don't, the person that tests it will and as soon as it fails, they'll kick it back to you.  so doing those checks aren't optional (and I recommend it so so you can save at least one kickback).
> 
> autoreconf is another setting.  it's probably something like "USES+= autoreconf".  check /usr/ports/Mk/Uses/autoreconf.mk or grep other ports to see how it's used.


Right, I've got the message now, thank you. Yes, I just missed it about autoreconf in porter's handbook. It works now.


----------



## kpa (Feb 22, 2016)

free-and-bsd said:


> Thank you very much for your answers.
> Now this one is my custom port based on GIT version of GRUB2 (not submitted officially) and I'm not using poudriere. Having fixed the python problem (replaced python with /usr/local/bin/python2.7 in autogen.sh), I've run into another similar one: /usr/local/bin/autoreconf symlink is not installed in build environment, so it can't find autoreconf in the same way it failed to find python.OK, that symlink belong to package autoconf-wrapper, but /usr/local/bin/python symlink seems to be part of python package.



I think the port should depend (build time dependency only probably) on lang/python if it wants to use the unqualified python interpreter without a version number. That port will provide the /usr/local/bin/python symlink. This should be cleaner solution that hacking the configure script.


----------



## free-and-bsd (Feb 22, 2016)

kpa said:


> I think the port should depend (build time dependency only probably) on lang/python if it wants to use the unqualified python interpreter without a version number. That port will provide the /usr/local/bin/python symlink. This should be cleaner solution that hacking the configure script.


Yes, I should think so, too. 

```
BUILD_DEPENDS= ${LOCALBASE}/bin/python:${PORTSDIR}/lang/python
```
 works fine.
Before it had only

```
USES= ... python ...
```
 Well, actually I stole this file from another grub port, that's why. Borrowed it, modified accordingly, but didn't know yet which tool is the right one to work with ports.

So, thank you marino@ again for your amazing tool. My impressions are VERY positive. Rebuild of ALL my packages took considerably shorter time than with portmaster, which was originally used. Yea, system was perfectly _workable_ during the time of the build, no overloads or something. Sure, firefox and libreoffice were a bit slower during the build time, but not so much so as to make working a torture. So, having "ridiculous amount of memory" is not entirely useless after all , and I'm glad it isn't. When building that machine I figured that 16G of RAM would cost much less than quadcore CPU, which would in turn require much RAM to do its thing anyway.

In addition to that, synth has helped me to learn how to create ports properly, and a good deal of information that goes along with that. Great! Thank you all for help.


----------



## kpa (Feb 22, 2016)

I couldn't find out if this is provided by Mk/Uses/python.mk so the BUILD_DEPENDS method has to be used I guess, the PORTSDIR is now an optional prefix in the path because it is filled in automatically so it becomes:


```
BUILD_DEPENDS+= ${LOCALBASE}/bin/python:lang/python
```


----------



## free-and-bsd (Feb 22, 2016)

kpa said:


> I couldn't find out if this is provided by Mk/Uses/python.mk so the BUILD_DEPENDS method has to be used I guess, the PORTSDIR is now an optional prefix in the path because it is filled in automatically so it becomes:
> 
> 
> ```
> ...


Right, I just added another line to this effect to other BUILD_DEPENDS mentioned in Makefile. Thank you for your help, it works all right.
Maybe you could advise on debugging a similar problem with this same port?

The script grub-core/genmod.sh uses symbol @TARGET_STRIP@ which causes build to fail for some reason:

```
if test xpc != xemu; then
@TARGET_STRIP@ --strip-unneeded \
  -K grub_mod_init -K grub_mod_fini \
  -K _grub_mod_init -K _grub_mod_fini \
  -R .note.gnu.gold-version -R .note.GNU-stack \
  -R .note -R .comment -R .ARM.exidx $tmpfile || exit 1
fi
```
So I had to create a patch to replace it with /usr/local/bin/strip to make it work. But like the previous example, I should prefer a cleaner way of managing it.


----------



## kpa (Feb 22, 2016)

Anything like @FOO_BAR@ is supposed to be filled in by autotools (devel/autoconf etc.) so it's not detecting our /usr/bin/strip for some reason. There is STRIP_CMD provided by the ports system but it's supposed to be used in post-install. You'll probably have to hack the configure script (and that genmod.sh script) anyway to either fix the @TARGET_STRIP@ part or replace it with our own STRIP_CMD that is done in post-install.

https://www.freebsd.org/doc/en/books/porters-handbook/install.html


----------



## free-and-bsd (Feb 22, 2016)

OK, don't bother. It's been fixed upstream. 

I discussed this for some time with GRUB2 developer team and it was because our CURRENT uses a rather old version of strip & friends. But then they upgraded their binutils to the latest version and it behaves the same way as our strip. So they finally had to fix this and I don't need to bother, just update my port to the latest GIT.


----------



## marino (Feb 22, 2016)

kpa said:


> I couldn't find out if this is provided by Mk/Uses/python.mk so the BUILD_DEPENDS method has to be used I guess, the PORTSDIR is now an optional prefix in the path because it is filled in automatically so it becomes:
> 
> 
> ```
> ...



using PORTSDIR optionally isn't done operationally yet (i think it's because the quarterly branch doesn't know it and that would cause problems if the trunk ports removes it and then the port is backported).

I think the tree is going to be actively switched to no PORTSDIR when the next quarterly is branched, but until then even new ports will/should still have it.


----------



## Pedja (Mar 4, 2016)

Since sysutils/py-salt is broken because lang/python27 is built against
an old version of security/openssl, as this error seems to suggest


```
pedja@freebsd102:/usr/ports/lang # salt
Traceback (most recent call last):
  <snip>
  import ssl
  File "/usr/local/lib/python2.7/ssl.py", line 97, in <module>
  import _ssl  # if we can't import it, let the error propagate
ImportError: /usr/local/lib/python2.7/lib-dynload/_ssl.so: Undefined symbol "SSLv2_method"
```
I wanted to rebuild all the dependencies for Salt.

After some googling and reading synth(1), I came up with this:
`pkg query %do py27-salt-2015.8.7 | awk -vORS=" " '1' | xargs synth force`

But, Synth exits with the error just after the build of the packages.
Build log:

```
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
py27-pytz-2015.7,1.txz failed dependency check.
py27-six-1.9.0.txz failed dependency check.
py27-Babel-2.2.0_1.txz failed dependency check.
py27-backports_abc-0.4.txz failed dependency check.
py27-dateutil-2.5.0.txz failed dependency check.
py27-jmespath-0.9.0.txz failed dependency check.
py27-singledispatch-3.4.0.3.txz failed dependency check.
py27-docutils-0.12.txz failed dependency check.
py27-certifi-2015.11.20.txz failed dependency check.
Scanning existing packages.

The task is complete.  Final tally:
Initial queue size: 25
  packages built: 25
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:05:52
The build logs can be found at: /var/log/synth
Would you like to rebuild the local repository (Y/N)?

raised ADA.IO_EXCEPTIONS.END_ERROR : a-textio.adb:608
```

Should I just nuke the synth repository and start from scratch?
Or I am just DoingItwrong[tm]?

Synth-1.11, FreeBSD-10.3-BETA3, latest ports revision.

Thank you.


----------



## marino (Mar 4, 2016)

try run the same command but with "just-build" instead.

"raised ADA.IO_EXCEPTIONS.END_ERROR : a-textio.adb:608" is showing because it's asking a question and there's no terminal (apparently).  Since you already deleted the packages, "just-build" should work.  it doesn't ask questions.  Probably synth needs to detect a missing terminal and skip the questions.

OR

just run `synth rebuild-repository` or `synth upgrade-system`
It looks like it finished building. you don't need to feed it any more ports.


----------



## marino (Mar 4, 2016)

Pedja@ the next release will be able to handle your first command:
https://github.com/jrmarino/synth/commit/beae1cdf271659d1aca23d77959803cbd567bdb0


----------



## marino (Mar 5, 2016)

marcpearsonti said:


> If Portmaster can show the versions when showing the "status", Synth should be able to too . Each ports has a version and revision in the Makefile. For the current installed ports, pkg(8) version could be great ! But I'm not a pro !



Hi Marc,
I added this functionality ( https://github.com/jrmarino/synth/commit/276d8309dab8aa9c95c3b165fb3de614e8df980b ) and it will be available on the next release of Synth.

Note that the version display compares what is packaged in the profile's package directory NOT what is installed on the host system.  If you think about it, this makes sense.  (usually what's packaged is also installed but there's no rule where that is always true).

There are two status lists, one for  the screen and one logged in the /tmp directory.  The screen version shows N/R/U (for New package, Rebuild existing package (same version) and Upgrade package (new version)).  For the latter, the old version and new version are also shown on the screen.  For the log, versions are also shown for "New" and "Rebuild" entries.   I didn't show them on the screen because it made the display hard to read.  Anyway, I think you should be happy with the implementation.


----------



## Pedja (Mar 5, 2016)

(Apologies for the late reply, I was debugging libvirt, so the FreeBSD VM was offline for a while).

Anyway, `synth upgrade-system` didn't work, for some reason
(synth got so confused at one point that on consecutive runs either built some packages, or reported that all is OK.)
So, in the spirit of the old adage 'When in doubt, nuke.', I did just that.

Default configuration, `synth prepare-system` did its thing, all good.
And I think I know why `synth upgrade-system` doesn't reinstall the freshly built packages.

If I am understanding pkg.conf(5) correctly,  because of the 
	
	



```
CONSERVATIVE_UPGRADE
```
pkg() prefers to use the repository from which the packages were originally installed (upstream FreeBSD, in my case), and since
I am trying to reinstall and not update, it just does what its supposed to do when this option is set.

What I do not understand is how and when is the repository priority evaluated?
Just on the package update (when the package in Synth repo has higher port version/revision number than upstream),
or if the repo has higher priority than the official one, then it is used to update _and_ reinstall packages, provided it has the package and
all its dependencies)?
(Also, how high does the priority number go?pkg.conf(5) just says "Higher values are preffered, default is 0.")

So, Synth helpfully built all the packages I needed, now to figure out how to actually install them, preferably without
breaking something 

Off-topic: marino@, I very much enjoyed your interview on BSDNow, particularly the story about Ada.
Question, if I may: what does 
	
	



```
Building pkg
```
 when `synth prepare-system` is run actually do?
I haven't had the chance to look at the source yet.

I'll stop here with the questions 
(I am new to FreeBSD, so naturally I have a _lots _of them).


----------



## marino (Mar 5, 2016)

Pedja said:


> If I am understanding pkg.conf(5) correctly,  because of the CONSERVATIVE_UPGRADE
> pkg() prefers to use the repository from which the packages were originally installed (upstream FreeBSD, in my case), and since
> I am trying to reinstall and not update, it just does what its supposed to do when this option is set.



Normally conservative upgrade is only an issue if you run `pkg upgrade` manually.  internally synth tells package to use the synth repository exclusively for upgrades.  But I'm not sure I agree with option's default, you should probably disable it.



> What I do not understand is how and when is the repository priority evaluated?
> Just on the package update (when the package in Synth repo has higher port version/revision number than upstream),
> or if the repo has higher priority than the official one, then it is used to update _and_ reinstall packages, provided it has the package and
> all its dependencies)?
> (Also, how high does the priority number go?pkg.conf(5) just says "Higher values are prefered, default is 0.")



As far as I can tell, "priority" doesn't work.  I think it's a decoy (aka bug).  The only way I can get a repository to be taken preferentially is to define the conf file lower alphabetically.  this is why the conf file is named 00_synth.conf, which should come before any others.



> So, Synth helpfully built all the packages I needed, now to figure out how to actually install them, preferably without
> breaking something



method 1: `synth upgrade-system`
method 2: (assuming 00_synth.conf already created) `pkg upgrade -r Synth` (or something similar -- -r limits to synth repo)



> Off-topic: marino@, I very much enjoyed your interview on BSDNow, particularly the story about Ada.
> Question, if I may: what does
> 
> 
> ...



Thanks.  `synth prepare-system` gets pkg(8) to return a list of all installed ports, and pipes that list into synth (think of it as a list of port origins), it incrementally builds what is necessary, and when it's done, it rebuilds the repository.

The idea is that somebody would run `pkg upgrade -r Synth` manually so they can be present during the sytem upgrade (or it's building a repo for clients other than the localhost).  Basically it constructs the repo without changing the localhost.


----------



## marino (Mar 5, 2016)

Oh, you mean why is it building pkg(8)?   pkg(8) is needed inside the builders for all sorts of reasons.  It's the first package built, always.  Poudriere does the same thing.


----------



## Pedja (Mar 17, 2016)

marino@ said:


> <snip>
> method 1: `synth upgrade-system`
> method 2: (assuming 00_synth.conf already created) `pkg upgrade -r Synth` (or something similar -- -r limits to synth repo)
> <snip>


`pkg upgrade -fr Synth` worked.
I was a bit concerned because it downgraded a few packages, but
when I ran `pkg update` after, those packages were upgraded, too.

So, all is OK now (famous last words...)

Thanks for such a prompt reply, btw


----------



## free-and-bsd (Mar 18, 2016)

marino@ said:


> ... If you use ccache, it will rebuild huge ports in minutes.


 Doesn't seem to effect www/webkit2-gtk3 this way. It's obviously just rebuilding the port due to some modifications in some other ports, yet the build is taking the same huge amount of time it did originally (already 1hr 30 minutes have passed, the same slow pace).

Are there any other ccache settings to take into account that would really speed up the build process? My ccache size limit is set to be 20G.


----------



## marino (Mar 18, 2016)

free-and-bsd@ 
I saw webkit2-gtk3 take 15 minutes to build yesterday, with ccache.

The most likely explaination is that you haven't configured synth for ccache correctly. How *exactly* (be detailed) did you do it?


----------



## free-and-bsd (Mar 18, 2016)

marino@ said:


> free-and-bsd@
> I saw webkit2-gtk3 take 15 minutes to build yesterday, with ccache.
> 
> The most likely explaination is that you haven't configured synth for ccache correctly. How *exactly* (be detailed) did you do it?


Thanks for prompt reply!
I did as you suggested earlier: in `synth configure` I inserted the path to ccache dir where synth wanted it. It's running now, so I can't show configuration.
As for ccache itself, `sudo ccache -p` shows:

```
(default) base_dir =
(default) cache_dir = /root/.ccache
(default) cache_dir_levels = 2
(default) compiler =
(default) compiler_check = mtime
(default) compression = false
(default) compression_level = 6
(default) cpp_extension =
(default) direct_mode = true
(default) disable = false
(default) extra_files_to_hash =
(default) hard_link = false
(default) hash_dir = false
(default) log_file =
(default) max_files = 0
(/root/.ccache/ccache.conf) max_size = 20.0G
(default) path =
(default) prefix_command =
(default) read_only = false
(default) read_only_direct = false
(default) recache = false
(default) run_second_cpp = true
(default) sloppiness =
(default) stats = true
(default) temporary_dir =
(default) umask =
(default) unify = false
```
 So I typed in /root/.ccache as ccache dir for synth.


----------



## marino (Mar 18, 2016)

With root on another tty, type `ccache -s`
You should see the statistics change as synth builds (run command repeatedly).  If not, it's not working. 

Remember you get the time savings on the second run. the first build is normal time.


----------



## scottro (Mar 18, 2016)

Firstly, thanks for synth.  One thing I haven't seen. (I haven't read every post in the thread, but I've looked at several, and tried to find this in the man page with no luck).

With portmaster, if I wanted to add an option at build time--such as with dwm, the window manager, I would run

`portmaster -m 'DWM_CONF=/usr/home/scott/dwm/conf.h install x11-wm/dwm`.  

Is there something similar for the -m to add options in synth that I've overlooked?


----------



## free-and-bsd (Mar 18, 2016)

marino@ said:


> with root on another tty, type `ccache -s`
> You should see the statistics change as synth builds (run command repeatedly).  If not, it's not working.
> 
> Remember you get the time savings on the second run. the first build is normal time.


Yep, it _is_ changing. Well maybe you're right and I set ccache after the first build of webkit2-gtk3 took place. I remember starting and stopping synth multiple times before the original repo got finished.
I'll check this once I'm finished with this rebuild.


----------



## marino (Mar 18, 2016)

scottro said:


> Is there something similar for the -m to add options in synth that I've overlooked?


looking up in the portmaster man page:

```
-m    arguments for make
```

The answer is no, synth will not let you do this (frankly that looks dangerous to me).
Your only option for customizaton beyond configuring port options is to use a [profile]-make.conf

obviously you can set 
	
	



```
DWM_CONF=/usr/home/scott/dwm/conf.h
```
 in /usr/local/etc/synth/LiveSystem-make.conf and x11-wm/dwm will use it, but so will every other port.


----------



## marino (Mar 18, 2016)

free-and-bsd said:


> Yep, it _i s_ changing. Well maybe you're right and I set ccache after the first build of webkit2-gtk3 took place. I remember starting and stopping synth multiple times before the original repo got finished..



webkit2-gtk3 is quirky.  It seems to forgot the cached bits --- almost like something invalidates the whole thing.  I know I had built it under ccache, and yesterday I built it 3 days.  The first time was 2.5 hours, as if it had not been cached.   I am pretty sure it was, but it didn't work.  The next run it did though.  Maybe something in the source code (header change) invalidated the whole thing.


----------



## free-and-bsd (Mar 18, 2016)

marino@ said:


> ...I know I had built it under ccache, and yesterday I built it 3 days....


 You mean 3 hours, of course :O ??


----------



## marino (Mar 18, 2016)

No, I meant 3 *times*.


----------



## free-and-bsd (Mar 18, 2016)

Ok ok, that's much more reassuring lol.


----------



## free-and-bsd (Mar 18, 2016)

Took 3hr 30 min for www/webkit2-gtk3. On the other hand, other apps got built very quickly, for example gimp took 3mins.


----------



## scottro (Mar 18, 2016)

Thanks for the quick answer. It's not a problem, I can simply do that one package with the standard cd into the port directory and running the make option there.


----------



## free-and-bsd (Mar 18, 2016)

Hm, editors/libreoffice failed to build. It's version 5.0.5_2.


----------



## marino (Mar 18, 2016)

look at the build log that been saved to disk.
ports do fail through no fault of a builder.  It's pretty common actually.


----------



## free-and-bsd (Mar 18, 2016)

marino@ said:


> look at the build log that been saved to disk.
> ports do fail through no fault of a builder.  It's pretty common actually.


Yes, I know this happens. I've checked the log, it says:

```
...
[build LNK] CppunitTest/libtest_svtools_graphic.so
[build CUT] sw_uwriter
[build LNK] CppunitTest/libtest_sw_htmlexport.so
Abort trap (core dumped)
OK (1)
Fatal error 'mutex is on list' at line 409 in file /usr/src/lib/libthr/thread/thr_mutex.c (errno = 0)
```
 The build output of libreoffice is not very enlightening, though ...


----------



## free-and-bsd (Mar 18, 2016)

Well, the build succeeded on the second run. Took much less time, too.


----------



## marino (Mar 18, 2016)

I've seen that before myself.  libreoffice test units are pretty fragile.


----------



## free-and-bsd (Mar 18, 2016)

marino@ said:


> No.  Just add "VALID_CATEGORIES=mycategory" to /etc/make.conf (or the synth profile version of make.conf)


This only seems to work with the synth version of make.conf, that is /usr/local/etc/synth/LiveSystem-make.conf, as per man page.


----------



## free-and-bsd (Mar 25, 2016)

Dear marino@, I thank you once again for all your help and assistance in using this great app of yours and related things.

There's one question related to the idea of repositories I can't decide yet. How can one use a synth created repo for a machine with a different video configuration? Because on my synth machine I have nvidia card, so the created repo is built for this particular hardware. But then I have a laptop with Intel HD video, where most of those packages will work except those nvidia related.

How is this usually handled? Should I run synth on my laptop and make it use the existing repository, so it would only have to rebuild the video-related software? My guess is, if I copy the .ccache dir to my laptop, it might speed up the build considerably  ? Finally, I can insert the laptop's HDD into my synth machine, copy the .ccache dir over to it, boot from it and run synth locally in a normal way (yes, and also copy the repository there, too).

Or is it possible to build such packages on my synth machine, which just has good enough specs for repository building? Something like a separate repo, I should expect?

Thanks again for your king help


----------



## marino (Mar 25, 2016)

free-and-bsd said:


> There's one question related to the idea of repositories I can't decide yet. How can one use a synth created repo for a machine with a different video configuration? Because on my synth machine I have nvidia card, so the created repo is built for this particular hardware. But then I have a laptop with Intel HD video, where most of those packages will work except those nvidia related.



You making a mental relationship between what is installed on the synth system and what you want in the repository.  This is your issue.

For people with exactly one computer, and they use synth to keep that computer up to date, the above relationship is valid.

However, for people with multiple computers that use synth on one computer to keep all of them up to date, the relationship is invalid.  In this case, you would maintain a combined list of packages to populate the repository.

e.g.
on machine A: `pkg query -e  '%a=0' '%o'> prime.list.A`
on machine B: `pkg query -e  '%a=0' '%o' > prime.list.B`
on machine C: `pkg query -e  '%a=0' '%o' > prime.list.C`

on synth machine after transferring the above files: `cat prime.list.[ABC] | sort -u > repo.contents`
`synth just-build repo.contents`
`synth rebuild-repository`

You'd need to manually set up pkg repo conf files for Machine A, B, and C that point to the newly created repository.

The above assumes machine A, B, and C are all the same architecture, thus only one repository is needed

edited: `pkg prime-list` doesn't show origins, so the raw query is needed


----------



## free-and-bsd (Mar 25, 2016)

Thanks a lot! I just though there must be some way to be machine-independent with repo building.


----------



## PacketMan (Mar 26, 2016)

marino@ said:


> `synth just-build repo.contents`
> `synth rebuild-repository`



I'm not sure that works that well. As you know I am running synth on multiple machines.  However one machine is a designated synth builder. There are no other ports installed.  I run `synth status` on the target machine, and take its output and feed it to the designated builder. After the building is done, I copy the packages over to the target machine and run `synth upgrade-system`.  What happens is a whole bunch of dependency checks fail, so synth ends up rebuilding them anyway where they get installed afterwards. (My guess is in a key port or two I have changed an option.)  Back to free-and-bsd's post, where he is running different video cards, my guess is he may have issues there, and will have to run synth locally.  I personally am quite okay with this.  Yesterday my designated builder rebuilt 300+ ports for my GNOME3 machine in a few hours.  And now on my old cpu GNOME3 machine synth only has to rebuild 75 ports to fix up the dependency boo boos.

I've been meaning to experiment with synth profiles for just this sort of scenario but haven't had the time yet.


----------



## marino (Mar 26, 2016)

PacketMan said:


> I'm not sure that works that well.



Unless free-and-bsd wants the same port with different options on different machines, it works wonderfully.  This is the exact concept of operations: One repository, many clients.  The `upgrade-system` command is only a convenience command for the common case where the sysadmin wants to create a repository based one what's installed on the synth machine.  It, and it's companion `prepare-system` command are not necessary and can be omitted.  They are just there to create ports lists quickly (pkg(8) generates the list based on what is installed).  You can just maintain the same list manually.


----------



## garry (Mar 26, 2016)

marino@ said:


> ...`upgrade-system` command is only a convenience command for the common case where the sysadmin wants to create a repository based one what's installed on the synth machine.  It, and it's companion `prepare-system` command are not necessary and can be omitted.  They are just there to create ports lists quickly (pkg(8) generates the list based on what is installed).  You can just maintain the same list manually.



Just to elaborate:

On my machine root has a shell script called `printworld`:

```
#!/bin/sh
pkg query -e '%a = 0' %o | sort -d
```
and the routine for updating the local machine via synth is

```
./printworld >world
synth just-build world
synth rebuild-repository
pkg upgrade
```


----------



## marino (Mar 26, 2016)

That one assumes there's only one repo, "Synth".  Slightly better, the last command would be
`pkg upgrade -r Synth`, just to be explicit and avoid the possibility of upgrading from FreeBSD repository.


----------



## ANOKNUSA (Mar 26, 2016)

garry: Just thought I'd chime in and mention that you can create an alias for that in /usr/local/etc/pkg.conf. No need for a shell script.  There's already a similar alias called "leaf" available, but it shows *[package]-[version]*, not *origin/port*.


----------



## free-and-bsd (Mar 27, 2016)

PacketMan said:


> ...Back to free-and-bsd's post, where he is running different video cards, my guess is he may have issues there, and will have to run synth locally...


Logically, big repo building machines don't ALL have to have all sorts of video-cards etc to maintain repo(s) suited to different video-drivers. The difference being, for example, in some packages being build differently depending on whether you have a nVidia provided GL or OpenGL.
You know, if synth (or poudriere for that matter) rebuilds ~200 packages over a _slight _possibility of changes in relation to _actual_ changes in less than 30 of them, as was discussed earlier in this thread, then why should I be afraid that it would overlook such _evident_ change of dependencies in video-driver and GL related stuff?

So, thanks for your concern, but it worked perfectly well for me. I built the repo on one machine, then uploaded it to my laptop and pkg(ng) upgraded. And pkgng knows perfectly well when dependency changes happen and doesn't miss the opportunity to complain whenever there is any slightest problem of this sort. I've had that multiple times when upgrading with portmaster created packages.


----------



## mirco (Apr 4, 2016)

Hi marino@,

I have a non-working ports-mgmt/synth, version 1.33_1, on FreeBSD 10.3-RELEASE #0.

I was wondering, since after I created the profile, the  /usr/local/etc/synth/[profile-name]-make.conf file didn't exist.  But it doesn't matter if the file is empty or configured, the debugging output stays the same.  Also the /usr/local/etc/synth/LiveSystem-make.conf file is not there.


```
# synth status
Builder mounts detected; attempting to remove them automatically ...
Dismounting successful!
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Illegal instruction (core dumped)
```


```
# gdb synth
...
This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
(gdb) core synth.core
Core was generated by `synth'.
Program terminated with signal 4, Illegal instruction.
Reading symbols from /usr/local/lib/libncurses.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libncurses.so.6
Reading symbols from /usr/local/lib/libtinfo.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libtinfo.so.6
Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000000462369 in ?? ()
[New Thread 801806400 (LWP 101383/<unknown>)]
(gdb) bt
#0  0x0000000000462369 in ?? ()
#1  0x0000000000000060 in ?? ()
#2  0x000000000047a6bc in ?? ()
#3  0x3470bb414db74db7 in ?? ()
#4  0xdffff010ba490048 in ?? ()
#5  0x90e3ff4900007fff in ?? ()
#6  0x05f0bb41004623b9 in ?? ()
#7  0xdffff010ba490047 in ?? ()
#8  0x90e3ff4900007fff in ?? ()
#9  0x00007fffdffff130 in ?? ()
#10 0x00007fffdffff060 in ?? ()
#11 0x0000000000004db7 in ?? ()
#12 0x00000000004834e7 in ?? ()
#13 0x0000000000000000 in ?? ()
(gdb)
```

Since I am _kind _of a Luser I don't really know what's going on,
nor how to solve this. But I am good in following instructions
and eager to learn.

EDIT:

Using ports-mgmt/pkg works, but I get following info:


```
pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
repository Synth has no meta file, using default settings
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Unable to update repository Synth
```


----------



## marino (Apr 4, 2016)

how did you obtain synth?
A) you built it from ports
B) you got it from freebsd official packages (if so, where packages built on 10.2 or 10.3?)

I am guessing you didn't built it and there's some kind of incompatibility between your new 10.3 system and whatever built the synth package.


----------



## mirco (Apr 4, 2016)

marino@ said:


> how did you obtain synth?
> A) you built it from ports
> B) you got it from freebsd official packages (if so, where packages built on 10.2 or 10.3?)
> 
> I am guessing you didn't built it and there's some kind of incompatibility between your new 10.3 system and whatever built the synth package.



It's a fresh 10.3, no upgrade. I wanted to get the laptop being ready quickly, so I used `pkg` and the official packages to install everything.

Then I decided to try `synth upgrade-system`.  But synth was not available from the freebsd official packages.  So I did `portsnap fetch extract` and build it with `make`. All dependencies for synth, all but gcc6, were installed using official packages. gcc and synth were build using make.

Upgrade-system first worked, but then, after having build some ports, I gave it a graceful shutdown, since I wanted to let it finish over night. When trying to continue upgrade-system later on, above error message came up.


----------



## marino (Apr 4, 2016)

it's weird that synth isn't available in packages.   According to portsmon, it's been built (on 10.1 machine).

I'm shooting in dark here, but you might try:

`synth just-build ports-mgmt/synth`
`pkg delete synth`

`pkg add /path/to/synth/packages/All/synth-133_1.txz`
And see if your synth-built synth works better.


----------



## mirco (Apr 5, 2016)

marino@ said:


> it's weird that synth isn't available in packages.   According to portsmon, it's been built (on 10.1 machine).


Now it's available to me also. I don't know what went wrong, but I am embarrassed. My mistake, maybe.



marino@ said:


> I'm shooting in dark here, but you might try:
> 
> `synth just-build ports-mgmt/synth`
> `pkg delete synth`
> ...


Sorry, no good news. Same same, none different.
`synth status` and `synth upgrade-system` are still aborting with message "Illegal instruction".

Worth mentioning is, that `synth just-build`ing of synth did hang up(?) after finishing gcc.  I recognized it, because the load went down.  Graceful shutdown (Ctrl-Q) was not possible, so I Ctrl-C'd and started again: now it built synth and put the txz into place.  From the logs in /var/log/synth I learned, that the built of gcc6-aux actually was fine.  It just didn't went any further.


----------



## marino (Apr 5, 2016)

i won't be able to comment until I upgrade my system to 10.3 and try using synth.


----------



## free-and-bsd (Apr 5, 2016)

mirco said:


> ...
> I was wondering, since after I created the profile, the  /usr/local/etc/synth/[profile-name]-make.conf file didn't exist.  But it doesn't matter if the file is empty or configured, the debugging output stays the same.  Also the /usr/local/etc/synth/LiveSystem-make.conf file is not there....


 Neither was it there for me until I created it. Running `synth configure` doesn't seem to create that file.

Hey, mirco, you can at least do this as root:

```
cd /usr/ports/ports-mgmt/synth
make reinstall clean (that assuming you have synth installed, else you `make install clean`)
```

Then try the resulting synth. My idea is, if something's wrong with your system, it may show up during the port building process. If nothing comes up and it builds fine, then see how it works.
I have had this on my system that packages didn't work, but ports built from sources did.
Anyway, don't worry: whatever is wrong it will be found out.

Oh, yes: debugging is no use unless the app is built with debugging symbols. Which by default they are not.


----------



## free-and-bsd (Apr 5, 2016)

BTW, what kind of hardware do you have? _Not_ an old machine one is about to trash but then decides to install FreeBSD instead? Just asking, you know .


----------



## kpa (Apr 5, 2016)

I have an upgraded 10.3 and Synth works just fine on it.


```
% uname -a
FreeBSD freebsd10 10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297391: Tue Mar 29 19:52:26 EEST 2016     root@freebsd10:/usr/obj/usr/src/sys/GENERIC  amd64
```


```
% pkg -v
1.7.1
```


```
% synth version 

====================================================================
  Custom package repository builder for FreeBSD and DragonFly 1.33
====================================================================
               Copyright (C) 2015-2016 John R. Marino
```


----------



## mirco (Apr 5, 2016)

kpa said:


> I have an upgraded 10.3 and Synth works just fine on it.


Yes, I don't think it's an 10.3 issue.


----------



## mirco (Apr 5, 2016)

free-and-bsd said:


> BTW, what kind of hardware do you have? _Not_ an old machine one is about to trash but then decides to install FreeBSD instead? Just asking, you know .


 No, not an too old machine. 2012 lenovo notebook Z560.


----------



## marino (Apr 5, 2016)

You shouldn't exclude 10.3 being the issue.  Nobody has noted anything like this on 10.2.


----------



## marino (Apr 6, 2016)

To follow up: I upgraded my 10.1 amd64 system to 10.3-RELEASE.

Synth is working perfectly.  No illegal instructions; everything looks nominal.


----------



## scottro (Apr 6, 2016)

I've been using synth on 10.3 for a few days now without issues.  The one complaint I have left is that I liked how portmaster, if you didn't use the -y flag, would stop before proceeding and show you what it would do.  I know we can do this with synth status and don't know how many others would like that as default. (For example, _I_ like the fact that synth expects you to do all configuration before hand--that is, unlike portmaster, you don't get configuration dialogs by default, but others seem to prefer the portmaster way.)


----------



## free-and-bsd (Apr 6, 2016)

I've been using it on 11.0-CURRENT from the start, everything's going fine.


----------



## kpa (Apr 6, 2016)

scottro said:


> I've been using synth on 10.3 for a few days now without issues.  The one complaint I have left is that I liked how portmaster, if you didn't use the -y flag, would stop before proceeding and show you what it would do.  I know we can do this with synth status and don't know how many others would like that as default. (For example, _I_ like the fact that synth expects you to do all configuration before hand--that is, unlike portmaster, you don't get configuration dialogs by default, but others seem to prefer the portmaster way.)



It's not exactly ports-mgmt/portmaster's requirement to do configuration while running it, it's more of a consequence of it not setting BATCH by default. You can very well do all configuration beforehand and then just fire up portmaster(8) with (I hope I remember this right) -m -DBATCH and you won't see a single options dialog.


----------



## scottro (Apr 6, 2016)

kpa thanks, I vaguely remember that now that you mention it, but at the time, whenever that time was, decided against it because there were a few ports that I wanted to configure.


----------



## free-and-bsd (Apr 7, 2016)

This isn't exactly the right place to ask about package dependencies, but I'll put it the way it will concern synth functionality.

Now I have checked all of the installed ports on my system for the presence of any ports depending on lang/clang36 which used to be installed in the past as some build dependency and it pulled in devel/llvm36 with itself. I have deleted both, none of the currently installed ports depends on either of them, as per `make all-depends-list` command for each port (I wrote a script running this for each installed port).

Still, when I run `synth prepare-system` it starts building devel/llvm36 among other ones. I guess, it has long been replaced with devel/llmv37 which has already been built. So why does it insist on building llvm36? What does it know which I don't?


----------



## kpa (Apr 7, 2016)

It comes from DEFAULT_VERSIONS that are in effect when building anything from scratch in a clean environment unless overridden by make.conf(5) or by some other means. Built packages are not known to the build process and can not influence what gets selected as a dependency, they are just pulled in if the already built package matches a build dependency selected by the ports infrastructure and DEFAULT_VERSIONS.


----------



## free-and-bsd (Apr 7, 2016)

kpa said:


> It comes from DEFAULT_VERSIONS that are in effect when building anything from scratch in a clean environment unless overridden by make.conf(5) or by some other means. Built packages are not known to the build process and can not influence what gets selected as a dependency, they are just pulled in if the already built package matches a build dependency selected by the ports infrastructure and DEFAULT_VERSIONS.


I must have misunderstood your meaning somehow...
I've checked the file bsd.default-versions.mk, and neither clang nor llvm are even there.
And none of the already built|installed packages requires either lang/clang36 or devel/llvm36, this I specifically checked against all installed ports. Which also includes all build dependencies which I have kept.


----------



## kpa (Apr 7, 2016)

Can you show an example of a port that causes devel/llvm36 to be built in Synth?


----------



## kpa (Apr 7, 2016)

Oh ok, I forgot something that affects the build just like DEFAULT_VERSIONS does. Many of the ports are using USES += compiler (/usr/ports/Mk/Uses/compiler.mk) in their Makefile and the selection of compiler to lang/clang36 comes from that.


----------



## free-and-bsd (Apr 7, 2016)

kpa said:


> Oh ok, I forgot something that affects the build just like DEFAULT_VERSIONS does. Many of the ports are using USES += compiler (/usr/ports/Mk/Uses/compiler.mk) in their Makefile and the selection of compiler to lang/clang36 comes from that.


Is this not supposed to come out when you `cd /usr/ports/your-port/dir && make build-depends-list` ? And if not that, then `make all-depends-list` must get it at last, no?


----------



## kpa (Apr 7, 2016)

Yes it should show up in the depends list but keep in mind that Synth's build environment may not be the same as the one you have on the host. Compare your /etc/make.conf (not read by Synth!) with /usr/local/etc/synth/LiveSystem-make.conf if you have one.


----------



## free-and-bsd (Apr 7, 2016)

kpa said:


> Yes it should show up in the depends list but keep in mind that Synth's build environment may not be the same as the one you have on the host. Compare your /etc/make.conf (not read by Synth!) with /usr/local/etc/synth/LiveSystem-make.conf if you have one.


They're identical, cause I created them so. 

It finally gave up building devel/llvm36 after N times it tried to and I had to interrupt it ungracefully. Hope I didn't break anything? There was some mentioning of it somewhere in the thread.

And know what? All this experience has encouraged me to look more carefully whether I actually need some other long-builders. Like www/webkit2-gtk2, which last time took unbelievable 5 hours to build! How funny it was to discover, then, that it was only needed because I unwisely checked "google" option for devel/gvfs . Which option I never used, nor was going to use any time in the future LOL, because gvfs is needed for fm/pcmanfm to mount SMB and other suchlike stuff.


----------



## free-and-bsd (Apr 8, 2016)

...In fact, I actually needed neither of the webkits on my system. What a waste of time checking every option available!

Here is another aspect of how useful synth may be: watching the build process more closely you pay more attention to what's being built. Good to know for someone who cares about keeping things simple and uncomplicated.


----------



## marino (Apr 8, 2016)

it must feel nice to purge 10 hours worth of c++ ports from the local repository ...


----------



## free-and-bsd (Apr 8, 2016)

Does not quite tell it! So the more reasons to thank you for your app


----------



## free-and-bsd (Apr 9, 2016)

It is not yet clear to me how to make synth install the built packages into the root defined in the profile as "system root directory". Because `synth upgrade-system` obviously upgrades synth's own system despite the "system root directory" setting, as I learned from my recent experience.


----------



## marino (Apr 9, 2016)

one thing doesn't have anything to do with the other.  The system root is to define the build environment only.

You configure pkg(8) to use the new repository as necessary (e.g. another system connects to repository over a network or webserver).  In that use case, pkg(8) is responsible for package installing.  Synth only does it as a convenience command for the local system to save a step.


----------



## free-and-bsd (Apr 9, 2016)

marino@ said:


> one thing doesn't have anything to do with the other.  The system root is to define the build environment only.
> 
> You configure pkg(8) to use the new repository as necessary (e.g. another system connects to repository over a network or webserver).  In that use case, pkg(8) is responsible for package installing.  Synth only does it as a convenience command for the local system to save a step.


So you mean synth will imitate the things found under "system root" dir while creating its jailed build environment?

Yes, I was lacking clarity on this point . I was thinking it was the "_system_ root" as mentioned in `synth upgrade-[I]system[/I]`.


----------



## free-and-bsd (Apr 9, 2016)

There are also these words in the beginning of the man page:

```
synth will build packages in a clean environment can exactly mirror the system that
  they are built on...
```
 This phrase is somewhat incomplete and needs correction. But apart from that, the word "can" sort of makes one think this is not the default behaviour "in a clean environment". But in reality this is what it _will_ do if no list of packages to build is supplied (via commands that accept such option). For example, `synth status` (no pkg list given) will show the status of the build that will "exactly mirror" the _current_ system. Then this behaviour can be called "default", right?

So maybe it would be clearer in the form like "...synth _will_ exactly mirror the system that [the packages] are built on, unless ..." if that is what it will do anyway? Just to make it clear from the start, you know. 

In public speaking it is a well known rule that the audience will remember most clearly what's mentioned _at the start_ and in the _end_. That is why, while the introduction is always short, it must be precise in its terminology.


----------



## marino (Apr 9, 2016)

It's making a distinction between how poudriere does it and how synth does it.
Poudriere creates a reproducible clean environment.   If you follow the same steps on 3 different machines (same arch) you'll get identical jails.

For some people, this is a problem because they have modified their host system and they want to build package for that exact system.  By using a system root of "/" you get an exact mirror.

However, you can also dump another system (e.g. FreeBSD 9.3-RELEASE) in another location and set the profile's system root to that.  Then the clean environment is identical to what poudriere will do.

so it is "can", not "will".  The quoted text above is accurate.  The default is "will", yes, because the default system root is "/", but that can be configured.



> So you mean synth will imitate the things found under "system root" dir while creating its jailed build environment?


Not imitate, mount.  It mounts a selected set of directories like bin, sbin, usr/bin, usr/sbin etc.  Others are created like /etc, /tmp, etc.


----------



## taz (Apr 9, 2016)

I want to use synth, instead of portmaster,  in my automated script that bootstraps a fresh system. 

So I installed the 10.3 release VM switched pkg to use latest and updated the ports tree with portsnap fetch extract. Ran pkg to bootstrap pkg.
At this point pkg is the only thing installed, ports tree is up to date but I do not know what is the correct way to bootstrap synth?

If I install synth with pkg and then run synth status, synth wants to rebuild synth and all of it's dependencies (gcc-aux,...).
If I install synth using ports and then run synth status it also wants to rebuild synth and all of it's dependencies (gcc-aux, ...).

So how do I install synth so that synth dose not want to rebuild it self?

Also how do I use synth in a script? I know I can have multiple profiles but how do I tell synth to use a profile without curses and without first running synth configure? Is there some switch option that can point synth to a profile file?

Thanks.


----------



## marino (Apr 9, 2016)

to bootstrap it, I suggest you download and install the officially built Synth package (using the bootstrapped pkg), then use Synth to rebuild Synth and ncurses and reinstall them.

See the SYNTHPROFILE environment variable how how to run a specific non-curses profile as a command.
The profile has to pre-exist though.


----------



## tankist02 (Apr 9, 2016)

Clean install of 10.3. When preparing the system with Synth got the following error while building x11/qt5-x11extras:


```
c++ -Wl,--as-needed -fstack-protector -fuse-ld=gold -Wl,--enable-new-dtags -pthread -Wl,-rpath,/usr/local/lib -shared -Wl,-Bsymbolic-functions -Wl,-soname,libQt5X11Extras.so.5 -o libQt5X11Extras.so.5.5.1 .obj/qx11info_x11.o  -L/construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/lib -L/usr/local/lib -lQt5Gui -lQt5Core -lGL
c++: error: invalid linker name in argument '-fuse-ld=gold'
*** [../../lib/libQt5X11Extras.so.5.5.1] Error code 1

make[3]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/src/x11extras
1 error

make[3]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/src/x11extras
*** [sub-x11extras-all] Error code 2

make[2]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/src
1 error

make[2]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1/src
*** [sub-src-all] Error code 2

make[1]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1
1 error

make[1]: stopped in /construction/xports/x11/qt5-x11extras/work/qtx11extras-opensource-src-5.5.1
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make: stopped in /xports/x11/qt5-x11extras
```

It looks that ld.gold is missing. Should I file PR?


----------



## marino (Apr 9, 2016)

to whom would you file the PR?


----------



## tankist02 (Apr 10, 2016)

I don't know - that's why I'm asking. Is it a x11/qt5-x11extras port problem? Is it something wrong with my system? Perhaps I misconfigured synth?


----------



## marino (Apr 10, 2016)

it's a problem with the port.  The tree should not be considered infallible.

ld.gold is the gold linker.  The gold linker is not installed in FreeBSD base, which means it would have to come from an installed devel/binutils.

For one, I question why fuse-ld.gold is even being invoked.  If the port only exclusively builds with the gold linker, then x11/qt5-x11extras is missing a dependency.  If it doesn't, then qt5-x11extras is misconfiguring itself.  Either way, it's a problem with the port.


----------



## PacketMan (Apr 10, 2016)

tankist02 said:


> Clean install of 10.3. When preparing the system with Synth got the following error while building x11/qt5-x11extras:
> 
> 
> ```
> ...



I just run `synth upgrade-system` a 2nd time and that fixed it. I see that every now and then when something fails in synth and a 2nd run of synth it runs through with green lights all the way. For this particular case I can't remember if I ran `portsnap fetch update` a 2nd time a day later or not. I don't think I did though.


----------



## tankist02 (Apr 12, 2016)

I ran `synth prepare-system` one more time, second time qt5-x11extras was successfully fetched instead of built, probably because its official package was updated:


```
root@bclinton:~# synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
vlc-2.2.1_8,4.txz VDPAU is OFF but port says it must be ON
The following packages will be fetched:

New packages to be FETCHED:
   qt5-x11extras-5.5.1 (0.04% of 37 MiB: 17 KiB)
   nvidia-driver-346.96 (99.96% of 37 MiB: 37 MiB)

The process will require 37 MiB more space.
37 MiB to be downloaded.
Fetching qt5-x11extras-5.5.1.txz: 100%  17 KiB  17.3kB/s  00:01  
Fetching nvidia-driver-346.96.txz: 100%  37 MiB  4.3MB/s  00:09  
00:00:33 => [01] Builder launched
00:00:33 => [02] Builder launched
00:00:33 => [02]  Shutting down
00:02:30 => [01] 00:01:57 Success multimedia/vlc
00:02:31 => [01]  Shutting down



The task is complete.  Final tally:
Initial queue size: 1
  packages built: 1
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:02:30
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 864 ports serially.
Scanning existing packages.
No packages are required to be fetched.
Integrity check was successful.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
```

I did not update ports tree between first and second runs.


----------



## scottro (Apr 14, 2016)

What I'm finding a bit disturbing (while freely admitting I may be overlooking something that everyone but me knows) is that even when I ran `synth status graphics/ImageMagick` it didn't indicate that it was going to remove the chrome browser.  As I knew the build would take awhile, I didn't supervise it after that, only to find the results printed--that it had installed this, upgraded that, and removed something else.  I think, (I've seen other mention of this, but again, don't know their circumstances, and am open to the fact that they and myself have missed something), that it would be preferable to do something like portmaster does, that unless there is a -y flag (or similar), show what it will do before hitting yes or no.   

(I am ALMOST sure that synth status didn't show it was going to remove chrome, but not completely sure.)


----------



## marino (Apr 14, 2016)

are you using `synth upgrade-system`?  if so, until you are comfortable, only use `synth prepare-system`.  
People tend to blame Synth for things that pkg(8) does.  
It is also important to note *when* chromium was removed.  It might have been removed during the repository build.  In theory, it should be shown during "status" when that happens.


----------



## scottro (Apr 14, 2016)

In this case, I just did `synth install graphics/ImageMagick` (which was not previously installed.)  Before doing so I ran the synth status command mentioned before.  Using pkg would also first say, going to install this, remove that before proceeding.  I might be misunderstanding, but I thought the status option should show me that will remove something. I'm sorry I didn't note when it was removed.  I started the command in another workspace then ignored it till later.  (Again, I'm not positive that the status option didn't warn me.


----------



## marino (Apr 14, 2016)

`synth status graphics/ImageMagick` or `synth status`?

status shows what synth will remove and rebuild.   The `synth install graphics/ImageMagick` command will first build everything (and rebuild anything) and that should match what the status command said would happen.  Then, the *entire* set of packages will be scanned and validated before rebuilding the repository.  It's here that packages can be removed, but these should be packages that were NOT on the status list before.

If you rebuild everything, then you shouldn't see any removals, but it can happen if the port makefile has misleading dependency information.  (rare but it happens). 

But yes, I can see it removing the unrelated chromium package with an *install* command during repository building.  It's because chromium needs rebuilding.  It's not really a problem because chromium is still installed in your system (I assume).


----------



## kpa (Apr 14, 2016)

If you encounter such strange removal requests from pkg you should first run `# pkg update -f` to see if that clears it.


----------



## xtaz (Apr 14, 2016)

Reading these last few posts has made me think, I've always thought it would be nice if there was a switch or environment variable that I could set which would make `synth upgrade-system` behave as if I had run `synth prepare-system && pkg upgrade -r Synth` where pkg displays what it is going to actually do and then prompts you to answer y or n depending on if you like what you see. I assume synth just passes -y to the pkg upgrade command. Obviously if I want to do that I can use the second set of commands that I mentioned, but sometimes it's nice to just type less stuff!


----------



## mirco (Apr 14, 2016)

marino@ said:


> To follow up: I upgraded my 10.1 amd64 system to 10.3-RELEASE.
> 
> Synth is working perfectly.  No illegal instructions; everything looks nominal.



I played around with with `synth force [already-installed-port]` to have a look what works and what doesn't. Those are only a few examples.

For instance, everything went fine with textproc/docbook, www/owncloud, www/elinks, www/links, multimedia/gstreamer

"Illegal instructions" abortion happend with deskutils/owncloudclient, net-p2p/transmission, www/w3m, www/firefox, www/chromium, x11-wm/xfce4, editors/libreoffice,

With x11/xorg: all packages built, but did not went further, hung up(?), no requesting permission to rebuild the repository, but all packages built are available in .../live_packages/All

I wished, that thing just worked like it did before.


----------



## scottro (Apr 14, 2016)

marino@, no, it removed Chrome. From what you say though, I probably just missed seeing it in status, which seems the most likely scenario.


----------



## marino (Apr 15, 2016)

mirco said:


> "Illegal instructions" abortion happend with deskutils/owncloudclient, net-p2p/transmission, www/w3m, www/firefox, www/chromium, x11-wm/xfce4, editors/libreoffice.
> I wished, that thing just worked like it did before.



So far all evidence is pointing that this is something specific to your computer.  Nobody else is reporting this.  Wishing it "worked like it did before" implies there's a problem with Synth but I don't see what I can do if nobody else can reproduce it, including me.


----------



## marino (Apr 15, 2016)

scottro said:


> marino@, no, it removed Chrome. From what you say though, I probably just missed seeing it in status, which seems the most likely scenario.



I don't think you understood me completely.  I gave you 2 places where removal could have happened, and suggested it happened in the second place.  Saying "it removed Chrome" doesn't tell me anything because you don't indicate if it happened as a result of building or if it happened as a result of reconstructed the repository (the far more likely case).  I explained how `synth install <single port>` could do that, especially on the ports trunk branch.  It's much less likely to occur on quarterly branches.

By the way, the quarterly branch is pretty close to the trunk now.  I couldn't recommend Q1, but Q2 might be the best option for some people.


----------



## scottro (Apr 15, 2016)

Sorry if I wasn't clear. I 'm just making guesses, as I didn't observe the build, just started it with the command to install, then only looked again once it was complete.   Yes, you did explain and I made what was probably an incorrect assumption.


----------



## marino (Apr 15, 2016)

the other thing that is very unclear to me is what you mean by "removed chrome".  I've been working under the idea you mean the chrome package in the synth repository was deleted (which is relatively normal).  If you actually mean that pkg(8) uninstalled chrome without reinstalling it, that's a horse of a different color (and almost certainly a bug in pkg(8))


----------



## ANOKNUSA (Apr 15, 2016)

xtaz said:


> Reading these last few posts has made me think, I've always thought it would be nice if there was a switch or environment variable that I could set which would make `synth upgrade-system` behave as if I had run `synth prepare-system && pkg upgrade -r Synth` where pkg displays what it is going to actually do and then prompts you to answer y or n depending on if you like what you see.



The manual explicitly states that the intended purpose of `synth upgrade-system` (and `synth install` for that matter) is the exact opposite of what you want. And what you want obviously already exists. Pointless duplication is pointless.


----------



## free-and-bsd (Apr 18, 2016)

This has been discussed in one of the threads, but not with relation to synth.

So here's my question: x11-toolkits/linux-c6-qt47-x11 can be built with nvidia libGL, but will insist on installing x11-drivers/nvidia-driver, and not nvidia-driver-304* or other legacy versions with linux compatibility support. Which is not the way to go on a system with "legacy" cards, evidently.

Reportedly, the manual edit of the x11-toolkits/linux-c6-qt47-x11's Makefile solves this -- for a _manual_ build using portmaster, for example. 
But will synth recognize such edited port without ignoring it, as seems to be its custom in some similar cases? Because this edit of Makefile is beyond what is available via configurable options.

I just have to build for a machine with a legacy nvidia card, so these few packages present a problem, because synth follows the Makefiles, which upgrades nvidia-driver to the latest version. Meaning no video after upgrade and manual recompile to get it back.


----------



## scottro (Apr 18, 2016)

marino@ sorry to take so long to answer, especially when you're so quick to answer questions.  Yes, I mean that after I ran the command synth install graphics/ImageMagick the www/chromium package was removed and not reinstalled.


----------



## marino (Apr 18, 2016)

free-and-bsd,
Synth can't detect if a port has been manually edited.  It doesn't "ignore" ports because they've been edited.  As long as your manual edit doesn't break the port, Synth will use the modified makefile.

In fact, this is one of the big features of synth: port testing.
If a specific port fails testing, the maintainer/tester modifies it until it passes and then commits/submits the fixes.


----------



## free-and-bsd (Apr 21, 2016)

marino@ said:


> free-and-bsd@,
> Synth can't detect if a port has been manually edited.  It doesn't "ignore" ports because they've been edited.  As long as your manual edit doesn't break the port, Synth will use the modified makefile.
> 
> In fact, this is one of the big features of synth: port testing.
> If a specific port fails testing, the maintainer/tester modifies it until it passes and then commits/submits the fixes.


Yes, thank you. I've noticed this by now and that's great. I just wasn't sure, because vermaden 's sysutils/automount is always ignored, I never knew why. Could it be because the port just consists of a script to be installed, so it doesn't need rebuilding?


----------



## marino (Apr 21, 2016)

No need to guess why, just look at the 00* or 04* log, it will list the reasons for ignore.  (or 03*, can't remember if the ignore log starts with 03_ or 04_)


----------



## free-and-bsd (Apr 21, 2016)

marino@ said:


> No need to guess why, just look at the 00* or 04* log, it will list the reasons for ignore.  (or 03*, can't remember if the ignore log starts with 03_ or 04_)


04_* is skipped ports, and it just says:

```
sysutils/automount by sysutils/fusefs-exfat
```
then 03_* says further:

```
00:00:42 sysutils/fusefs-exfat: License Microsoft-exFAT needs confirmation, but BATCH is defined
```
Now I wonder where I could define otherwise (non-BATCH)? Does it mean to run the port individually?


----------



## marino (Apr 21, 2016)

Just disable the licenses. I don't have a system open but it's something like DISABLE_LICENSES=yes in make.conf.
Alternatively you can accept specific licenses in make.conf.
See Mk/bsd.license* for more details.


----------



## marino (Apr 21, 2016)

make.conf means "[profile]-make.conf" of course.


----------



## free-and-bsd (Apr 21, 2016)

marino@ said:


> make.conf means "[profile]-make.conf" of course.


Oh yeah, that I have got through experience


----------



## STREBLO (Apr 24, 2016)

I have a few questions if you'll indulge me.

Say one wants to build a custom repository using synth, what is the recommended way to connect to it? Setup a webserver I guess? If I wanted to build a custom package repository in one of my servers jails and then connect to it and use it as my pkg repository, is synth the right tool for this?

I noticed you said it is not intended to be running to jail, but is there any reason it isn't recommended? Or is it just not necessary?

I have a small server no one's using, an old Windows Home Server HP470 with 2GB of RAM I was recently given, that I could use where the only thing it would be used for is building ports. Or I could run in my much faster server, a FreeNAS server with 32GB of RAM and a Xeon E3, but it would be run in a jail alongside several other jails I use for other things. Or, would I be better off just running synth on the system I want to update itself? What would you recommend?


----------



## marino (Apr 24, 2016)

STREBLO said:


> Say one wants to build a custom repository using synth, what is the recommended way to connect to it? Setup a webserver I guess?



see pkg.conf(5).  The URL takes several protocols including http://, https://, ftp://, file://, and ssh://.  You have many options.



> If I wanted to build a custom package repository in one of my servers jails and then connect to it and use it as my pkg repository, is synth the right tool for this?



It's capable of doing it.



> I noticed you said it is not intended to be running to jail, but is there any reason it isn't recommended? Or is it just not necessary?



It's not necessary and there's no benefit to do it.  At the time, it didn't work, but on the github/jrmarino/synth README.md, there's a FAQ entry on how to configure the jail so Synth can run inside one.  It's only for people that refuse to run Synth in anything other than a jail for no particular reason.



> I have a small server no one's using, an old Windows Home Server HP470 with 2GB of RAM I was recently given, that I could use where the only thing it would be used for is building ports. Or I could run in my much faster server, a FreeNAS server with 32GB of RAM and a Xeon E3, but it would be run in a jail alongside several other jails I use for other things. Or, would I be better off just running synth on the system I want to update itself? What would you recommend?



Memory is the #1 consideration, the more the better.  Then it's number of cores.  Then it's newness. .
Use the FreeNAS server but be conservative on the number of builders in case the other jails need lots of resources too.

I've never had a problem multitasking on DragonFly but some people with older FreeBSD systems (probably very old?) claim that the system gets unusable while Synth is running.  I assume that's a combination of a taxed system and the performance edge of DragonFly showing as well.


----------



## scottro (Apr 24, 2016)

Using  ZFS, 8GB of RAM but an i3 processor (and that's what compiling taxes), I've not yet run into any issues running synth, even with things like libre office.  This will be while running, generally, a browser, usually Firefox, (but several sessions--this is a multimonitor setup and I might have it in two or three windows, and before adding swap, Firefox or Thunderbird  would sometimes just die, but that was memory), and Thunderbird.  So, I won't say I've really been taxing resources, but anyway, I've run synth in a different tag (dwm's version of workspaces, more or less),  while getting on with my work and had no resource issues at all on FreeBSD-10.3


----------



## marino (Apr 24, 2016)

scottro said:


> Using  ZFS, 8GB of RAM but an i3 processor (and that's what compiling taxes)...



Until you run out of memory and swapping to a mechanic drive kicks in.  Agreed that CPU limits, but only if memory is plentiful with tmpfs option.
8Gb is a good amount though, good for 4x4 (4 builders by 4 jobs) if the swap is an SSD (4x4 will dip into swap with 8GB, but usually less than 5%)


----------



## scottro (Apr 24, 2016)

On the i3 machine, no it's all, including swap, on spinning drives.  And still no trouble with synth.


----------



## PacketMan (Apr 26, 2016)

STREBLO said:


> I have a small server no one's using, an old Windows Home Server HP470 with 2GB of RAM I was recently given, that I could use where the only thing it would be used for is building ports. Or I could run in my much faster server, a FreeNAS server with 32GB of RAM and a Xeon E3, but it would be run in a jail alongside several other jails I use for other things. Or, would I be better off just running synth on the system I want to update itself? What would you recommend?



So do I. See https://forums.freebsd.org/threads/47319/

I would say your HP470, as dear as it is to you and me will be slowwwwwwwww for ports building efforts.  I use it as a miniNAS (running FreeBSD) and I use net-p2p/btsync to sync my files with my main FreeBSD based NAS.  I do have Synth on it, but only for keeping it up to date.  For bigger port building efforts I currently use a Dell PowerEdge 1950 ( see  https://forums.freebsd.org/threads/54413/ )and it works real well with FreeBSD on bare metal.


----------



## STREBLO (Apr 26, 2016)

PacketMan said:


> So do I. See https://forums.freebsd.org/threads/47319/
> 
> I would say your HP470, as dear as it is to you and me will be slowwwwwwwww for ports building efforts.  I use it as a miniNAS (running FreeBSD) and I use net-p2p/btsync to sync my files with my main FreeBSD based NAS.  I do have Synth on it, but only for keeping it up to date.  For bigger port building efforts I currently use a Dell PowerEdge 1950 ( see  https://forums.freebsd.org/threads/54413/ )and it works real well with FreeBSD on bare metal.


When you say slow, how slow exactly do you mean? To compile all the ports I might use on a desktop, what timeframe would I be looking at? 

Also, is it possible to use a combination ports from a private repository and the official repositories? So only compile the parts that aren't available in the official repository. Then tell my computer to look to my private repo, then if its not there fallback to the official repo.


----------



## ANOKNUSA (Apr 27, 2016)

STREBLO said:


> To compile all the ports I might use on a desktop, what timeframe would I be looking at?



Your wording isn't very clear. We don't know what ports you would personally use on a desktop, and it's probably a bad idea to try and build _all_ the ports you _might_ use on a desktop.   But compiling is a very resource-intensive process, particularly on the CPU. If you're using a low-powered machine to build an entire desktop environment and complementary ports, it's not unusual for it to take a full 24 hours or more.



STREBLO said:


> Also, is it possible to use a combination ports from a private repository and the official repositories? So only compile the parts that aren't available in the official repository.



Yes, Synth can do this. It will build only those ports that (a) you customize yourself, and (b) are dependencies of those customized ports which, due to your own customizations, must then be custom-built themselves. For everything else, Synth will simply download the packages from the repo, place them in your local repository, and add them to your local repository's database. See the `synth configure` section of the man page.


----------



## zirias@ (Apr 27, 2016)

ANOKNUSA said:


> But compiling is a very resource-intensive process, particularly on the CPU. If you're using a low-powered machine to build an entire desktop environment and complementary ports, it's not unusual for it to take a full 24 hours or more.


I just built a ports list essentially including kde4, libreoffice and chromium on an older desktop machine using poudriere: 51 hours. This is just for testing, I'll soon have my new server for such things  So yes, it takes a *very* long time. It's probably a bit faster with synth (without the jails overhead, preparing them here, especially without ZFS, takes some time) but I wouldn't expect a huge difference.


----------



## kpa (Apr 27, 2016)

Zirias said:


> I just built a ports list essentially including kde4, libreoffice and chromium on an older desktop machine using poudriere: 51 hours. This is just for testing, I'll soon have my new server for such things  So yes, it takes a *very* long time. It's probably a bit faster with synth (without the jails overhead, preparing them here, especially without ZFS, takes some time) but I wouldn't expect a huge difference.



Jail runtime overhead is totally negligible and synth uses the same kind of techniques anyway,  setting up chroot(8) is about as fast as jail cloning with ZFS and mount_nullfs(5) works the same on both. Where poudriere loses is when you are not using ZFS, the jail cloning becomes an expensive operation because it's done with cpdup(1).


----------



## marino (Apr 27, 2016)

kpa said:


> Jail runtime overhead is totally negligible and synth uses the same kind of techniques anyway,  setting up chroot(8) is about as fast as jail cloning with ZFS and mount_nullfs(5) works the same on both. Where poudriere loses is when you are not using ZFS, the jail cloning becomes an expensive operation because it's done with cpdup(1).


This is actually not true.  Chroots are much, much faster than using jails.  We have anecdotes that using Synth cuts 25% off the time.  Earlier in this thread, someone reported a job that reproducibly takes 24 hours on poudriere only takes 18 hours on synth.  The same exact task.  Others have reported noticibly shorter build times.

Once the jails are up, operations are similar, but the jail starting/stopping, even with persistence, costs significant time.


----------



## talsamon (Apr 27, 2016)

Maybe, Synth is faster with the same number of packages, but this does not matter, cause synth compiles everytimes more packages than poudriere.


----------



## marino (Apr 27, 2016)

talsamon said:


> Maybe, Synth is faster with the same number of packages, but this does not matter, cause synth compiles everytimes more packages than poudriere.


This is a false statement.  Both programs use the exact same logic to determine rebuilding and both are equally aggressive at removing packages for rebuilding.  Your statement is simply incorrect.


----------



## talsamon (Apr 27, 2016)

`Poudriere` does not recompile a package if it does not change, `Synth` does           (e.g webkit-gtk is recompiled every second run).


----------



## marino (Apr 27, 2016)

You are wrong.  If you continue to disagree, please show the code and not anecdotes to prove your point.  Before you start, note that I am very well informed on poudriere code too (I've contributed to the code base).


----------



## talsamon (Apr 27, 2016)

I have not looked in the poudriere code, I had made a quick look in the synth code. But it does not mater, I only know synth had made problems, that poudriere never does. (I have give up to work with synth a month ago, after it deletes 30 packages from the system without reason - someone on pipermail reported synth has deleted 290 packages on his system).


----------



## marino (Apr 27, 2016)

You are making a allegation, proof matters.  Since you have no data backing up your statement, can you just not comment anymore?  I thought you stopped using synth a long time ago.  That "pipermail" post was also full of issues and missing tons of details, so no one can make conclusions (note that nobody else is providing confirming stories or data).  I responded to that person and he never responded back even though 5 people were participating in the conversation.

Bottom line: Your stories are worthless.  it's not like you have two machines identically set up and you make the same exact request to both and compare.  You're just making your own conclusions with limited (and inaccurate) data and assumptions.

I get you don't like Synth -- but others do so just let others answer the questions instead of spreading false information.  Thank you.


----------



## PacketMan (Apr 27, 2016)

STREBLO said:


> When you say slow, how slow exactly do you mean? To compile all the ports I might use on a desktop, what timeframe would I be looking at?



Based on what I know, a x11/gnome3 desktop, not counting any other applications (ports) you have installed I would say take easy 48 hours at least on a HP470, maybe even closer to 72 hours after you toss in a few ports.  Use it for what it was intended (a NAS) and you should love FreeBSD on it. Using it as a builder box and it will take quite some time.


----------



## tankist02 (Apr 29, 2016)

I'm trying to set up a custom repo built with Synth which is available to remove machines. I followed this guide for poudriere: https://www.digitalocean.com/commun...m-to-create-packages-for-your-freebsd-servers. I can see the list of live_packages on the client in a browser. But when I try to use pkg I get the following error:


```
root@bclinton:~# pkg update
Updating bobama repository catalogue...
Repository bobama has a wrong packagesite, need to re-create database
pkg: http:///bobama:8080/live_packages/meta.txz: No address record
repository bobama has no meta file, using default settings
pkg: http:///bobama:8080/live_packages/packagesite.txz: No address record
Unable to update repository bobama
```


/usr/local/etc/nginx/nginx.conf

```
root@bobama:~# cat /usr/local/etc/nginx/nginx.conf
load_module modules/ngx_mail_module.so;
load_module modules/ngx_stream_module.so;

worker_processes  1;

events {
  worker_connections  1024;
}

http {
  include  mime.types;
  default_type  application/octet-stream;

  sendfile  on;
  keepalive_timeout  65;

  server {
  listen  8080;
  server_name  192.168.1.6;

  location / {
  root  /usr/local/www/nginx;
  index  index.html index.htm;
  }

  location /live_packages {
  root  /var/synth/;
  autoindex on;
  }

  error_page  500 502 503 504  /50x.html;
  location = /50x.html {
  root  /usr/local/www/nginx-dist;
  }
  }
}
```

/usr/local/etc/pkg/repos/bobama.conf

```
root@bclinton:~# cat /usr/local/etc/pkg/repos/bobama.conf
bobama: {
  url  : http:///bobama:8080/live_packages,
  mirror_type: "http",
  enabled  : yes,
}
```

All other pkg repos are disabled.


----------



## marino (Apr 29, 2016)

google FreeBSD + "no address record".  I might be something misconfigured in your /etc/resolv.conf  file.  It's not really a synth issue and barely a pkg(8) issue, it's a generic network configuration issue I think.


----------



## tankist02 (Apr 30, 2016)

I searched and didn't find anything relevant. I'm not an expert in networking, but everything else network-related works just fine on my 2 computers, except pkg + custom repo. So I changed sharing of my custom repo from HTTP to NFS + file. Worked without any problem.


----------



## tankist02 (May 3, 2016)

Have a strange failure when building news/pan with option NLS off:


```
synth prepare-system
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
pan-0.140.txz NLS is ON but port says it must be OFF
00:00:17 => [02] Builder launched
00:00:17 => [04] Builder launched
00:00:17 => [01] Builder launched
00:00:17 => [03] Builder launched
00:00:17 => [02]  Shutting down
00:00:17 => [04]  Shutting down
00:00:17 => [03]  Shutting down
piped_mute_command failure:
=> umount: unmount of /usr/obj/synth-live/SL01/construction failed: Device busy

piped_mute_command failure:
=> umount: unmount of /usr/obj/synth-live/SL01/dev failed: Device busy

piped_mute_command failure:
=> umount: unmount of /usr/obj/synth-live/SL01/bin failed: Device busy

piped_mute_command failure:
=> umount: unmount of /usr/obj/synth-live/SL01 failed: Device busy

piped_mute_command failure:
=> rm: /usr/obj/synth-live/SL01/bin/pax: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/echo: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/hostname: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/cp: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/cat: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/pwait: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/[: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/realpath: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/rcp: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/chmod: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/rm: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/chflags: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/sync: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/dd: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/sleep: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/kill: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/chio: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/ln: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/getfacl: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/red: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/setfacl: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/csh: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/pgrep: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/stty: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/unlink: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/kenv: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/rmdir: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/ls: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/ed: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/ps: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/expr: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/date: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/tcsh: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/domainname: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/uuidgen: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/rmail: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/sh: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/pwd: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/pkill: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/mkdir: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/test: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/df: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/mv: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/link: Read-only file system
rm: /usr/obj/synth-live/SL01/bin/freebsd-version: Read-only file system
rm: /usr/obj/synth-live/SL01/bin: Device busy
rm: /usr/obj/synth-live/SL01/dev/reroot: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/fd: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/led: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/usb: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/msdosfs: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/gpt: Operation not supported
rm: /usr/obj/synth-live/SL01/dev/pts: Operation not supported
rm: /usr/obj/synth-live/SL01/dev: Device busy
rm: /usr/obj/synth-live/SL01/construction: Device busy
rm: /usr/obj/synth-live/SL01: Device busy

00:00:50 => [01] 00:00:32 Failure news/pan
00:00:50 => [01]  Shutting down



The task is complete.  Final tally:
Initial queue size: 1
  packages built: 0
  ignored: 0
  skipped: 0
  failed: 1

Duration: 00:00:49
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 879 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
```

/usr/local/etc/synth/synth.ini:

```
; This Synth configuration file is automatically generated
; Take care when hand editing!

[Global Configuration]
profile_selected= LiveSystem

[LiveSystem]
Operating_system= FreeBSD
Directory_packages= /data/live_packages
Directory_repository= /data/live_packages/All
Directory_portsdir= /usr/ports
Directory_options= /var/db/ports
Directory_distfiles= /usr/ports/distfiles
Directory_buildbase= /usr/obj/synth-live
Directory_logs= /var/log/synth
Directory_ccache= /root/.ccache
Directory_system= /
Number_of_builders= 4
Max_jobs_per_builder= 4
Tmpfs_workdir= true
Tmpfs_localbase= true
Display_with_ncurses= false
leverage_prebuilt= true
```

System:

```
uname -a
FreeBSD bobama 10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297264: Fri Mar 25 02:10:02 UTC 2016  root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
```

Added static libs to news/pan to fix startup crash:

```
root@bobama:pan# svn diff
Index: Makefile
===================================================================
--- Makefile   (revision 414425)
+++ Makefile   (working copy)
@@ -21,7 +21,7 @@
USE_GCC=   any
GNU_CONFIGURE=   yes
CPPFLAGS+=   -I${LOCALBASE}/include
-LDFLAGS+=   -L${LOCALBASE}/lib -lgnuregex ${ICONV_LIB}
+LDFLAGS+=   -L${LOCALBASE}/lib -lgnuregex ${ICONV_LIB} -static-libgcc -static-libstdc++
OPTIONS_DEFINE=   GTKSPELL GNUTLS NLS
OPTIONS_RADIO=   GTK
```


----------



## marino (May 3, 2016)

Is it reproducible?  If you reboot your computer and try just to build pan separately (`synth force news/pan`), does it still happen?


----------



## tankist02 (May 4, 2016)

Yes, it happens on 2 different machines. Just to be sure I rebooted one, toggle NLS option ON - success. Toggle it OFF - failure as I described. 

Let me know what else I can provide.


----------



## marino (May 4, 2016)

Hmm, this port doesn't build on DragonFly with NLS off due to broken dependencies.


```
checking for xgettext... no
checking for msgmerge... no
checking for msgfmt... no
checking for gmsgfmt... no
configure: error: GNU gettext tools not found; required for intltool
===>  Script "configure" failed unexpectedly.
Please report the problem to gnome@FreeBSD.org [maintainer] and attach the
"/construction/news/pan/pan-0.140/config.log" including the output of the
failure of your make command. Also, it might be a good idea to provide an
overview of all packages installed on your system (e.g. a
/usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

Stop.
make[1]: stopped in /xports/news/pan
```

what does your synth/logs/news___pan.log show?  You haven't mentioned it yet.


----------



## tankist02 (May 5, 2016)

The port fails at configure phase:


```
===>  pan-0.140 depends on shared library: libgtk-x11-2.0.so - found (/usr/local/lib/libgtk-x11-2.0.so)
===>  Returning to build of pan-0.140
===>  pan-0.140 depends on shared library: libpango-1.0.so - found (/usr/local/lib/libpango-1.0.so)
===========================================================================

========================< phase : configure  >========================
===>  Configuring for pan-0.140
configure: loading site script /xports/Templates/config.site
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... (cached) /bin/mkdir -p
checking for gawk... (cached) /usr/bin/awk
checking whether gmake sets $(MAKE)... yes
checking whether gmake supports nested variables... yes
checking whether UID '0' is supported by ustar format... yes
checking whether GID '0' is supported by ustar format... yes
checking how to create a ustar tar archive... (cached) /usr/bin/tar
checking whether to enable maintainer-specific portions of Makefiles... yes
checking whether gmake supports nested variables... (cached) yes
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++48 accepts -g... yes
checking for style of include used by gmake... GNU
checking dependency style of g++48... gcc3
checking for gcc... gcc48
checking whether we are using the GNU C compiler... yes
checking whether gcc48 accepts -g... yes
checking for gcc48 option to accept ISO C89... none needed
checking whether gcc48 understands -c and -o together... yes
checking dependency style of gcc48... gcc3
checking how to run the C preprocessor... cpp48
checking for grep that handles long lines and -e... (cached) /usr/bin/grep
checking for egrep... (cached) /usr/bin/egrep
checking for ANSI C header files... (cached) yes
checking whether time.h and sys/time.h may both be included... yes
checking for localtime_r... yes
checking for close... yes
checking for tr1/unordered_set... yes
checking whether the compiler implements namespaces... yes
checking whether the compiler has ext/hash_set... yes
checking for gawk... (cached) /usr/bin/awk
checking whether gmake sets $(MAKE)... (cached) yes
checking for ranlib... /usr/local/bin/ranlib
checking for sys/types.h... (cached) yes
checking for sys/stat.h... (cached) yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for memory.h... (cached) yes
checking for strings.h... (cached) yes
checking for inttypes.h... (cached) yes
checking for stdint.h... (cached) yes
checking for unistd.h... (cached) yes
checking for errno.h... (cached) yes
checking for fcntl.h... (cached) yes
checking whether NLS is requested... no
checking for intltool >= 0.40.6... 0.51.0 found
checking for intltool-update... /usr/local/bin/intltool-update
checking for intltool-merge... /usr/local/bin/intltool-merge
checking for intltool-extract... /usr/local/bin/intltool-extract
checking for xgettext... no
checking for msgmerge... no
checking for msgfmt... no
checking for gmsgfmt... no
configure: error: GNU gettext tools not found; required for intltool
===>  Script "configure" failed unexpectedly.
Please report the problem to gnome@FreeBSD.org [maintainer] and attach the
"/construction/xports/news/pan/work/pan-0.140/config.log" including the output
of the failure of your make command. Also, it might be a good idea to provide
an overview of all packages installed on your system (e.g. a
/usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

Stop.
make: stopped in /xports/news/pan
===========================================================================
```


----------



## marino (May 5, 2016)

so the problem is identical on FreeBSD.  Congratulations, with Synth you've discovered a bug in the pans port.  Please open a PR at https://bugs.freebsd.org/bugzilla/ and attach your build log as proof so that the port maintainer gets informed.


----------



## free-and-bsd (Aug 1, 2016)

Hi marino@.
Got another question on synth: just how does one actually configure "external repository"? 
In my file /usr/local/etc/pkg/repos/00_synth.conf the local repo /var/synth/live_packages is listed as repo. Will synth then fetch prebuilt packages from that repo if I configure to fetch prebuilt packages?

Thank you.


----------



## marino (Aug 1, 2016)

free-and-bsd@
`pkg -vv`
You should see two repositories enabled: *FreeBSD *and *Synth*

prefetching pulls from *FreeBSD *repository.

`pkg -vv | grep CONS`

Make sure CONSERVATIVE_UPGRADE is set to false (adjust pkg.conf as necessary to accomplish this)


----------



## free-and-bsd (Aug 5, 2016)

marino@ said:


> free-and-bsd@
> `pkg -vv`
> You should see two repositories enabled: *FreeBSD *and *Synth*
> 
> ...


OK thank you very much. I've checked pkg.conf(5)() manual page for options, too. But the thing is I _don't want_ packages from FreeBSD repo because I've had problems with them crashing for no obvious reason while the locally built one worked fine.
(to moderator: I have no idea why it attaches "()" after pkg.conf(5))


----------



## marino (Aug 5, 2016)

free-and-bsd said:


> But the thing is I _don't want_ packages from FreeBSD repo because I've had problems with them crashing for no obvious reason while the locally built one worked fine.



Then leave prefetching option at the default, which is "don't".  Synth will never fetch in that case.
With CONSERVATIVE_UPGRADE=false, freebsd repo would only be used if a required package isn't in synth repository.
You might even want to deinstall everything and reinstall from synth repository (explicitly with -r Synth to be sure), then you'll know everything is locally built (only valid if there were already packages installed on the system)


----------



## free-and-bsd (Aug 5, 2016)

Actually, I've checked my configuration and the only enabled repo is that of live_packages. And I've already replaced all the local packages with those built with synth.
I was asking because, while building a new port, synth started to build packages already built, to my best knowledge. Meantime I don't remember updating the ports tree after the last build... which means nothing, of course. But then I also wanted to get once and for all clear about this interesting option of fetching packages. In portmaster that didn't seem to work nor to be going to work.


----------



## archfan (Aug 12, 2016)

I think I found a bug here. Synth complains about a circular dependency when I try to override the default gcc setting in make.conf.


```
doas synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.

ports-mgmt/pkg scan aborted because a circular dependency on devel/binutils was detected.
... backtrace devel/gmake
... backtrace devel/binutils
... backtrace ports-mgmt/pkg
Unexpected pkg(8) scan failure!
Unfortunately, the system upgrade failed
```

Contents of /usr/local/etc/synth/*make.conf


```
DEFAULT_VERSIONS+=ssl=libressl
OPTIONS_UNSET=BONJOUR AVAHI PGSQL MYSQL DEBUG V4L CUPS DOCS EXAMPLES HAL IPV6 NLS TEST PULSEAUDIO
OPTIONS_SET=DEVD
VIDEO_DRIVER=nvidia
ftp_curl_UNSET=  TLS_SRP
USE_GCC?=6
```

It works fine without "USE_GCC?=6". However I don't intend to use an ancient gcc compiler.



talsamon said:


> `Poudriere` does not recompile a package if it does not change, `Synth` does          (e.g webkit-gtk is recompiled every second run).



I have the same problem. Whenever a library gets a slight bump, Synth recompiles ALL packages that depend on that library. Is this intended or just accidental, as it causes MAJOR recompiles everytime. For instance I've just recompiled freetype2 to enable LCD_FILTERUNG and it recompiled > 100 packages, including big ones like Firefox and Chromium (and worse llvm37). It kind of defeats the point of using shared libraries.


----------



## kpa (Aug 12, 2016)

Try to reproduce that outside Synth, it's highly likely that the bug is in the ports system and not in Synth at all.


----------



## archfan (Aug 12, 2016)

kpa said:


> Try to reproduce that outside Synth, it's highly likely that the bug is in the ports system and not in Synth at all.



I just tested this without Synth and it works fine with USE_GCC?=6 enabled.


```
doas synth install net-im/telegram-purple

ports-mgmt/pkg scan aborted because a circular dependency on devel/binutils was detected.
... backtrace devel/gmake
... backtrace devel/binutils
... backtrace ports-mgmt/pkg
Unexpected pkg(8) scan failure!
```

cd /usr/ports/net-im/telegram-purple && make install clean works as intended.


----------



## marino (Aug 12, 2016)

It's not a bug.  The circular dependency is *detected*. You caused it with "USE_GCC?=6" which says "use GCC for every port".  GCC 6 has many dependencies, so every port has the same dependencies.  The detection is valid.

I'm not addressing the point of "shared libraries" because it reflects a fundamental misunderstanding of how repository builders like poudriere and synth work.  TDLR; it's not "accidental".


----------



## marino (Aug 12, 2016)

You need to figure out the proper way to change ports compilers, if there even is one.  setting USE_GCC in make.conf is not valid.  You could start with Mk/bsd.defaults.mk and change GCC_DEFAULT but that only affects ports already using GCC I suspect.


----------



## archfan (Aug 12, 2016)

> I'm not addressing the point of "shared libraries" because it reflects a fundamental misunderstanding of how repository builders like poudriere and synth work. TDLR; it's not "accidental".



Good to know, thanks. Does poudriere exhibit the same behavior, e.g. rebuilding "everything" when a library changes slightly?



> change GCC_DEFAULT but that only affects ports already using GCC I suspect.



This is actually what I want. I'm fine with using clang for ports that support it but I want to use a different GCC version for all ports that depend on GCC.


----------



## marino (Aug 12, 2016)

archfan said:


> Good to know, thanks. Does poudriere exhibit the same behavior, e.g. rebuilding "everything" when a library changes slightly?



They have exactly the same behavior, but it's not based on "library changes".  library versions, symbols, contents have nothing to do with it.  The thing triggering a rebuild could be 3-4 levels removed.



> This is actually what I want. I'm fine with using clang for ports that support it but I want to use a different GCC version for all ports that depend on GCC.



Nothing beyond the standard 4.8 is guaranteed to work.  Most ports will but not all.  If even 4.9 would fully work, the default would be GCC 4.9.


----------



## ooglek (Aug 16, 2016)

I've used Synth a few times and it's pretty great for building in parallel.

However, I'm still confused about exactly how it goes about doing so, why it does certain things, and where I might be going wrong.

I've got 166 ports installed. I used to have lang/php56 but now I use lang/php70.

However, when I ran `synth status` it told me that it was going to rebuild lang/php56 -- but pkg doesn't report that it is installed.

So I checked dependencies and reverse dependencies as well -- no mention of lang/php56.

So I thought to run `synth rebuild-repository` but it STILL says it's going to build lang/php56 (which conflicts with lang/php70, which breaks stuff). Plus instead of rebuilding, it's going to build everything "N" New.

I want to use synth, but either I've found a bug in the ports tree, or there's something weird on my system, or there's another issue.

What do I need to post here that would help? There's nothing in /var/log/synth/* that was output recently.

I *suspect *that databases/adodb5 is to blame, as it requires php56, though adodb.org states it supports PHP7 (and I've confirmed that it does). I've reached out to the maintainers.


----------



## marino (Aug 16, 2016)

ooglek@, the default version of php is 5.6.  see Mk/bsd.default-versions.mk
Unless you specifically declare another version in <profile>-make.conf, synth in going to take the defaults.  (no, it doesn't matter what's in /etc/make.conf, that's not used.  You need to define a new *make.conf per profile).


----------



## ooglek (Aug 17, 2016)

marino@, Thank you! That seems to have resolved the php56 issue for me. 

When I run `synth rebuild-repository` why does the indicator of N (new), R (rebuild) and U (upgrade) all turn to New, when the versions installed haven't changed? Prior to running that, the N/R/U indicators seemed accurate and helped give me a good sense of what synth was going to do.

Also, like a recent thread, `pkg audit -F`, which pulls the latest vulnerability list from FreeBSD, came as a surprise to many, including me. For 90% of system admins, beyond `synth status` and `synth upgrade-system`, are there commands that one needs to run regularly or occasionally in the course of time passing?

Feature Request: a single synth command that runs `make -C /usr/ports/category/port config` on all of the out-of-date configurations.


----------



## marino (Aug 17, 2016)

ooglek said:


> When I run `synth rebuild-repository` why does the indicator of N (new), R (rebuild) and U (upgrade) all turn to New, when the versions installed haven't changed? Prior to running that, the N/R/U indicators seemed accurate and helped give me a good sense of what synth was going to do.



Because they are different ports.  php 70 version of something is a different port than php 56 version of the same thing.  



> Also, like a recent thread, `pkg audit -F`, which pulls the latest vulnerability list from FreeBSD, came as a surprise to many, including me. For 90% of system admins, beyond `synth status` and `synth upgrade-system`, are there commands that one needs to run regularly or occasionally in the course of time passing?



I'm not sure why you're asking me this.  Synth just keeps a local repository up-to-date.  You can utilitize that repository with *pkg *commands (maybe with -r Synth) or you can get Synth to manipulate *pkg* with commands like `synth install`, but I consider `pkg audit` out of scope.   Synth will build the latest available, regardless of it's vulnerability state.



> Feature Request: a single synth command that runs `make -C /usr/ports/category/port config` on all of the out-of-date configurations.



With synth, you want the _least_ number of configs.  If the saved config would be the same as the defaults, you don't want it at all.  You only want configs that differ from the defaults.  There is even a script to help remove redundant configurations at Tools/scripts/redundant-opt-files.sh.  The vast majority of out-of-date configuration would be removed, not re-saved.  It's a one-time effort that needs to be done when moving to Synth which removes all the cruft caused by building live or with portmaster.  So such a command is counter-productive.


----------



## Sylhouette (Sep 9, 2016)

Hello all.
Sorry if this is not the place to ask but I have a question regarding the use of synth.

I need two different versions of packages, a php70 version and a php56 version.
I created a profile through synth configure named php70 and one named php56
I created a file /usr/local/etc/synth/php70-make.conf
I created a file /usr/local/etc/synth/php56-make.conf

I also created a file /usr/local/etc/synth/php70-build.list
I also created a file /usr/local/etc/synth/php56-build.list
The build file looks like this

```
www/apache24
lang/php70
lang/php70-extensions
```

My ports options directory is set to
[D] Port options directory  /var/db/synth/ports-php70
if I do `synth build /usr/local/etc/synth/php70-build.list` It builds all the ports, but I do not get the ports options screens. I need to adjust some settings!
How do I get these? 
Or must I do a `make -C /usr/ports/category/port` and copy the options from /var/db/ports/ to /var/db/synth/ports-php70?

If I want to build for the profile php56 I do it now by going to the configure screen and select php56.
Can I use a command line option to select the profile?

Thanks for your time.

regards


----------



## marino (Sep 9, 2016)

There are two ways of doing it.

You can set and unset individual port options in phpXX-make.conf
Option [D] under configure is where the port options files are.  You might have different ones for each profile, they might be shared, and they might be in common with the system version.
What you could do is "cd <category/portname> && make config" and then copy /var/db/ports/<category>_<portname> over to the profiles's ports options directory (assuming you don't just use /var/db/ports)


----------



## jonfr (Dec 2, 2016)

What can I do about this. This is the reason for my recent troubles (amount other errors I suspect). The thing is that mysql56-client is not installed at the moment.


```
synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.

devel/subversion scan aborted because a circular dependency on databases/mysql56-client was detected.
... backtrace security/cyrus-sasl2
... backtrace net/openldap24-sasl-client
... backtrace security/heimdal
... backtrace ftp/curl
... backtrace devel/cmake
... backtrace databases/mysql56-client
... backtrace devel/apr1
... backtrace devel/subversion
Unfortunately, the system upgrade failed.
```


----------



## jonfr (Dec 2, 2016)

I got this error. Not sure why that is.


```
pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
Updating Synth repository catalogue...
pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
repository Synth has no meta file, using default settings
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Unable to update repository Synth
```


----------



## marino (Dec 2, 2016)

jonfr@, regarding the circular dependency, this is surely caused by non-compatible options that have been selected in the past.
Unless you know for a fact which ports require modifying, you might want to clear out your options directory (e.g. /var/db/ports).
The devel/apr1 might have the MySQL option selected for example.
Basically a circular dependency between ports A and B mean that both of them need the other to build.  It doesn't happen with default options but easily could with non-default options.

secondly, are you using synth version v1.65?
in any case, remove /usr/local/etc/pkg/repos/00_synth.conf to clear those second set of errors.


----------



## jonfr (Dec 2, 2016)

marino@ The system I have is a complete mess. I don't know why it happened, it just did, so I'm doing a re-install of everything from ground up (takes about 1 day now from start to finish). I don't know why I'm using v1.65. This is what `make install` installed when I ran that command. I suspect it might be related to the error mess I have been dealing with for the past few hours.


----------



## marino (Dec 2, 2016)

jonfr@ the version 1.65 is the latest, that's good.
As I said, if you are starting from scratch, remove all the options in /var/db/ports (assuming that is what you configured the options to be).
That will give you default options again.
If there are any ports you definitely want to customize, just use `make -C /usr/ports/<category>/<port> config` to set each one specifically.


----------



## jonfr (Dec 4, 2016)

I got one question about ports-mgmt/synth. It marks already installed packages (I install using `make install clean` so I can configure everything correctly) as new packages. I'm not ready to rebuild everything that has already been installed properly.

I get a output like this. Thanks for the help.


```
synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  N => print/indexinfo
  N => devel/gettext-runtime
  N => devel/gettext-tools
  N => devel/pkgconf
  N => devel/gmake
  N => databases/gdbm
  N => lang/perl5.24
  N => devel/libffi
  N => devel/p5-Locale-gettext
  N => devel/readline
  N => misc/help2man
  N => devel/xorg-macros
  N => lang/python27
  N => print/texinfo
  N => textproc/libxml2
  N => x11/xproto
  N => devel/m4
  N => devel/py-setuptools27
  N => textproc/expat2
  N => security/libgpg-error
  N => security/libgcrypt
  N => textproc/libxslt
  N => devel/libpthread-stubs
  N => devel/libcheck
  N => x11/libXau
  N => x11/libXdmcp
  N => x11/xcb-proto
  N => x11/libxcb
  N => x11/xtrans
  N => lang/python2
  N => x11-fonts/xf86bigfontproto
  N => devel/autoconf-wrapper
  N => x11/bigreqsproto
  N => x11/inputproto
  N => x11/kbproto
  N => x11/xcmiscproto
  N => x11/xextproto
  N => textproc/xmlcatmgr
  N => math/gmp
  N => devel/autoconf
  N => devel/automake-wrapper
  N => devel/libtool
  N => x11/libX11
  N => devel/automake
  N => graphics/png
  N => security/ca_root_nss
  N => devel/libevent2
  N => devel/py-pytz
  N => textproc/iso8879
  N => textproc/xmlcharent
  N => devel/bison
  N => devel/jansson
  N => devel/libev
  N => devel/py-babel
  N => print/freetype2
  N => textproc/docbook-sgml
  N => textproc/docbook-xml
  N => textproc/py-MarkupSafe
  N => textproc/py-pystemmer
  N => textproc/sdocbook-xml
  N => net/openldap24-client
  N => www/spdylay
  N => math/mpfr
  N => devel/py-Jinja2
  N => devel/py-six
  N => devel/scons
  N => security/krb5
  N => security/libssh2
  N => multimedia/librtmp
  N => dns/libidn2
  N => graphics/py-imagesize
  N => textproc/docbook
  N => textproc/py-alabaster
  N => textproc/py-docutils
  N => textproc/py-pygments
  N => textproc/py-snowballstemmer
  N => textproc/py-sphinx_rtd_theme
  N => archivers/liblz4
  N => archivers/lzo2
  N => www/nghttp2
  N => devel/binutils
  N => devel/cmake-modules
  N => devel/jsoncpp
  N => textproc/py-sphinx
  N => archivers/libarchive
  N => ftp/curl
  N => x11-fonts/fontconfig
  N => devel/cmake
  N => x11/libXext
  N => textproc/docbook-xsl
  N => x11/fixesproto
  N => x11/libXfixes
  N => misc/pciids
  N => devel/libedit
  N => devel/ninja
  N => x11/videoproto
  N => devel/libpciaccess
  N => devel/llvm37
  N => x11/damageproto
  N => x11/libXv
  N => devel/libclc
  N => devel/libdevq
  N => devel/makedepend
  N => x11/dri2proto
  N => x11/glproto
  N => x11/libXdamage
  N => x11/libXvMC
  N => x11/libxshmfence
  N => x11/presentproto
  N => graphics/libdrm
  N => devel/icu
  N => devel/pcre
  N => x11/libICE
  N => converters/libiconv
  N => graphics/libglapi
  N => devel/glib20
  N => x11/libSM
  N => x11/xf86vidmodeproto
  N => textproc/html2text
  N => x11/libXxf86vm
  N => x11/renderproto
  N => x11/xcb-util
  N => graphics/gbm
  N => textproc/minixmlto
  N => sysutils/gnome_subr
  N => x11-fonts/libfontenc
  N => devel/dbus
  N => x11/libXrender
  N => x11/pixman
  N => x11/xcb-util-renderutil
  N => graphics/libEGL
  N => graphics/libGL
  N => x11-fonts/mkfontscale
  N => devel/nasm
  N => security/libtasn1
  N => security/nettle
  N => emulators/tpm-emulator
  N => dns/libidn
  N => graphics/cairo
  N => textproc/p5-XML-Parser
  N => x11-fonts/mkfontdir
  N => devel/dbus-glib
  N => devel/gobject-introspection
  N => devel/libdaemon
  N => security/p11-kit
  N => security/trousers
  N => graphics/jpeg-turbo
  N => textproc/intltool
  N => security/gnutls
  N => graphics/jbigkit
  N => net/avahi-app
  N => print/libpaper
  N => graphics/tiff
  N => print/cups
  N => graphics/giflib
  N => archivers/unzip
  N => x11/libXi
  N => x11/recordproto
  N => java/java-zoneinfo
  N => archivers/zip
  N => x11-fonts/dejavu
  N => x11/libXtst
  N => x11-toolkits/libXt
  N => java/bootstrap-openjdk
  N => java/javavmwrapper
  N => audio/alsa-lib
  N => print/gsfonts
  N => java/openjdk7
  N => graphics/jbig2dec
  N => graphics/lcms2
  N => graphics/svgalib
  N => graphics/webp
  N => shells/bash
  N => math/mpc
  N => devel/talloc
  N => print/ghostscript9-agpl-base
  N => java/openjdk8
  N => graphics/gd
  N => graphics/jasper
  N => textproc/py-libxml2
  N => lang/gcc6-aux
  N => devel/apache-ant
  N => devel/ncurses
  N => devel/p5-Parse-Yapp
  N => devel/popt
  N => devel/tevent
  N => graphics/netpbm
  N => graphics/peps
  N => graphics/scr2png
  N => textproc/docbook-xsl-ns
  N => textproc/igor
  N => textproc/iso-schematron-xslt
  N => textproc/itstool
  N => textproc/scr2txt
  N => textproc/xhtml
  N => databases/tdb
  N => archivers/p7zip
  N => www/links1
  N => x11-fonts/droid-fonts-ttf
  N => x11-fonts/gentium-plus
  N => x11-fonts/lohit
  N => misc/freebsd-release-manifests
  N => misc/ini_file_manager
  N => devel/adacurses
  N => devel/gamin
  N => devel/libinotify
  N => devel/p5-Parse-Pidl
  N => dns/py-dnspython
  N => textproc/docproj
  N => textproc/fop
  N => japanese/font-ipa
  N => chinese/arphicttf
  N => databases/ldb
  N => databases/ntdb
  N => korean/nanumfonts-ttf
  N => sysutils/libsunacl
  N => misc/freebsd-doc-en
  N => devel/py-iso8601
  N => ports-mgmt/dialog4ports
  N => ports-mgmt/poudriere
  N => ports-mgmt/synth
  N => dns/dnsmasq
  N => net/dhcpd
  N => net/samba42
  N => net/whois
  N => editors/nano
Total packages that would be built: 226
The complete build list can also be found at:
[/U]
```


----------



## ANOKNUSA (Dec 4, 2016)

jonfr said:


> I'm not ready to rebuild everything that has already been installed properly.



It seems you've misunderstood what Synth is and what it does. The purpose of Synth is to create a local, custom package repository. While it is capable of installing individual ports or upgrading installed ports, it does so by first updating the local repository, then installing or upgrading packages using that repository. It is not simply a front-end to the ports system; before it can do anything at all, it must create the package repository. You therefore need to build everything the first time around.

The only possible way around that is to set "Fetch prebuilt packages" to "true," which will fetch packages from the official repository when possible. But even that will only fetch packages which you have not customized, and which are not affected by any customizations you have made---meaning that it will almost certainly have to rebuild some ports that are already installed, and if you've set any global customizations using make.conf(5) you'll probably have to build almost everything anyway.


----------



## abishai (Dec 11, 2016)

marino@ said:


> It's called "Synth" and one of its main purposes is to replace portmaster


Sorry for not able to read all 33 pages, however I have a question.
I'm thinking to replace `portmaster` on my system, however when I tried `synth` I found it's lacks the ability to ask for port options. So, basically, it's worse than portmaster for me as I need to run `make config` first. I don't want to do it for all run-deps manually. Are you planning to add interactive mode like `portmaster` has?
Thanks.


----------



## marino (Dec 11, 2016)

no, I'm never adding an "interactive" mode.  It's not needed (despite what you may believe).  In reality, people only change the options on a handful of ports.  For all the rest they use the defaults.  Synth uses the stored defaults (if any) or the defaults.  It works fine.

You're just stuck thinking that the "portmaster way" is the "correct way".  You have to expand your thinking.
In your case it would be easier to delete all the currently stored option (because you're sure to have dozens of obsolete ones, courtesy of portmaster) and redo any that need to be changed from defaults.  Global changes you can do through the profile -make.conf file.


----------



## kpa (Dec 11, 2016)

Stored options are pain in my opinion, I prefer to set everything in the builder's make.conf like this:


```
OPTIONS_UNSET= CUPS DBUS X11

DEFAULT_VERSIONS+= ssl=openssl

ports-mgmt_poudriere-devel_SET= ZSH

#devel/git
devel_git_UNSET = SEND_EMAIL

# editors/vim
editors_vim_SET= CONSOLE
editors_vim_UNSET= RUBY TCL ATHENA GNOME GTK2 MOTIF
```

And so on. This is easier for me because of all of the options are now centralized into one single file that I can edit and update in one go. The downside is that there's a bit of extra initial work to get a list of the options in a format suitable for the make.conf file.


----------



## abishai (Dec 11, 2016)

marino@ said:


> In reality, people only change the options on a handful of ports.


I change a lot of options. I know what they do and what I need on my laptop. If I don't I'd probably switched to packages. Can you explain what is "portmaster way"? I can delete all port options and set them again, that's not the problem, but I don't understand how can I do this again initially (and maintain) with synth? Looks like reinvention of poudriere in more solid and simple way. And no, I am not trolling, but I see that portmaster is not maintained for years and I need to know what to do when it finally stops completely.

Even If one need to change 1 port option, he *must* run `make config`. What if this is run dep? We must start building with investigating of dep tree and how changes affecting it? I'd say synth is impossible to use in portmaster way, so it's impossible to replace it.



kpa said:


> I prefer to set everything in the builder's make.conf like this


Gentoo way If I recall their system correctly. However the problem of knowing exact dep tree exists for this option as well. For example, if you want to install something new, you are forced to do it *not* in synth, because you need to gather initial options somehow. And to maintain - if option list changes.


----------



## marino (Dec 11, 2016)

step 1) `rm -rf /var/db/ports` (or whatever directory you use for storing ports options)
step 2) identify the ports that you have different options
step 3) `make -C <category>/<port> config` for those identified in step 2
step 4) build the first set of packages given a specific build list or querying the host system with "synth prepare-system"
step 5) periodically repeat step 4.  Generally you don't touch options again.

that's it.
You don't have to worry about anything else, you don't have to worry about recursion or anything.

It's a "replacement" of functionality.  It's quite possible to replace portmaster, many already have.


----------



## marino (Dec 11, 2016)

note that you don't have to delete old options.  You can set PORT_DBDIR in /etc/make.conf to a new directory and then make synth use that new directory after you set the options.


----------



## marino (Dec 11, 2016)

kpa said:


> Stored options are pain in my opinion, I prefer to set everything in the builder's make.conf like this.



The downside of this approach is that Synth can't detect when the options are obsolete.  When the option set is saved as a snapshot it's easy to determine this, which alerts the user that the port options have changed.


----------



## abishai (Dec 11, 2016)

step 2) is a tricky one as I don't know what was changed. What would you suggest? Personally, I'm thinking to `config-recursive` all top level ports and set 'pristine good options' directory for synth to use.


----------



## marino (Dec 11, 2016)

well, it's been recommended NOT to do that.  The recommendation is only set options on packages that need options that differ from default.

You can use /usr/ports/Tools/scripts/redundant-opt-files.sh to remove all the redundant options files you currently have.  what's left are either "bad" saved options (which synth will detect) and the changed ones which you want.

In the end, you are free to use the tools in non-recommended ways though.  It's at your own risk.


----------



## abishai (Dec 11, 2016)

redundant == 'identical to port's default' ? Looks like a plan 

edit: right. looked into script for 'redundant' meaning.


----------



## Datapanic (Dec 16, 2016)

marino@ said:


> Insist on building ports from source no matter what.
> 
> Need ports with options other than defaults.
> 
> ...



Thank you so much for your work on this - it's awesome!  

I answer YES to all of the above except #5 - portmaster has not been pleasant in recent months with FreeBSD 11-Release and I am not happy with failed port updates using `/usr/local/sbin/portmaster -a -d --no-confirm -G` in my custom script that updates everything.  Even adhering to /usr/ports/UPDATING doesn't help prevent numerous 'gotcha's' that just happen when updating ports.  I'm tired of spending time manipulating port installs that want python 2 or python 3, rust 12 or rust 13, can't figure out which libboost to use and so on.  #1 above is virtually impossible with portmaster  now.

I also think it's great to see something written in Ada!  I took Ada in college ages ago (Janus Ada) and thought it was before it's time.  Ada is an excellent choice for Synth.  Back then, we avoided Ada at all costs and went with C 

Finally, portmaster holdouts - if you are wondering about synth, the switch over is not painful at all.


----------



## abishai (Dec 21, 2016)

How can I set knob in make.conf for 1 port ? I need explicitly link security/strongswan to base openssl.

```
DEFAULT_VERSIONS+=ssl=libressl perl5=5.24 pgsql=9.6
OPTIONS_UNSET= DOCS EXAMPLES IPV6
```


----------



## marino (Dec 21, 2016)

read the man page or instructions on github regarding [profile]-make.conf.
/etc/make.conf is NOT used, every profile has their own profile-specific file that it uses instead.


----------



## abishai (Dec 21, 2016)

marino@ said:


> read the man page or instructions on github regarding [profile]-make.conf.
> /etc/make.conf is NOT used, every profile has their own profile-specific file that it uses instead.


I know about this.

```
cat /usr/local/etc/synth/LiveSystem-make.conf
DEFAULT_VERSIONS+=ssl=libressl perl5=5.24 pgsql=9.6
OPTIONS_UNSET= DOCS EXAMPLES IPV6

.if ${.CURDIR:N*/ports/security/strongswan} == ""
DEFAULT_VERSIONS=ssl=base
.endif
```
For general make.conf this code should work, however if is ignored when port being build under synth.


----------



## marino (Dec 22, 2016)

that's not legal.  You can't have different ssl versions.  It's either base or something else, but a mixture is wrong (by definition).   It's not a matter of opinion, this has been stated quite a few times.


----------



## abishai (Dec 22, 2016)

marino@ said:


> that's not legal.  You can't have different ssl versions.  It's either base or something else, but a mixture is wrong (by definition).   It's not a matter of opinion, this has been stated quite a few times.


strongswan with libressl has issue  and not working at all.
I already reported it https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212149 and use linking against base as temporary solution. I suppose I should make a different profile for that port?


----------



## marino (Dec 22, 2016)

I am saying I am not sure your temp solution even functions.   There's a reason default ssl is set once, the different versions can conflict (built against one and pulls in the other, libraries affected etc.).

Your hack doesn't work because your pattern is wrong.   It would be "N/xports/security/strongswan".  The ports aren't put in /usr/ports internally.


----------



## abishai (Dec 22, 2016)

marino@ said:


> I am saying


I'm using it for 2 months. Yes, I understand that it's not very good solution and hope switch to openiked which was recently added to port tree.
Thanks for the hint, I should notice it in MAKE_ENV section of synth logs.

BTW, synth acts better here than portmaster - I patched Mk files to break openssl conflict detection, it's not necessary for synth.


----------



## ShelLuser (Dec 22, 2016)

Well, I finally learned of this software and gave it a try. I'll share my findings and leave it at that.

I can say: it's not for me. First of all I dislike some of the approach which I pick up as plain out arrogant. It even shows in the OP: "_Aimed at people who think Portmaster is great_" as well as in the synth(1) manual page: "_It was hoped that it would attract users of the older PortMaster and PortUpgrade ports management tools by providing them with a superior and well-maintained alternative._". Call me old fashioned but approaches like these tick me off. Live and let live is my motto, which means showing respect for other people's work.

Unfortunately for me this also sets a mindset.

So here I am in /usr/ports/devel/apache-ant and I want to check which ports Synth found to be new. Picture me surprised...


```
peter@macron:/usr/ports/devel/apache-ant# synth status
Please change the current directory; Synth is unable to launch from here.
```
As more experimentation showed me: Synth refuses to run inside the /usr/ports structure. I'm sure there are good reasons for that, but it's not always very desirable nor user friendly. Especially when using commands which do very little but show me the current status.

But more so because I sometimes use Portmaster for specific tasks. Including, if needed, a careful upgrade of one specific port (-o). Which means that I'll have to specify the origin. Doable by utilizing `#pkg info -ox <name>` but also by being in the physical directory itself. That of course doesn't work.

Of course I agree that this is but a detail, and I'm also not ignoring the possibility that there are most likely good reasons for this. Obviously it is important to use a tool in a proper and sometimes intended way. Still, I think it's fair to mention nonetheless given the direct comparison with Portmaster regarding "superiority" in the overall (which, for me, includes user friendliness).

So then I tried this again, this time in the proper directory called /root/updates where I keep some of my custom made scripts (utilizing Portmaster). This time I was greeted with another error which made me seriously scratch my head:


```
peter@macron:~/updates#synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.

raised REPLICANT.SCENARIO_UNEXPECTED : /sbin/mount -t tmpfs tmpfs /usr/obj/synth-live/SL09 => failed with code 1
```
Why in the world would Synth rely on a (memory wasting) temporary file system while /tmp and /var/tmp have been carefully set up for this very specific reason? This machine I'm using isn't the most modern one, but it can maintain a package repository just fine.

Sure, I may have completely ignored any required steps to set up and configure Synth. But there's no mentioning of this requirement in the manual page. Obviously I learned about /usr/local/etc/synth/synth.ini, also because I was curious about the tmpfs approach and this file does have some possibly related options (Tmpfs_workdir and Tmpfs_localbase). Of course all the synth(1) manualpage tells me is that I don't have to touch this file, and the file itself doesn't have a manualpage at all.

So I learned of `# synth configure`, used that and confirmed my suspicions: The options K and L which provide me with a way to make Synth use (or not use) tmpfs for the work area and localbase (or both of course).

Just too bad that these options apparently get totally ignored. It's the only conclusion I can come up with.

So yeah, each to their own but this is definitely not for me. Synth might be more extensive feature wise (I simply cannot say, I only base myself on what I read in the manualpage) but for me Portmaster can take care of all the tasks I need while also ensuring that I remain fully in control over what it can and cannot do. For example the option to ensure that I always have a backup package the moment it upgrades a port. I don't see anything like that mentioned in the Synth documentation. It might be a supported feature, and I cannot rule out the option that I overlooked it, but for me that also adds up to user friendlyness.

I personally prefer the Unix mindset where a program performs a small task which allows for it to become part of a bigger task. Which is exactly what Portmaster provides. It probably takes more work to configure and set it up, no arguments there, but Portmaster easily allows me to set up my own package repository and distribute those to my other servers.

It might not be an "all in one" solution, but that definitely does not make it inferior, as hinted at in your documentation.

Alas, I do wish you good luck with the project and hopefully there's still something useful in here besides my (positively meant) criticism


----------



## ASX (Dec 22, 2016)

ShelLuser said:


> I can say: it's not for me. First of all I dislike some of the approach which I pick up as plain out arrogant. It even shows in the OP: "_Aimed at people who think Portmaster is great_" as well as in the synth(1) manual page: "_It was hoped that it would attract users of the older PortMaster and PortUpgrade ports management tools by providing them with a superior and well-maintained alternative._". Call me old fashioned but approaches like these tick me off. Live and let live is my motto, which means showing respect for other people's work.



What you define "arrogant", is what I define "truth", and until now I have followed all threads and some PR too regarding the comparison,  and what marino@ say is simply the truth, and to my knowledge *no one until now has posted some fact to demonstrate that marino@ is wrong.* 

Instead this forum is plenty of threads which demonstrated that very often portmaster / portupgrade doesn't do the right job., it was also already mentioned that portmaster was abandoned from the original author and just forcefully maintained now by someone else.

About option K / L and tmpfs, while Synth allow to use or not tmpfs for these two areas, there are of course other places where tmpfs are used without questions, so it is not that the options are ignored, it is that tmpfs are used elsewhere for other things, ultimately for performance reasons.

I hope you will give Synth another try, while Synth has a simple interface, it is not a simple tool and do lot of things better than portmaster, starting from building in a clean environment up to building a whole repository (or more repositories if needed).

By doing all that Synth effectively may be somewhat heavier from a system usage point of view, but also it perform lot of checks on behalf of the users to assure the quality of the results. (detecting circular dependency is just one case that come up to my mind, there are other of course).


----------



## PacketMan (Dec 22, 2016)

Synth is awesome.  It can abort on some old slow cpu machines when building certain ports, and for that reason I think portmaster needs to stay the official tool, but other than that Synth is the slickest thing since round wheels.


----------



## ASX (Dec 23, 2016)

PacketMan said:


> think portmaster needs to stay the official tool,



portmaster is not the official tool, and it is also not the "recommended" tool: PR 206922, see post #9


----------



## tobik@ (Dec 23, 2016)

PacketMan said:


> It can abort on some old slow cpu machines when building certain ports


What's the reason it aborts and which ports?


----------



## ASX (Dec 23, 2016)

tobik said:


> What's the reason it aborts and which ports?


It was not an "abort", the package was webkit2-gtk3, and while running on an old slow machine the watchdog that is implemented in Synth to reap stalled builds,  killed that build. There is a whole thread about that: Thread 58614


----------



## marino (Dec 23, 2016)

ShelLuser said:


> So here I am in /usr/ports/devel/apache-ant and I want to check which ports Synth found to be new. Picture me surprised...
> 
> 
> ```
> ...



A very good reason.  If the cwd is null mounted inside the slave builders, the run will fail in spectular and completely unpredictable ways.  solution: `(cd / && synth status)`.  In other words, run in a subshell to change directory first.  While portmaster is designed to assume it's being run in ports tree, synth requires the opposite never happen for technical reasons, but it's not hard to work around.



> But more so because I sometimes use Portmaster for specific tasks. Including, if needed, a careful upgrade of one specific port (-o). Which means that I'll have to specify the origin. Doable by utilizing `#pkg info -ox <name>` but also by being in the physical directory itself. That of course doesn't work.



confirmed, you have to specify the origin.  it's an intentional requirement given it's impossible to run synth in the ports tree due to null mount issue.  It's a minor feature of portmaster that doesn't translate.



> Why in the world would Synth rely on a (memory wasting) temporary file system while /tmp and /var/tmp have been carefully set up for this very specific reason? This machine I'm using isn't the most modern one, but it can maintain a package repository just fine.



All current releases of FreeBSD comes with tmpfs by default.  It can only be turned off by rebuilding the kernel that way I think.  Confirmed, tmpfs is required although the big mounts (localbase and the work directory) can use a disk optionally but not all uses of tmpfs have alternatives.  But no, I don't think it's specifically documented that the kernel has to support tmpfs.



> So I learned of `# synth configure`, used that and confirmed my suspicions: The options K and L which provide me with a way to make Synth use (or not use) tmpfs for the work area and localbase (or both of course).
> 
> Just too bad that these options apparently get totally ignored. It's the only conclusion I can come up with.



which as I just said is a faulty conclusion.  K and L work just fine, but tmpfs is used in many other places.



> So yeah, each to their own but this is definitely not for me.



Okay, well.  thanks for looking and good luck.


----------



## PacketMan (Dec 23, 2016)

ASX said:


> It was not an "abort", the package was webkit2-gtk3, and while running on an old slow machine the watchdog that is implemented in Synth to reap stalled builds,  killed that build. There is a whole thread about that: Thread 58614



I guess that depends on your definition of abort.

"Abort - bring to a premature end because of a problem or fault."   Hmmm what's the definition of problem or fault?


----------



## ASX (Dec 23, 2016)

PacketMan said:


> guess that depends on your definition of abort.



Among developers "abort" is usually intended as premature end of the program itself (Synth), not as stopping to produce its output (webkit2-gtk3 pkg), more precisely "abort" refer to events that will generate a SIGABRT signal or more in general a different signal, one of those cases where a core file will be dumped on your filesystem.
signal(3) abort(3)


----------



## abishai (Dec 26, 2016)

I try to install net/wireshark-lite with synth, however looks like I can't do it

```
abishai@sphinx:~ % doas synth status net/wireshark-lite
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, pgrade):
  N => net/wireshark-lite
Total packages that would be built: 1

abishai@sphinx:~ % doas synth just-build net/wireshark-lite
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.

abishai@sphinx:~ % doas synth force net/wireshark-lite
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
```
This command should work, what makes wireshark-lite special?


----------



## marino (Dec 26, 2016)

step 1 is understanding the problem.
how does one understand what synth does?  They look at the logs.

There are two explanations for what you've seen:

the package has already been built and is current
The package can't be built because the port is setting IGNORE
The 00* log will tell you which one.  So check the logs.


added: The "force" command means the first bullet can't be happening.  You've got an IGNORE set somewhere.


----------



## abishai (Dec 26, 2016)

Right, KRB_BASE is incompatible with libressl and I forgot to make config. I thought synth will print it. This tool is not newbie friendly!


----------



## scottro (Dec 26, 2016)

*Synth using memory till swap is full*

(Not sure if this is the best place for this--perhaps give it its own thread or link to the thread below, that ended in May? )  

I see that someone had this issue back with FreeBSD-10.x, where synth started using swap at an unexpected rate.  https://forums.freebsd.org/threads/56171/   Lately, this has started happening to me on 2 FreeBSD-11 systems, both with reasonably good processors, zfs, and 8 GB of RAM.  Adding a second swap partition didn't help.  This seems to happen when building large packages, such as libreoffice, however, using portmaster or just doing make install from a port doesn't do this--it will use a lot of CPU, but swap stays at unused or just a little bit used.  This happens when I run synth prepare-system.  I haven't seen any mention of it save for that one thread I linked, so it may be one of those Just Me(TM) things, but I do see it on two different machines.  I see swap gradually climbing till it reaches 100 percent and the machine becomes unresponsive.  Logs show things like

```
09:28:47 s kernel: swap_pager: out of swap space
Dec 26 09:28:47 s kernel: swap_pager_getswapspace(5): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s last message repeated 6 times
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
```


----------



## marino (Dec 26, 2016)

it's definitely not a good thread for this.  the thread is already 35 pages long and nobody reads (good) information contained anymore.
There are 2 possibilities: 

you've set too many builders + jobs for the resources you have for every case
you've got a leak somewhere.  Something is using but not releasing swap space.  grep log directory for watchdog failures to see if those are happening (they really should never happen)


----------



## scottro (Dec 26, 2016)

Thanks, (as always) for the super quick answers. I will try tonight, and see if I can find something useful. If so, I'll open a new thread (or add to the old one, as it will show up in new posts)

For what it's worth, I used defaults, and looking through logs I don't see any watchdog errors.  This may mean more work in tracking down a memory leak than I have the ability (or time) to do right now.


----------



## fernandel (Dec 26, 2016)

scottro said:


> *Synth using memory till swap is full*
> 
> (Not sure if this is the best place for this--perhaps give it its own thread or link to the thread below, that ended in May? )
> 
> ...



I had the same problem still, now on FreeBSD 11-RELEASE. Not many times but it happened.
Not so long I use Synth and at the same time it built LibreOffice and Blender. SWAP raise to 36% and after LibreOffice was done dropped to 13.6%. And when was everything done swap stayed on 13%??
Thank you.


----------



## marino (Dec 26, 2016)

fernandel said:


> Not so long I use Synth and at the same time it built LibreOffice and Blender. SWAP raise to 36% and after LibreOffice was done dropped to 13.6%. And when was everything done swap stayed on 13%??



Is this is a question?
There's nothing wrong with swap staying high if that's the only fact available.
In other words, just because memory becomes available doesn't mean that contents stored in swap move back to memory (and consequently swap goes back to zero).  It's OS-specific, but I don't think FreeBSD works that way either.


----------



## abishai (Dec 27, 2016)

8Gb of ram is not enough for synth on my system. When 2 builders start to build Firefox/Thunderbird it starts swapping heavily and eventually system crashes. I disabled tempfs for workarea in Synth config and everything runs smoothly after that.


----------



## marino (Dec 27, 2016)

how much swap do you have?  maybe you find a cheap 64gb or higher SSD and use that (assuming we're not talking about a space constrained machine like a laptop here).

The original problem was likely your *swap* partition was way too small, but if the swap is a spinning disk, it's not much better than just turning off tmpfs for the workspace.

alternatively, you can shift the synth build root to an SSD partition so that building without tmpfs won't be such a big performance hit.


----------



## scottro (Dec 27, 2016)

Today I started a synth update, but first tried limiting vfs.zfs.arc_max from its default of RAM minus 1 GB to about 4 GB.  I think something is leaking memory, but don't have the expertise to track it down. In my case (on a ZFS mirror in one case, on a single disk pool in another--haven't tried on that machine yet), it seems to have fixed the issue.   (The build is not yet complete, but yesterday, buidling llvm37 and 39 and rust it swapped out, this time swap use is staying, so far, at around 12 percent.  I'll update in the other thread if this solution fails)

marino@, I realize that at this point you don't consider this a synth issue, so I'm copying these few posts, (with thanks for their input to abishai and fernandel) to a fresh thread.  

https://forums.freebsd.org/threads/59107/


----------



## marino (Dec 27, 2016)

Thanks scottro@, can you copy my last response there too?
reading your new thread, I realize there was an obvious 3rd explanation that nobody mentioned: The swap partition is simply too small (thus running out of swap is a legitimate message)


----------



## scottro (Dec 27, 2016)

Done.


----------



## rhsbsd (Jan 2, 2017)

For me a uneventful synth run starts like this:

`service cron onestop`
disable screensaver and sleep functions in my KDE system settings
set scrollback in Konsole to 'none'. Just imagine how much memory your consuming in 8 hrs if your using `top 'vmstat -m'`
remove/dont use troublesome usb devices during build. This is obvious but included for completeness.


----------



## bsduser2112 (Jan 23, 2017)

I have an issue with synth and was told to post my problem in this thread.  The issue is with synth suspending to the command line during compile of the port lang/sbcl. All the information I posted about it can be found at this thread on the forum: https://forums.freebsd.org/threads/59430/#post-340810.

Thanks


----------



## PacketMan (Jan 24, 2017)

I would say continue using the previous discussion thread you posted.

Edit: Whoops, logged onto the forum and saw this alert. Carry on.


----------



## horseflesh (Mar 17, 2017)

I used synth today to upgrade my packages and noticed something new. Normally I have 2 build processes running constantly. This time, one or the other slot would periodically go idle. I haven't watched synth work with rapt attention in the past but this seems to be a new behavior. Is it expected for a build slot to remain idle with work left to be done? If that doesn't sound right what data should I collect for a bug report?


----------



## rigoletto@ (Mar 17, 2017)

Probably, all the ports waiting had the single port that was being done as dependency. So, they need to wait that one to be done before get a slot.


----------



## Eric A. Borisch (Mar 17, 2017)

lebarondemerde said:


> Probably, all the ports waiting was the single port that was being done as dependency. So, they need to wait that one to be done before get a slot.



Yep, that is what is happening. They will change to "shutdown" rather than "idle" when #builders > #packages yet to build (or something similar -- not sure if it has look-ahead planning to see how many might yet run in parallel.)


----------



## marino (Mar 17, 2017)

it does.  It knows exactly how many builders can run in parallel through the end of the run.   As soon as there are more builders than can run in parallel, the extra builder(s) are shut down.  It's quite clever.


----------



## priyadarshan (Mar 18, 2017)

Yes, Synth is a real gem of a tool.


----------



## rigoletto@ (Mar 22, 2017)

Hello,

How can I force ports-mgmt/synth to rebuild all installed packages without the need to use `synth force` with every package?

Thanks!


----------



## marino (Mar 22, 2017)

Build the repository and then use pkg(8) commands.
Whenever you talk about installed package, you're actually taking about pkg(8).  Synth just builds them.


----------



## basbebe (Apr 12, 2017)

I'm stuck getting to compile the new pecl-redis port to compile against PHP 7.
I'm trying to build it in a master where no PHP is installed.
But in my `LiveSystem-make.conf` I have

```
DEFAULT_VERSIONS+= php=7.1
```
But when I try to install the build in one of my jails which has PHP installed, png pkg wants to uninstall PHP7 extensions and install pecl-redis together with php56.

What am I missing?


----------



## Chris_H (Apr 12, 2017)

Just a hunch; But do you have _anything_ defined in the make.conf(5) on the jail(8).
If you _do_, it must not be 7.x. If you _don't_, it's picking the _default_ for/from the ports framework.

HTH

--Chris


----------



## basbebe (Apr 12, 2017)

Chris_H
This is my `/etc/make.conf` ind the jail:

```
WRKDIRPREFIX=        /var/ports
DISTDIR=        /var/ports/distfiles
PACKAGES=        /var/ports/packages
INDEXDIR=        /var/ports
WITH_OPENSSL_PORT=      yes

DEFAULT_VERSIONS= php=7.1
DEFAULT_VERSIONS+= mysql=10.1m
```

This is `/usr/local/etc/synth/LiveSystem-make.conf` in the base system:

```
#MAKE_JOBS_UNSAFE=yes
DEFAULT_VERSIONS= mysql=10.1m
DEFAULT_VERSIONS+= ssl=libressl
DEFAULT_VERSIONS+= php=7.1
DEFAULT_VERSIONS+= python3=3.6
```


----------



## Chris_H (Apr 12, 2017)

It's a dependency factor then. I have found that Synth is _quite_ pedantic about this. Which frustrated me at first,
but have now become _quite_ grateful for it. I have no idea what you're building against. So I can only assume
that something in the list either includes php5x, or, perhaps more likely; that you already have the php5x-extensions
installed, and since png was built against it previously, it will call upon it again. Have you looked at whether the php5x
extensions asked for png? Or was png built with a php5x-extension option? I know this much, looking at php56-extensions;
`IGNORE_WITH_PHP=   70` is defined -- meaning; if you already have the lang/php56-extentions installed,
and you ask Synth to build php7, which wants any extensions. Synth will _rightfully_ bail. Based on the evidence
in the php56-extensions Makefile.
My suggestion;
`pkg delete lang/php56-extensions`
Also possibly lang/php56, if it's already installed. It doesn't look to me like they'll _both_
live in harmony (php56 && php7x).
Then attempt this process again. Which _may_ require re-building your pkg(8) repo.

HTH

--Chris


----------



## basbebe (Apr 12, 2017)

Chris_H :
I just realized that autocorrect made "png" out of "pkg". Sorry for the misunderstanding – I corrected it in my post.

As I said I don't have any PHP versions installed on the base system synth is running in.
But in my list of packages I build for all my jails, there also is php56 and php70 (and their extensions)…


----------



## Chris_H (Apr 12, 2017)

basbebe said:


> Chris_H :
> I just realized that autocorrect made "png" out of "pkg". Sorry for the misunderstanding – I corrected it in my post.


Ahh. Thanks.


basbebe said:


> As I said I don't have any PHP versions installed on the base system synth is running in.
> But in my list of packages I build for all my jails, there also is php56 and php70 (and their extensions)…


IMHO I think this will simply be an exersize in foot shooting. Meaning; Synth will dutifully attempt to grant your request.
But will _likely_ find during the build session, that _both_ php56 && php7x can't be fully satisfied, and will
simply fail on some of your list entries. I think you have to make a choice between php56, or php7x.
That's what _I_ would do, if I were in your situation. 

Good luck!

--Chris


----------



## basbebe (Apr 12, 2017)

Chris_H said:


> Ahh. Thanks.
> 
> IMHO I think this will simply be an exersize in foot shooting. Meaning; Synth will dutifully attempt to grant your request.
> But will _likely_ find during the build session, that _both_ php56 && php7x can't be fully satisfied, and will
> ...


Thanks for the input!
I thought adding php to make.conf would be plenty enough.
I'll try this tomorrow (removing all but one php version from my list) and come back to report.


----------



## basbebe (Apr 13, 2017)

Chris_H 
You were right!
The make.conf and the order of the ports in the list doesn't make any difference in this case.
I removed all but one php version in the list and then it worked.
Thanks!


----------



## hrenznaet (Apr 15, 2017)

Dear synth developer, could you please make synth finally be a replacement for portmaster by making it first scan through make configs of the ports to be built and if there is either no saved config or saved config is outdated (new options appeared, some options got obsoleted) - it would run a series of make config prompts prior to starting the building queue.
I guess that would need to be run recursively, since checking new options may increase the build queue and thus the new port-dependencies need to also be scanned for make config first and they also may increase the build queue... and so on.
Unfortunately, I still have to use portmaster from time to time because synth is just not capable of that.


----------



## hrenznaet (Apr 15, 2017)

Also, synth failed to build graphics/dri for me:

```
In file included from buffer.c:36:
In file included from ./va_private.h:35:
/usr/local/include/va/va_backend.h:33:10: fatal error: 'linux/videodev2.h' file not found
#include <linux/videodev2.h>
^
In file included from config.c:35:
In file included from ./va_private.h:35:
/usr/local/include/va/va_backend.h:33:10: fatal error: 'linux/videodev2.h' file not found
#include <linux/videodev2.h>
^
In file included from context.c:37:
In file included from ./va_private.h:35:
/usr/local/include/va/va_backend.h:33:10: fatal error: 'linux/videodev2.h' file not found
#include <linux/videodev2.h>
^
1 error generated.
gmake[5]: *** [Makefile:689: buffer.lo] Error 1
gmake[5]: *** Waiting for unfinished jobs....
1 error generated.
1 error generated.
gmake[5]: *** [Makefile:689: context.lo] Error 1
gmake[5]: *** [Makefile:689: config.lo] Error 1
gmake[5]: Leaving directory '/construction/xports/graphics/dri/work/mesa-17.0.3/src/gallium/state_trackers/va'
gmake[4]: *** [Makefile:606: all-recursive] Error 1
gmake[4]: Leaving directory '/construction/xports/graphics/dri/work/mesa-17.0.3/src/gallium'
gmake[3]: *** [Makefile:860: all-recursive] Error 1
gmake[3]: Leaving directory '/construction/xports/graphics/dri/work/mesa-17.0.3/src'
gmake[2]: *** [Makefile:651: all] Error 2
gmake[2]: Leaving directory '/construction/xports/graphics/dri/work/mesa-17.0.3/src'
gmake[1]: *** [Makefile:648: all-recursive] Error 1
gmake[1]: Leaving directory '/construction/xports/graphics/dri/work/mesa-17.0.3'
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1
Stop.
make: stopped in /xports/graphics/dri
```
While portmaster built it just fine for me.


----------



## talsamon (Apr 15, 2017)

see  PR 218642 ( or set `VAAPI` to off)..`Portmaster` does not recognize all errors. `Poudriere` has another error message and hangs during compilation.


----------



## talsamon (Apr 15, 2017)

Following the PR I could successful compile it with `poudriere`
if I add 
	
	



```
VAAPI_BUILD_DEPENDS=    ${LOCALBASE}/include/linux/videodev2.h:multimedia/v4l_compat
```
 to the Makefile.
(But no guarantee. I am not clear about the `VAAPI_LDFLAGS` question in the  PR).


----------



## basbebe (Apr 16, 2017)

I'm having two problems:

www/nginx-devel fails for me, I don't know why:


```
[…]

 /usr/local/include/luajit-2.0  -I objs  -I src/http  -I src/http/modules  -I src/http/v2  -I /construction/xports/www/nginx-devel/work/ngx_devel_kit-0.3.0/src  -I /construction/xports/www/nginx-devel/work/ngx_devel_kit-0.3.0/src  -I /construction/xports/www/nginx-devel/work/ngx_devel_kit-0.3.0/objs  -I objs/addon/ndk  -o objs/addon/src/ngx_http_lua_ssl_ocsp.o  /construction/xports/www/nginx-devel/work/lua-nginx-module-0.10.8/src/ngx_http_lua_ssl_ocsp.c
/construction/xports/www/nginx-devel/work/lua-nginx-module-0.10.8/src/ngx_http_lua_ssl_ocsp.c:493:15: error: no member named 'tlsext_status_expected' in 'struct ssl_st'; did you mean 'tlsext_status_type'?
    ssl_conn->tlsext_status_expected = 1;
              ^~~~~~~~~~~~~~~~~~~~~~
              tlsext_status_type
/usr/local/include/openssl/ssl.h:864:6: note: 'tlsext_status_type' declared here
        int tlsext_status_type;
            ^
1 error generated.
*** Error code 1

Stop.
make[2]: stopped in /construction/xports/www/nginx-devel/work/nginx-1.12.0
*** Error code 1

Stop.
make[1]: stopped in /construction/xports/www/nginx-devel/work/nginx-1.12.0
*** Error code 1

Stop.
make: stopped in /xports/www/nginx-devel



--------------------------------------------------
--  Termination
--------------------------------------------------
```

*EDIT
*
It was Lua – I removed it from the ports options and now it works…

---

And:

I'm trying to automate this command with cron:
`portsnap cron update && synth just-build /usr/local/etc/synth/jail_ports.txt && synth prepare-system && ezjail-admin update -P`

This is my `/usr/local/etc/synth/LiveSystem-environment`:

```
TERM=dumb
```
I also tried `TERM=xterm`

But in the email following the cron execution I still always get:

```
[…]

Building new INDEX files... done.
Please define TERM in environment first and retry.
Please define TERM in environment first and retry.

[…]
```

---

*UPDATE
*
I'm now also having a problem with developing/pecl-judy being ignored:


```
[…]

--------------------------------------------------------------------------------
--  Phase: package
--------------------------------------------------------------------------------
===>  Building package for pecl-judy-1.0.2_1
pkg-static: Warning: @exec is deprecated, please use @[pre|post][un]exec
file sizes/checksums    [10]: . done
packing files           [10]: . done
packing directories      [0]: . done



--------------------------------------------------
--  Termination
--------------------------------------------------
```


----------



## talsamon (Apr 16, 2017)

For www/nginx-devel see  PR 218595.


----------



## abishai (Apr 16, 2017)

hrenznaet said:


> Dear synth developer, could you please make synth finally be a replacement for portmaster by making it first scan through make configs of the ports to be built and if there is either no saved config or saved config is outdated (new options appeared, some options got obsoleted) - it would run a series of make config prompts prior to starting the building queue.
> I guess that would need to be run recursively, since checking new options may increase the build queue and thus the new port-dependencies need to also be scanned for make config first and they also may increase the build queue... and so on.
> Unfortunately, I still have to use portmaster from time to time because synth is just not capable of that.


I asked him before. The answer was about my 'outdated portmaster thinking', so I use portmaster as well


----------



## talsamon (Apr 17, 2017)

The error with devel/pecl-judy is a little
bit strange cause there is no pkg-plist and the Makefile contains no `@exec`.
It installs with port and `poudriere` but throws the same warning.
I guess this is a `portlint` error.
I opened PR 218695.
*Edit: *seems it is caused by /usr/ports/Mk/Uses/php.mk (line 278-284) and seems misinterpreted by `synth`..


----------



## talsamon (Apr 18, 2017)

This may be a general question. If `@unexec` and `@exec`
in the pkg-plist files and Makefiles throws an error, more ports may affected.
~200 ports uses still this keywords (among them x11-toolkits/gtk20, x11/gdm, security/tripwire, www/nginx, security/rkhunter. some of the mod_* ports). Also bsd.tex.mk, bsd.qt.mk and php.mk.
At least in the *.mk files it should changed (if failure or not, it
is deprecated)..
I don't use `synth`, but this is a question to all `synth` users, if there are more failures like this.


----------



## basbebe (Apr 18, 2017)

talsamon said:


> The error with devel/pecl-judy is a little
> bit strange cause there is no pkg-plist and the Makefile contains no `@exec`.
> It installs with port and `poudriere` but throws the same warning.
> I guess this is a `portlint` error.
> ...


Turns out it doesn't build because it only builds with php56. I have only php70 in my list.

```
~ # ❯❯❯ cat /var/log/synth/03_ignored_list.log
00:00:13 devel/pecl-judy: cannot be installed: doesn't work with lang/php70 port (doesn't support PHP 7.0 7.1)
```

But I still don't understand the problem with `TERM` not being defined…


----------



## talsamon (Apr 18, 2017)

Maybe it is `ncurses`. Try disable `ncurses`.


----------



## basbebe (Apr 18, 2017)

talsamon said:


> Maybe it is `ncurses`. Try disable `ncurses`.


Already did, thanks though.


----------



## marino (Apr 18, 2017)

basbebe said:


> But I still don't understand the problem with `TERM` not being defined…



The error is about the host environment, not the jail environment.
In other words, add "env TERM=xterm" to the command.

But given the cron command above, the entire thing looks wrong.
Create a dedicated "cron" profile, once that doesn't have ncurses enabled, and use "env SYNTHPROFILE=newprofile".  See the man page for environment variables that are used.

If you're still seeing that TERM is required even with a profile that doesn't use ncurses, that may be a recently introduced bug.  But lots of people use synth with cron and nobody else complained of a recent bug involving TERM.


----------



## basbebe (Apr 18, 2017)

marino said:


> The error is about the host environment, not the jail environment.
> In other words, add "env TERM=xterm" to the command.
> 
> But given the cron command above, the entire thing looks wrong.
> ...


I already have ncurses disabled for the default profile (LiveSystem).
I thought that that `LiveSystem-environment` file (which of course is in the host environment) would do exactly the same like `env TERM=xterm"` in the command…?


----------



## marino (Apr 18, 2017)

LiveSystem-environment is for the JAIL.  The term is missing from the HOST environment.

the immediate fix is prefix all synth commands with "env TERM=dumb".
It doesn't see to care about ncurses any more, it wants TERM defined always.

edit:
May sure you remove TERM from the environment profile.
The environment file is only used by < 1% of synth users.  You shouldn't need it all unless you've got a proxy or something.


----------



## nickednamed (Apr 24, 2017)

Hi all,

First, thank you jrmarino. Synth is working well for me. I have a question for which I'm sure there is an answer, but I can't get my head round it.

I'm tracking the quarterly pkg repo, and the quarterly ports repo, and I have the "Fetch prebuilt packages" set to "true" because I don't want to be building too often (running FreeBSD on a laptop), but I seem to be rebuilding things over and over.

For example, over the last few weeks the security/nss, and security/ca_root_nss ports have been updated multiple times, and each time `synth status` tells me I need to Update these two ports, and Rebuild around 94 others (sometimes more, sometimes less), including a few big programs like www/firefox, editors/libreoffice, print/texlive-texmf and others (some of which are build only dependencies).

From what I understand (I'm really nothing more than a casual user) when a pkg is upgraded all dependant ports/pkgs must be rebuilt. Fair enough.

But I seem to remember ports-mgmt/pkg simply reinstalling certain packages whose dependencies had been updated. Is this possible with Synth / pkg?

Is there some way to minimise the rebuilding of already installed, and not updated programs?


Also, as a sperate issue, after running `synth install security/nss security/ca_root_nss`, then `pkg upgrade -r Synth` (just to check) it shows that one of the two still needs updating:
	
	



```
Installed packages to be UPGRADED:
   ca_root_nss: 3.30.1 -> 3.30.2 [Synth]
```

Is this the expected behaviour?

EDIT: OK, so I looked into it more, and I guess this is the expected behaviour.  pkg will not reinstall rebuilt packages unless options or versions have changed. Now using `pkg upgrafe -rf Synth`.


----------



## hrenznaet (Apr 29, 2017)

I've tried synth as everyone was advertising synth (or poudrierre) over portmaster.
I don't know why people advise it, because in my opinion portmaster wins over synth.

Synth is not even able to invoke 'make config' when needed. So synth users have to use portmaster for that anyways. If I have to have and have to use portmaster anyways - why would I need synth?
It ignores ports that require license confirmation and skips ports dependent on those. Why not just ask the user's prompt at the start of the building queue?
It fails dependency checks. It doesn't explain anything on that, it just says it failed dependency checks.
Synth wastes time and resources on useless routines: if one runs 'synth upgrade-system' twice in a row and both times some port(s) fail - the 2nd run's queue will consist only of failing ports.
If a queue consists only of ports that are ignored/skipped/failed - there is no need to recursively rescan whatever synth scans and rebuild whatever db it rebuilds. But synth still does that.

One might think that the above issues are just bugs and missing features (which software has no bug?), but looks like it's author is against those features, so it becomes a political issue.

Sometimes synth wins over portmaster when port maintainer screwed up something and the port can't be built using portmaster but builds with synth. But there are also the cases when the opposite takes place.

That being said - I still use synth, but I don't really know why.


----------



## marino (Apr 29, 2017)

Everything you've written is a product of ignorance about the topic.


> If I have to have and have to use portmaster anyways - why would I need synth?


No.  If you have to "make config" you don't need portmaster, you just run "make -C <portpath> config".  So that's blatently false.


> It ignores ports that require license confirmation


That's the port system doing that.  I means you failed to pre-accept the license in your profile make.conf.  Synth is doing as instructed.  If you want it to do something different, you have to provide instruction.


> Why not just ask the user's prompt at the start of the building queue?


Why not preapprove the license?  It's a one time thing.  (or disable licence checks, see next post)  The premise is NEVER interact.  Once you understand that synth intentionally does not stop in the middle of a run (a glaring issue of both portmaster and portupgrade), design decisions may make more sense.


> It fails dependency checks. It doesn't explain anything on that, it just says it failed dependency checks.


There are this things called "logs".  Yes, it's logged.  Yes this information is on the man page.  Yes, there's a way to show this information on the screen in real time (this is also explained in the man page).


> f one runs 'synth upgrade-system' twice in a row and both times some port(s) fail - the 2nd run's queue will consist only of failing ports.


What does that even mean?  Most people want the queue to consist of the missing ports.  Are you saying it should rebuild EVERYTHING on the second try?  If you want that (again, most people would not), then use a build command that will force the rebuild.


> If a queue consists only of ports that are ignored/skipped/failed - there is no need to recursively rescan whatever synth scans and rebuild whatever db it rebuilds. But synth still does that.


There's no db.  Nothing is cached.  Any change to the ports system (e.g. option change, or ports tree update) would invalidate such a db.  As synth doesn't know if such a change occurred, it assumed is did.


> One might think that the above issues are just bugs and missing features...


Actually, one would think this person hasn't read the manual yet speaks on topics they are clearly over their head with.
With regard to "missing features", it's not missing if they were thoughtfully and intentionally excluded.  You may disagree with the decision, just as I disagree with your terminology.


Congrats, you managed to troll me into a response.

All of this, and the previous post, as been hashed out many timed in this very thread.


----------



## marino (Apr 29, 2017)

by the way, anybody that doesn't want licensing to ever block the build, put this in /usr/local/etc/synth/<profile>-make.conf:

```
DISABLE_LICENSES=yes
```

Similarly with packages, to prevent package restrictions, put this:

```
FORCE_PACKAGE=yes
```

Again, these are both handled by the ports tree.   The behavior is not coming from Synth.


----------



## talsamon (Apr 29, 2017)

I don't really want join the discussion.
But I don*t understand why
`portmaster` is able to call `dialog4ports` and
`synth` not. And you refuse to code a few lines that `synth` are able to do this (or something similar with `dialog` or a "macro" with `make -C /usr/ports/$1 config` or whatever)..
If ten ports changes the options, `synth` interrupts ten times....


----------



## fernandel (Apr 29, 2017)

talsamon said:


> I don't really want join the discussion.
> But I don*t understand why
> `portmaster` is able to call `dialog4ports` and
> `synth` not. And you refuse to code a few lines that `synth` are able to do this (or something similar with `dialog` or a "macro" with `make -C /usr/ports/$1 config` or whatever)..
> If ten ports changes the options, `synth` interrupts ten times....


When I did start with Synth from the first vesion I also thought that Synth interrupts you whatever times but not. If you are looking a little more above what messages are there you will saw all ten changes for example.


----------



## talsamon (Apr 29, 2017)

Ok. I don't follow all changes with synth. Maybe, so I 
misinterpreted some of the posts. If it is so, then sorry for it.


----------



## hrenznaet (May 1, 2017)

marino said:


> Everything you've written is a product of ignorance about the topic.
> 
> No.  If you have to "make config" you don't need portmaster, you just run "make -C <portpath> config".  So that's blatently false.


That's idiotic. Why would I make config a single port? I have a lot of ports, I need to work with them as a bunch, not with each one separately.



marino said:


> That's the port system doing that.  I means you failed to pre-accept the license in your profile make.conf.  Synth is doing as instructed.  If you want it to do something different, you have to provide instruction.


I don't need your explanations. Your util doesn't ask me whether I'd like to accept the license, which in my opinion a port managing tool should do.
I don't care if you think otherwise, I'm expressing my opinion about why your util sucks.
Feel free to ignore it, as you already got used to.



marino said:


> Why not preapprove the license?  It's a one time thing.  (or disable licence checks, see next post)  The premise is NEVER interact.  Once you understand that synth intentionally does not stop in the middle of a run (a glaring issue of both portmaster and portupgrade), design decisions may make more sense.


Because I might not want to preapprove licenses.
In my opinion a port updating tool should be able to show a license agreement prompt.
I also don't like the premise of never interact, I'd rather see the tool to run all the required interaction before it starts to build so that I could just forget about it later.
Some interaction is required.
If some people prefer to have absolutely no interaction - I think it should be done via a command key.
Vice versa (automatic by default + use a key for interactive mode) would also be acceptable (though not optimal, IMO).



marino said:


> There are this things called "logs".  Yes, it's logged.  Yes this information is on the man page.  Yes, there's a way to show this information on the screen in real time (this is also explained in the man page).


And that was the quote from /var/log/synth/04_skipped_list.log.
I probably should have looked at the exact logs of those skipped ports instead of 04_skipped_list.log, but I'd like to have a short summary on each skipped port in a single file.



marino said:


> What does that even mean?  Most people want the queue to consist of the missing ports.  Are you saying it should rebuild EVERYTHING on the second try?  If you want that (again, most people would not), then use a build command that will force the rebuild.
> There's no db.  Nothing is cached.  Any change to the ports system (e.g. option change, or ports tree update) would invalidate such a db.  As synth doesn't know if such a change occurred, it assumed is did.


No, what I meant is that if a run's result is that 0 ports got built - there's no need to do some steps, like recursively scanning ports tree. Because nothing got modified.



marino said:


> Actually, one would think this person hasn't read the manual yet speaks on topics they are clearly over their head with.
> With regard to "missing features", it's not missing if they were thoughtfully and intentionally excluded.  You may disagree with the decision, just as I disagree with your terminology.


I know some of the points were discussed before, I also do disagree with some of your decisions, I just expressed my opinion as someone else might be too lazy to try your util to get a gist of it, but might be curious about because quite very many people actively advertised your util as if it's something awesome.
To be fair, I also had a tiny hope that you, as a developer, might see that more and more people nag/complain about the same missing features that you 'thoughtfully and intentionally excluded' and maybe you'd change your mind, but reading your response I see an arrogant non-conformist behind it.
Well, that's understandable, because we don't pay you for what we ask and thus we are in no position to demand anything, yet we still have our right to just nag/whine, sorry.


----------



## marino (May 1, 2017)

hrenznaet said:


> That's idiotic. Why would I make config a single port? I have a lot of ports, I need to work with them as a bunch, not with each one separately.



FACT 1: The vast majority of ports don't need configuring.  The default options, the ones that the freebsd cluster uses are the ones desired
FACT 2: Synth runs if all the pre-saved options are valid.  Thus, once they are saved, they are known to be good.  Once that's not true, synth will not start a new run until they are fixed.
FACT 3: You have  a list of ports that you want built, either manually created or created from pkg(8) queries

Thus, you *should* know which of the very few ports you are build need non-standard options and setting those individually is not a burden.  In the worse case, where you are one of those few people that have the idea you MUST set every ports options, you can either run your list through a shell script or use another tool to the options recursively which you need to do ONCE.  Then if options change in the future, you can set it individually when synth notifies the options are obsolete.

Not to mention FACT 4 where synth uses the existing options set by ports/portmaster if you want.

conclusion: You've portrayed this as some necessary missing feature which actual use proves is anything but.



> I don't need your explanations. Your util doesn't ask me whether I'd like to accept the license, which in my opinion a port managing tool should do.
> I don't care if you think otherwise, I'm expressing my opinion about why your util sucks.


The util already has your input on the license, through the make.conf file.
It was YOUR failure to set the license or DISABLE licenses.
Are you one of these people that blames everyone and everything else for your own fails?  Seems like it.

conclusion: Synth has decided that the information it was given through configuration and specialized make.conf and will not pester you with "Are you sure" garbage questions.  It will assume that you are sure about the configuration that YOU set.



> Because I might not want to preapprove licenses.
> In my opinion a port updating tool should be able to show a license agreement prompt.



Like options, this is a one-time set and forget.
It is not outlandlish to expect you to pre-approve the VERY FEW licenses that require explicit approval.
These have got to be in sub 0.5% of the tree level.  So it's another mole hole you've made into a mountain.



> No, what I meant is that if a run's result is that 0 ports got built - there's no need to do some steps, like recursively scanning ports tree. Because nothing got modified.


 I sufficiently explained why there is no cache of previous runs.



> I know some of the points were discussed before, I also do disagree with some of your decisions, I just expressed my opinion as someone else might be too lazy to try your util to get a gist of it, but might be curious about because quite very many people actively advertised your util as if it's something awesome.


There's no antidote against laziness.



> To be fair, I also had a tiny hope that you, as a developer, might see that more and more people nag/complain about the same missing features that you 'thoughtfully and intentionally excluded' and maybe you'd change your mind, but reading your response I see an arrogant non-conformist behind it.



What, like all 2 of you?
There is one guy that thinks synth should automatically create save option files whenever a port's default options are accepted so that in the future if the options change, synth will alert so that port can be reviewed.  That was something to consider because it has a small bit of legitimacy to the use case, but only a single person requested it.

What you're doing is trying to turn synth back into portmaster so my response is, "just keep using portmaster if that's what you want".
Contrary to what you believe, there's isn't a bunch of people saying these features are so important that their unavailability is the sole reason they don't use Synth.
And finally, I don't care if you use it or not.  I only care when you spread fud and misinformation (albeit without intention) about a tool that you clearly haven't mastered.


----------



## hrenznaet (May 1, 2017)

marino said:


> FACT 1: The vast majority of ports don't need configuring.  The default options, the ones that the freebsd cluster uses are the ones desired


That's not a fact, that's some b/s.
I, as a user, decide which options I'd like the port to have. It's my choice.
Fact is that it's not always necessary to configure ports for minimal functionality.



marino said:


> FACT 2: Synth runs if all the pre-saved options are valid.  Thus, once they are saved, they are known to be good.  Once that's not true, synth will not start a new run until they are fixed.


Sorry, I do not understand. What's a 'pre-saved' option? Did you mean default options values for a port?
Even default options sometimes lead to port being unable to get built, because ports system is just a bin and maintainers can toss there whatever code they like and it may be that buggy that it doesn't compile. I regularly file issue reports about such ports.



marino said:


> FACT 3: You have  a list of ports that you want built, either manually created or created from pkg(8) queries
> 
> Thus, you *should* know which of the very few ports you are build need non-standard options and setting those individually is not a burden.  In the worse case, where you are one of those few people that have the idea you MUST set every ports options, you can either run your list through a shell script or use another tool to the options recursively which you need to do ONCE.  Then if options change in the future, you can set it individually when synth notifies the options are obsolete.


Yeah, looks like I'm one of _those_ people. I have 977 packages and each was built from ports and for many of the ports I have non-default make configs.
I'm also not going to do anything individually for ports with obsolete options, I prefer to work only with sets of ports (_read next about portmaster_).



marino said:


> Not to mention FACT 4 where synth uses the existing options set by ports/portmaster if you want.


That's the most important thing, that you, unfortunately, don't seem to take into account: we don't live in a stasis, the time flows and the ports in the repository do evolve. And they get new config options and lose the old ones.
So from time to time I have to re-configure the ports that were configured once already. And that's normal.
Unlike synth, portmaster shows make config screen once it figures out that my current config for a port is outdated.
Portmaster is even smart enough as to scan all ports configs prior to starting upgrading them, so currently I run portmaster first to check if there any new configs that require my attention, then I have to answer 'nay' to the prompt that asks to confirm the upgrade to the following ports and run synth.

I would use synth way less frequently (only for the ports that portmaster fails to build) if synth didn't have such an attitude that it has to re-build every port that was built outside of it. I hate that.



marino said:


> conclusion: You've portrayed this as some necessary missing feature which actual use proves is anything but.






marino said:


> The util already has your input on the license, through the make.conf file.
> It was YOUR failure to set the license or DISABLE licenses.





marino said:


> Like options, this is a one-time set and forget.
> It is not outlandlish to expect you to pre-approve the VERY FEW licenses that require explicit approval.
> These have got to be in sub 0.5% of the tree level.  So it's another mole hole you've made into a mount.
> I sufficiently explained why there is no cache of previous runs.



No, that's some misunderstanding on your side.
make.conf provides a way to override future prompts with a pre-defined decision.
What if 2 ports have the same license and I'd like to accept it for port A and reject for port B?
You could say that it is some idiocy, while it's actually not at all: port B's older version may have not required to accept a license, but newer versions do.
I might want say 'screw you, dear developer, I'll wait for a fork of the old version', but your approach doesn't leave me such freedom.
You kill freedom with your approach.



marino said:


> conclusion: Synth has decided that the information it was given through configuration and specialized make.conf and will not pester you with "Are you sure" garbage questions.  It will assume that you are sure about the configuration that YOU set.


Again, I don't want to configure everything proactively, I prefer minimal interaction to take place when needed.
Why do I have to check the options prior to upgrading? Why can't the machine do this work for me and ask for my interaction when it needs the human to make some decision? I want that. I want my life to be simpler rather than more complex in such things.



marino said:


> There's no antidote against laziness.


There is. It's automation.



marino said:


> What, like all 2 of you?
> There is one guy that thinks synth should automatically create save option files whenever a port's default options are accepted so that in the feature if the options change, synth will alert so that port can be reviewed.  That was something to consider because it has a small bit of legitimacy to the use case, but only a single person requested it.


There are more users of synth than the commenters in this topic. There are places in the internets other than this forum where users discuss synth (irc, for example).
Forums are not very useful for discussion, because no one is going to read those tens of forum topic's pages just to get familiar with what already got discussed.
Proper way for development is to have an issue tracker for each project.
Preferably on a centralized host like github where lots of users already have an account (because otherwise the project will get less attention and feedback as there is the laziness of registering new account vs using an existing one on a well-familiar centralized system).
In an issue tracker with a sane number of non-finished tickets it's quite easy to find whether feature X was discussed already so that more users could voice out.

But I don't know why I even explain these basics to you, as it's quite clear that you have a quite hostile attitude towards users of your util and not interested in all that.



marino said:


> What you're doing is trying to turn synth back into portmaster so my response is, "just keep using portmaster if that's what you want".
> Contrary to what you believe, there's isn't a bunch of people saying these features are so important that their unavailability is the sole reason they don't use Synth.
> And finally, I don't care if you use it or not.  I only care when you spread fud and misinformation (albeit without intention) about a tool that you clearly haven't mastered.


I'm not some crusader, I don't fight for making X be like Y.
I just want max comfort from the use of the tools I interact with.
The comfort in that sense is the balance between ease of use and functionality/configurability.

In my opinion, synth has only 2 strong points over portmaster:
- it can build ports in parallel (which is supposed to fasten the build process);
- afaiu, it builds ports in a 'clean environment' - I don't really value this feature very much, as it makes the the building process longer, while it usually is needed only when some conflicts across ports appear.

portmaster has strong points over synth:
- it knows when to show make config screens and it does show them;
- it builds 1 port faster than synth does (because unlike synth it doesn't build it in a 'clean environment';
- it shows accept license prompts (but does it in a stupid [all make config screens are shown *before* the global building process starts, but accept license prompts appear only when the building queue reaches the current port] and buggy [_-a_ key makes it hang up in background whenever portmaster reaches a port that has accept license prompt] way in some cases).

I want all of those strong points in 1 utility. I don't care about its name.


----------



## marino (May 1, 2017)

hrenznaet said:


> That's not a fact, that's some b/s.
> I, as a user, decide which options I'd like the port to have. It's my choice.
> Fact is that it's not always necessary to configure ports for minimal functionality.



It's an easily proven fact.
Given access to your machine, I could determine exactly which ports you have customized to be different than default options.  I am quite confident that number is low.
Don't say "BS" without providing any statistics.  You're talking to somebody very familiar with the 28,000 ports the collection.  As a 3-year committer to ports and a member of pkgsrc for 2 years longer than that, you're talking against a lot of experience.  What resume are you relying on to back up your facts?



> Sorry, I do not understand. What's a 'pre-saved' option? Did you mean default options values for a port?



No. When you have the configure screen on a port and select "ok", it saves your options to a file.  At that point it will never bring the screen up again unless the options change.



> Even default options sometimes lead to port being unable to get built, because ports system is just a bin and maintainers can toss there whatever code they like and it may be that buggy that it doesn't compile. I regularly file issue reports about such ports.



Exactly.  Non-default options are very often untested and don't work.  It's actually risky to customize options.  This is the norm, not an exception to the rule.



> Yeah, looks like I'm one of _those_ people. I have 977 packages and each was built from ports and for many of the ports I have non-default make configs.
> I'm also not going to do anything individually for ports with obsolete options, I prefer to work only with sets of ports (_read next about portmaster_).



"Many" could be 20.  20 of 977 is manageable especially when it's a one-time "set and forget".
Using the  redundant-opt-files.sh script makes it trivial to identify these and save them on a list.



> That's the most important thing, that you, unfortunately, don't seem to take into account: we don't live in a stasis, the time flows and the ports in the repository do evolve. And they get new config options and lose the old ones.



Huh?  As I already said, if the options change, Synth stops and tells you that and makes you fix them.  How is what you said relevant?



> So from time to time I have to re-configure the ports that were configured once already. And that's normal.



right, normal.  And not that frequent.



> Unlike synth, portmaster shows make config screen once it figures out that my current config for a port is outdated.



WRONG.  Synth tells you when the options are outdated.  You're spreading misinformation.



> Portmaster is even smart enough as to scan all ports configs prior to starting upgrading them, so currently I run portmaster first to check if there any new configs that require my attention, then I have to answer 'nay' to the prompt that asks to confirm the upgrade to the following ports and run synth.



What's your point? Synth does that too.  How do you think it is able to stop and say "fix the options" ?



> I would use synth way less frequently (only for the ports that portmaster fails to build) if synth didn't have such an attitude that it has to re-build every port that was built outside of it. I hate that.



That's just a side effect of not understanding how ports and package really work.



> No, that's some misunderstanding on your side.
> make.conf provides a way to override future prompts with a pre-defined decision.
> What if 2 ports have the same license and I'd like to accept it for port A and reject for port B?



You're beef is with the ports system then.  accepting a license is not a per-port thing, it's a per-license thing.  Reasonably if you accept the license on one port, you accept it for all.  You can control the build of ports in other ways, e.g. DON'T PUT IT ON THE BUILD LIST.



> You could say that it is some idiocy, while it's actually not at all: port B's older version may have not required to accept a license, but newer versions do.
> I might want say 'screw you, dear developer, I'll wait for a fork of the old version', but your approach doesn't leave me such freedom.
> You kill freedom with your approach.



Again, direct your derision where it belongs, the ports collection.  Synth is just following the directives its has been given.



> Again, I don't want to configure everything proactively, I prefer minimal interaction to take place when needed.
> Why do I have to check the options prior to upgrading? Why can't the machine do this work for me and ask for my interaction when it needs the human to make some decision? I want that. I want my life to be simpler rather than more complex in such things.



Nobody said you had to.  The ports system is designed around default options.  If you don't know exactly why you are changing an option, you'd better not change it.  It's not meant for interactive behavior, exactly the opposite.  All interactive ports were removed a couple of years ago and new ones banned.



> Preferably on a centralized host like github where lots of users already have an account (because otherwise the project will get less attention and feedback as there is the laziness of registering new account vs using an existing one on a well-familiar centralized system).
> In an issue tracker with a sane number of non-finished tickets it's quite easy to find whether feature X was discussed already so that more users could voice out.



Synth is developed on Github.



> But I don't know why I even explain these basics to you, as it's quite clear that you have a quite hostile attitude towards users of your util and not interested in all that.



You're the one that started blasting without any real knowledge.  You didn't even know synth is maintained on github with issue tracking and instructions, yet criticized for exactly not having that.  Why should people pay attention with your credibility?



> I'm not some crusader, I don't fight for making X be like Y.
> I just want max comfort from the use of the tools I interact with.
> The comfort in that sense is the balance between ease of use and functionality/configurability.



The tool is what it is.  In the case of requests that have been evaluated are not going to come in, you take the tool on its merits or don't use it.



> In my opinion, synth has only 2 strong points over portmaster:
> - it can build ports in parallel (which is supposed to fasten the build process);
> - afaiu, it builds ports in a 'clean environment' - I don't really value this feature very much, as it makes the the building process longer, while it usually is needed only when some conflicts across ports appear.



It has many more than that.  And if you don't value a clean environment in every case then you really deserve whatever consequences you get.



> portmaster has strong points over synth:
> - it knows when to show make config screens and it does show them;



some consider this a negative.



> - it builds 1 port faster than synth does (because unlike synth it doesn't build it in a 'clean environment';



maybe technically true but it's usually portmaster cheating because portmaster is too dumb to rebuild dependencies so when it's building 1 port it really should be rebuilding several others.   In the true case of single port, the time difference on a modern machine with SSD and tmpfs is negligible.  It's only noticeable on very old hardware with low ram and spinning disks.



> - it shows accept license prompts (but does it in a stupid [all make config screens are shown *before* the global building process starts, but accept license prompts appear only when the building queue reaches the current port] and buggy [_-a_ key makes it hang up in background whenever portmaster reaches a port that has accept license prompt] way in some cases).



And synth makes the user notice the license needs to be accepted which lets the port build in the future.  What's the real difference?



> I want all of those strong points in 1 utility. I don't care about its name.


Well, guess you're going to be perpetually disappointed since poudriere has the same "issues" you think synth has (although poudriere has an option to preconfigure options, but it's not during the run).  So there's no existing tool to give you what you want.  Do what you have to do.


----------



## marino (May 1, 2017)

Okay, let me fix your problem once and all.
The stated problem is that Synth expects ports to be already configured.  It assumes this has been done, but the problem occurs when it has not been done.
Here's a solution for a system that already has packages install.  Copy this to a script and run it:

```
#!/bin/sh

origins=$(pkg prime-origins)

for origin in ${origins}; do
   make -C /usr/ports/${origin} config-recursive
done
```
That's it.  You run that script (as needed, which is rarely) before you run synth.
Alternatively, you could feed a build list of origins into a script if the host system has a different set of packages than desired.
Note that "prime-origins" is a new alias that may have to be copied to pkg.conf from /usr/local/etc/pkg.conf.sample

So that should be simple enough for people that require configuring every port, right?


----------



## hrenznaet (May 2, 2017)

marino said:


> It's an easily proven fact.
> Given access to your machine, I could determine exactly which ports you have customized to be different than default options.  I am quite confident that number is low.
> Don't say "BS" without providing any statistics.  You're talking to somebody very familiar with the 28,000 ports the collection.  As a 3-year committer to ports and a member of pkgsrc for 2 years longer than that, you're talking against a lot of experience.  What resume are you relying on to back up your facts?
> "Many" could be 20.  20 of 977 is manageable especially when it's a one-time "set and forget".
> ...




```
ls /var/db/ports/ | wc -l
     243
pkg info | wc -l
     981
```
243/981=24.8%
I'm quite a newbie and don't really know if /var/db/ports also includes configs identical to default configs for their ports, but that doesn't really matter, it means synth would halt on each of those. This is insane.



marino said:


> hrenznaet said:
> 
> 
> > Unlike synth, portmaster shows make config screen once it figures out that my current config for a port is outdated.
> ...


Caught you on b/s again. How is what I wrote can be wrong? Read carefully. Halting upon every outdated option is stupid and non-comfortable, and it is not the same as showing make config screens.



marino said:


> That's just a side effect of not understanding how ports and package really work.


Then how can synth be used so it won't rebuild the ports built with portmaster? Or it can't and it's just another b/s 'point' of yours?



marino said:


> accepting a license is not a per-port thing, it's a per-license thing.


Sorry, didn't know that. Well, then your solution about pre-specifying a list of licenses to accept is an acceptable solution, thanks.



marino said:


> Synth is developed on Github.


And again I'm sorry I didn't know that. Found the link.



marino said:


> In the case of requests that have been evaluated are not going to come in, you take the tool on its merits or don't use it.


I also have a right to whine about how stupid some of your decisions are in my opinion and how I miss those features 



marino said:


> It has many more than that.  And if you don't value a clean environment in every case then you really deserve whatever consequences you get.


I've mentioned only what I noticed and found valuable (if compared to portmaster).



marino said:


> some consider this a negative.


Well, then I laugh at those _some_.



marino said:


> And synth makes the user notice the license needs to be accepted which lets the port build in the future.  What's the real difference?


The difference is in the level of comfort.



marino said:


> Do what you have to do.


I already do. Suffer and whine.



marino said:


> Copy this to a script and run it:
> So that should be simple enough for people that require configuring every port, right?


No. Because it's a stupid script: it will show make config screen for each of the configurable ports from the list of installed.
While I need to get a bunch of make config screens only for the ports with outdated configs. That's why my current alias for updating ports contains a portmaster call first before calling synth. I know it's a bit barbarian, but that's the only way to get both utils' strong points.

Sorry, that's probably my last reply in this thread, because looks like you aren't going to conform and thus arguing with you is counter-productive.
Sorry if you found some of my accusations to harsh, I'm a newbie but I value my own preferences in which I find comfort. Your preferences are just not for me.


----------



## marino (May 2, 2017)

hrenznaet said:


> ```
> ls /var/db/ports/ | wc -l
> I'm quite a newbie and don't really know if /var/db/ports also includes configs identical to default configs for their ports, but that doesn't really matter, it means synth would halt on each of those. This is insane.
> ```



Yes, the vast majority are identical to default and could be removed.  Why would it "halt on each of those" ?
It only halts when the config is BAD.  The existence of a valid config will not halt it.



> Caught you on b/s again. How is what I wrote can be wrong? Read carefully. Halting upon every outdated option is stupid and non-comfortable, and it is not the same as showing make config screens.


It's wrong because it's wrong.  It lists *ALL* bad configs, not one at a time.
This is a good thing.  Note that neither portmaster nor poudriere detect bad configs, they just take them.  That's unacceptable actually.



> I also have a right to whine about how stupid some of your decisions are in my opinion and how I miss those features


Calling me stupid signals the end of the subtopic.



> No. Because it's a stupid script: it will show make config screen for each of the configurable ports from the list of installed.
> While I need to get a bunch of make config screens only for the ports with outdated configs. That's why my current alias for updating ports contains a portmaster call first before calling synth. I know it's a bit barbarian, but that's the only way to get both utils' strong points.


I can think of 5 ways to make the script a whole of better.  The point is that I did that in about 20 seconds.
If somebody comes up with perfect script, I'll be happy to post in the FAQ with acknowledgement.
You missed the point that if if synth expects the ports to be preconfigured, it's easy to script.
And you really , really missed that it's pretty much an up-front thing.  You don't keep doing it.



> Sorry, that's probably my last reply in this thread, because looks like you aren't going to conform and thus arguing with you is counter-productive.  Sorry if you found some of my accusations to harsh, I'm a newbie but I value my own preferences in which I find comfort. Your preferences are just not for me.



Great!  Again, I don't get paid for each user so I really don't care if you use synth or not.  You paid nothing for it.  I think your "right to complain" is pretty limited.  I'm happy not to hear it anymore, especially after several solutions to you main issues were provided.[/code]


----------



## hrenznaet (May 7, 2017)

'synth upgrade-system' just uninstalled libreoffice for me.
I didn't ask for that, this is ridiculous.
I think it's time to try poudriere.


----------



## cbrace (May 8, 2017)

Can someone suggest a way of upgrading from php56 to php70 using synth?

I tried deleting php56 and all the php56 extensions with pkg, installing php70 with synth, and then installing www/zenphoto/ www/joomla3/ mail/roundcube mail/postfixadmin www/nextcloud databases/phpmyadmin, also with synth.

However, in doing so synth removed php70 and reinstalled php56 and everything all over again.

Thanks for pointing me in the right direction.


----------



## marino (May 8, 2017)

cbrace@,
Did you create /usr/local/etc/<profile>-make.conf ?  (LiveSystem-make.conf by default) ?
IIRC the default version of PHP is 56, so unless you specifically set the version to be 70, php56 is going to be built by default.
If you had that in /etc/make.conf, note that this file is NOT used by Synth.  Each profile has its own, optional, make.conf.


----------



## cbrace (May 9, 2017)

Hi marino 

OK, I created the profile and added this line:


```
DEFAULT_VERSIONS+= php=70
```
and activated the profile in `synth configure`.

But when now I run `synth upgrade-system`, nothing gets rebuilt.

If I I install www/php70, all the installed php-based packages are removed. If I try to reinstall www/joomla3 etc php70 is removed and php56 reinstalled.

I still seem to me missing something here. Any ideas?


----------



## marino (May 10, 2017)

when you say you "created the profile", did you mean you created the *-make.conf file?

My guess is that you messed the file name up.
The way you check is to look at any of the build logs.  There's a section that list out the builds's make.conf and any customized entries should be in that section.  If it's not, the profile makeconf doesn't exist.

So I think you didn't do it correctly.  Any new buildlog will confirm.


----------



## fernandel (May 11, 2017)

cbrace said:


> Hi marino
> 
> OK, I created the profile and added this line:
> 
> ...



For example, I have:
/usr/local/etc/synth/LiveSystem-make.conf 
and it works.


----------



## bhtooefr (May 19, 2017)

For what it's worth, just used synth on my FreeBSD 10.3 system, to install graphics/mesa-libs (due to the port consolidation, after noticing that synth skipped those ports due to them being deleted - and no, I never saw anything in pkg updating about it), and this happened:


```
Installed packages to be REMOVED:
        cairo-1.14.8,2
        tex-web2c-20150521_1
        cups-filters-1.13.5
        gtk-update-icon-cache-2.24.29
        fltk-1.3.3_4
        libglapi-17.0.3
        poppler-0.50.0
        poppler-utils-0.50.0_1
        vim-8.0.0596
        pango-1.40.4
        qt5-gui-5.7.1
        gtk2-2.24.31
        qt5-webkit-5.7.1
        qt5-widgets-5.7.1
        qt5-help-5.7.1
        qt5-assistant-5.7.1
        qt5-printsupport-5.7.1
        virtualbox-ose-5.1.22
        qt5-x11extras-5.7.1
        fldigi-4.0.3
        librsvg2-2.40.16
        ImageMagick-6.9.6.4_2,1
        poppler-glib-0.50.0
        libGL-17.0.3
        gbm-17.0.3
        libEGL-17.0.3
        libGLU-9.0.0_2
        sdl-1.2.15_7,2
        freeglut-3.0.0
        qt5-opengl-5.7.1
        harfbuzz-1.4.6_1
        qt5-quick-5.7.1
        qt5-designer-5.7.1
        qt5-uiplugin-5.7.1
        qt5-uitools-5.7.1
        qt5-linguist-5.7.1
        hpijs-2.1.4_11
        classpath-0.99_3
        flrig-1.3.30

New packages to be INSTALLED:
        mesa-libs: 17.0.4 [Synth]
```

And, no, it didn't reinstall the uninstalled packages that depended on the mesa libraries afterwards, so I'm having to do that manually. Can synth be made to handle that situation more cleanly? (Maybe somehow detect when it's replacing packages that depend on something, and offer to rebuild the depends?)


----------



## marino (May 19, 2017)

I think adding an "upgrade system" wrapper for pkg(8) was ultimately a mistake.
Synth isn't doing that.  Pkg(8) is.
Presumably the local repository is incomplete or getting confused with FreeBSD official repository.
Most of the issues we've seen are people issuing "synth upgrade system" and just trusting every package is flawlessly built and then an unattended upgrade occurs.
Best practice dictates that A) the root user verifies every package was indeed rebuilt without skips before the repositiory is created and B) probably pkg(8) should be called instead of Synth at that point.  The "convenience" command wrapper seems to lead some people astray and I regret adding it at this point.

To circle back: if those packages to be removed were in the local repository, they would be upgraded, not removed.

conclusion: Despite what early document says, probably nobody should use the "upgrade system" command.  Definitely they should not use it for snafu's like fixing mesa libs which is completely messed up so suspicions about issues should be high (meaning do NOT upgrade system unattended which upgrade system does)


----------



## bhtooefr (May 19, 2017)

In this case, it was a synth install graphics/mesa-libs that did this, by the way. (In fact, it was an upgrade-system run that failed (for another unrelated reason) that led me to do that.) But, looks like synth build graphics/mesa-libs would've been the safer way to go.

That said, I feel like the older package management tools were cleaner about this kind of scenario (at least when pkg updating actually told you to do the rebuild, and I don't recall ever seeing packages get deleted unexpectedly with portmaster builds). And, the convenience features _feel_ like they're supposed to do the right thing more automatically than they do...


----------



## marino (May 19, 2017)

The "safe" would be to build everything, then rebuild the repository when everything succeeds.  

As I said, I now feel like the convenience commands were a mistake on my part.  The consequences of misunderstanding what's happening (which is easy to do since pkg(8) is kind of opaque) can be significant.


----------



## rigoletto@ (May 19, 2017)

I was running `synth upgrade-system` at night until the point a package failed to build, and when I waked up pkg(8) was removed half of my installed packages. 

Since then I use `synth prepare-system` to build the upgrades and then run `pkg upgrade` manually to upgrade the system, as I can see what will happen.


----------



## jdp_03 (May 27, 2017)

Is it normal for synth to get into a state where it is only running one builder?  It started with six, but five are now idle.

Since taking the screenshot, below, the job finished and a new one has started on ID 3.  IDs 4 and 6 are showing 'Shutdown'.


----------



## marino (May 27, 2017)

yes, it's completely normal.


----------



## rigoletto@ (May 27, 2017)

Yes, this situation was already discussed here recently.

Basically, when the port being built is a direct or indirect dependency to all other ports, they will wait that one to finish before grab a slot. The slot will be in "Idle" state.

When a slot will not be necessary anymore, due to the dependencies correlation, synth will be in "Shutdown" state.


----------



## jdp_03 (May 29, 2017)

Firstly - many thanks to Marino for Synth, which is excellent.

I have got synth working on my development machine, and can use its repository to update packages on a VM on my laptop.  On my development machine I have a couple of jails which I installed with ezjail.  These require a base jail, which has its own ports tree.  I would like to move to updating the jails using the  Synth repository, and avoid using ports with jails.  Is this possible?


----------



## marino (May 29, 2017)

yes.  A repository is designed for more than one machine.  Your development machine can build a single repository that it, and both jails use.  You just put the repository in a place that the jail's pkg(8) can access, and update the pkg/repos conf files to find the repository.


----------



## jdp_03 (May 29, 2017)

Thanks marino.  I am already using the repository for two machines.  It was the template jail ports stuff that was confusing me a little.  Basically, I can just ignore that, and edit /etc/pkg/FreeBSD.conf in each jail to point to my little webserver which serves up the files.  I went ahead and tried it, and it just worked.


----------



## Wapcaplet (May 31, 2017)

John, are you aware of the breakage of lang/gcc6-aux in 12-CURRENT caused by the 64-bit inode commit?

PR 219667 was created to track the issue, FYI.


----------



## SirDice (May 31, 2017)

jdp_03 said:


> Basically, I can just ignore that, and edit /etc/pkg/FreeBSD.conf in each jail to point to my little webserver which serves up the files.


Don't edit that file, it can and will be overwritten during an update. Instead create files in /usr/local/etc/pkg/repos/.

Disable the FreeBSD repository with FreeBSD.conf:

```
FreeBSD: { enabled: no }
```
Then add your own repositories to /usr/local/etc/pkg/repos/. That way they'll never be modified or overwritten during an update/upgrade.


----------



## hrenznaet (Jun 10, 2017)

I am now a hostage of synth: it screwed something up, so I can't use portmaster anymore as it instantly quits with "===>>> Gathering distinfo list for installed ports   ===>>> Starting check of installed ports for available updates       ===>>> No origin available for devel/gmake-4.2.1 ===>>> Cannot continue ===>>> Aborting update".
How to fix this issue? synth brings more anger than satisfaction and I'd like to replace it.


----------



## marino (Jun 10, 2017)

remove /usr/ports and reinstall it.  The directory is corrupt.  Portsnap does that sometimes.  It's probably giving postmaster problems too.


----------



## stejoo (Jun 19, 2017)

I'm pretty new to synth. Seems like a very nifty tool. I've just installed it on a host, checked `synth configure` and did a `# synth upgrade-system` after syncing the ports tree. A lot of things got rebuild and apparently reinstalled as the ports version. So that went well, but now I'm trying to install a bit of Python software and it states it cannot find the package. Probably because the name changes as it becomes a package. I'll show you:

`# synth install archivers/py-borgbackup
Scanning existing packages.
After inspection, it has been determined that there are no packages that
require rebuilding; the task is therefore complete.
Stand by, recursively scanning 1 port serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Updating Synth repository catalogue...
Fetching meta.txz: 100%    260 B   0.3kB/s    00:01
Fetching packagesite.txz: 100%   29 KiB  29.8kB/s    00:01
Processing entries: 100%
Synth repository update completed. 85 packages processed.
All repositories are up to date.
pkg: No packages available to install matching 'archivers/py-borgbackup' have been found in the repositories
Unfortunately, the system upgraded failed.`

This is after an earlier try, so the dependencies have already been build.
But my question is how should I install archivers/py-borgbackup ?
Because as package it probably becomes archivers/py36-borgbackup and pkg would be unable to find it as py-borgbackup?


----------



## rigoletto@ (Jun 19, 2017)

archivers/py-borgbackup is definitely in the ports tree, however I could not find it at the FreeBD pkg repository (nothing called borgbackup is in there indeed). You can look at /var/log/synth to see build logs and discover what happened.

Alternativelly, you can look for it at your local synth repository using the `pkg search` function.


----------



## marino (Jun 19, 2017)

To understand what is happening, you have to understand what the commands you use actually do.

As an aside: Instead of using `synth upgrade-system`, instead use `synth prepare-system` with `package upgrade`

so what is `synth upgrade-system` doing?
The normal operation of synth is to take a list of ports to build.  This command says "get that list of ports from pkg(8) by querying what packages are already installed".  A better way is to maintain a list of ports as a separate file and just feed that file to synth.  However, nobody has such a list at first (although pkg(8) can be used to generate it off course).

What's going on is that that the port of an installed package got renamed.  The list that pkg(8) creates is in error because the installed py-borgbackup has the old port location and that's what is getting returned.  The bad list is passed to synth.

The fix?

Dump the list yourself with `pkg prime-origins` (or `pkg query -e '%a = 0' '%o'`), edit it and feed it to synth
or remove py-borgbackup package, build it manually via synth, and install it via synth repository.
In any case you'll have to remove the old py-borgbackup.  I'm not sure pkg(8) is smart enough to follow the rename.


----------



## stejoo (Jun 22, 2017)

Thanks for your replies.



lebarondemerde said:


> archivers/py-borgbackup is definitely in the ports tree, however I could not find it at the FreeBD pkg repository.


Yeah, it's only available as a port. That's one of the reasons why I changed to all ports instead of packages on this system. Didn't want to mix and match pkgs and ports together, plu I gain customization.


lebarondemerde said:


> Alternativelly, you can look for it at your local synth repository using the `pkg search` function.


I'll have synth build the port ans see if pkg finds it. 



marino said:


> As an aside: Instead of using `synth upgrade-system`, instead use `synth prepare-system` with `package upgrade`


Thanks, I just tried it and it works. I'll change to doing my updates like that in the future.



marino said:


> so what is `synth upgrade-system` doing?
> The normal operation of synth is to take a list of ports to build.  This command says "get that list of ports from pkg(8) by querying what packages are already installed".  A better way is to maintain a list of ports as a separate file and just feed that file to synth.  However, nobody has such a list at first (although pkg(8) can be used to generate it off course).
> 
> What's going on is that that the port of an installed package got renamed.  The list that pkg(8) creates is in error because the installed py-borgbackup has the old port location and that's what is getting returned.  The bad list is passed to synth.
> ...


Yeah, I had figured out the rename caused the trouble. There will probably be more ports with the same issue out there. 

Seeing as how py[36]-borgbackup isn't installed right now I instructed `# synth just-build archivers/py-borgbackup`. But it determined there are no packages that require rebuilding. So no package py-borgbackup got built.
I checked out /var/synth/live_packages/All/ and there I don't have it there either. Does look like all the dependencies are still there from the previously failed "install".

I would appreciate some more hints in getting the port archivers/py-borgbackup installed through synth. If you require more info from my end let me know and I'll provide.
(Might be a while before I reply because I'll be going on holiday soon.)


----------



## ANOKNUSA (Jun 22, 2017)

stejoo said:


> I would appreciate some more hints in getting the port archivers/py-borgbackup installed through synth.



Sure, here ya go: *don't bother.* Borg is a Python 3 application. The only way to get Borg to build with either Synth or Poudriere is to change the system-wide default Python version to Python 3, and changing the default Python version has the potential to break any Python 2 packages you might use. So while archivers/borg-backup might build after changing the default, other Python packages might not. That's why it's only available as a port.

If Borg is the only Python application you use, then changing the default is no problem; if you use other Python applications, you're probably stuck. I personally just use pre-compiled binary the project offers. You can also install devel/py-pip and install Borg that way.


----------



## hrenznaet (Jun 24, 2017)

marino said:


> remove /usr/ports and reinstall it.  The directory is corrupt.  Portsnap does that sometimes.  It's probably giving postmaster problems too.


I did 'rm -r /usr/ports/* && portsnap fetch extract', but the issue is still there.


----------



## marino (Jun 24, 2017)

I re-read your original post.  It seems that your complaint is with portmaster.
You also say synth "screwed something up" which makes little sense.  Synth is just a repository builder.  All it does is build packages for pkg(8).  It doesn't have the capability to effect portmaster.

I originally read your post as Synth was having an issue as well as portmaster, but if it's only portmaster with the issue, that's out of scope for me.   I have no interest in solving that problem.


----------



## nov1c3 (Jun 29, 2017)

Hello,

First of all, many thanks to marino for Synth! I started using it a couple of weeks ago, and it just perfectly fits my needs.

I'm having a bit of an issue with updating of readline package (recent update from 6.3.8 to 7.0.3). Actually, I believe my issue has nothing to do with Synth, but pkg, however I'd appreciate any hints.

I'm using Synth 1.69 on FreeBSD 11.1-BETA3 (r320268) to build packages for ~20 VMs running FreeBSD.

So I get a list of all packages installed on VMs, feed to synth, which builds and produces txz files, which are then exposed via http to VMs.  Everything is working/building just fine (I think).

Here is the content of /usr/local/etc/pkg/repos/FreeBSD.conf file from one of the VMs:

###
FreeBSD: { enabled: no }

Synth: {
  url: "pkg+http://repos.local.net/pkg/",
  mirror_type: "srv",
  signature_type: "none",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}
###

I have a server with readline-6.3.8_1 installed, which I expected to be updated to 7.0.3 when I execute 'pkg upgrade -r Synth' on that server, however pkg says everything is up to date:

Updating Synth repository catalogue...
Synth repository is up to date.
All repositories are up to date.
Checking for upgrades (0 candidates): 100%
Processing candidates (0 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.

I confirm that readline-7.0.3.txz was sucessfully built by Synth, and it's indeed present under repos.local.net/pkg/All. Web server logs also confirm that the VM does contact the repository:

192.168.10.33 repos.local.net - [29/Jun/2017:13:11:32 +0200] "GET /pkg//meta.txz HTTP/1.1" 304 0 "-" "pkg/1.10.1"
192.168.10.33 repos.local.net - [29/Jun/2017:13:11:32 +0200] "GET /pkg//packagesite.txz HTTP/1.1" 304 0 "-" "pkg/1.10.1"

However, no update is triggered, or meta.txz (packagesite.txz?) somehow doesn't reflect the change of versions?

Thank you.


----------



## marino (Jun 29, 2017)

did you run the `synth rebuild-repository` command before the update command?


----------



## nov1c3 (Jun 29, 2017)

Obviously, I didn't! Thank you very much marino -- the issue is solved.


----------



## rigoletto@ (Jul 4, 2017)

Hi!

I am experiencing a weird behavior I do not know if related with ports-mgmt/synth or something else. It is with sysutils/fusefs-simple-mtpfs, what should install a file called special_simple-mtpfs in /etc/autofs, what is not happening.

The Makefile show what seems to be some "glue" to make that file to be installed in there, and a comment point to PR 193596:


```
.if exists(/etc/autofs)
PLIST_FILES+=    /etc/autofs/special_${PORTNAME}
SUB_FILES+=    special_${PORTNAME}
.endif

post-install:
    (cd ${WRKSRC} && ${COPYTREE_SHARE} \
        "${PORTDOCS}" ${STAGEDIR}${DOCSDIR})
.if exists(/etc/autofs)
    @${MKDIR} ${STAGEDIR}/etc/autofs
    ${INSTALL_SCRIPT} ${WRKDIR}/special_${PORTNAME} \
        ${STAGEDIR}/etc/autofs
.endif
```

I had not tried using `make` directly, however the package (from the FreeBSD repository) install it correctly, and so I am assuming there is no problem with ports-mgmt/poudriere.

Thanks!


----------



## marino (Jul 4, 2017)

what that port doing is bogus.
It's tailoring the port to the local machine.
if autofs is needed, the port should install it, not adjust to its possible presence.

if you want to fake synth out, copy /etc/autofs to a temp location that synth uses, e.g. `/usr/bin/autofs` and modify the makefile accordingly.

I'm not recommending it, just saying how it could be possible to work around this extremely bad practice the port is doing.


----------



## rigoletto@ (Jul 4, 2017)

I will install from packages and copy that file to somewhere else , then I will install from the ports-mgmt/synth repository and copy that file back to /etc/autofs to see if it works this way.

If it works I think it would be better than mess with the Makefile.

Thank you marino.

EDIT: *it worked*.


----------



## Pawtuxet (Jul 5, 2017)

Hi, my apologies if this has been asked before; I'm using ports-mgmt/synth to maintain a local repository of all packages on my system and a number of jails.

My problem is, that many packages that are used in the jails keep vanishing from the repository when it is upgraded, leaving me to identify and manually instruct synth to rebuild these.
Is it possible to maintain a list of ports to keep upgraded, even if they do not appear to be installed? Or flag manually built ports to be kept updated?

Also, thanks for the work you've put into developing synth. It has worked really well for me.


----------



## marino (Jul 5, 2017)

From your description, the best practice is to maintain a text file with a list of ports you want present (a union between all jails and system).
You would not use "prepare-system" which is just a shortcut for creating a build list from current installation.

You can make the first draft of the list with `pkg prime-origins` (or `pkg 'query -e '%a = 0' '%o'`), save result and edit.
Then in the future you build packages by passing synth this file to build.

From your description ("packages keep vanishing"), it sounds like you are building ports one-by-one rather than as a integrated group.  If you had a static list, no packages would be missing because synth would identify obsolete and build them all.  So I can't tell what exactly your process is from your description, but at first blush it sounds it's not the correct one for you tasks.


----------



## Pawtuxet (Jul 5, 2017)

I agree my current approach was not the best. I guess I expected prepare-system to update the entire repository, not just what pkg reported as installed, but reading the docs again it's clear that is what is happening.

My exact process was to first update the ports tree for system `portsnap fetch update` and jails `ezjail-admin update -P`, then run `synth prepare-system`.
Once that had finished, I went through each jail and ran `pkg update` followed by `pkg upgrade`.
Then I would notice that some jails still had packages that needed updating. Specifically those that were not installed on the system.
Querying the repository revealed that these were no longer present and I had to instruct synth to build them again.
Once that was done, I could return to the jails and finish the upgrade.
A couple of months later, rinse and repeat.

I figured I was missing something along the lines of `pkg set -A 0 <package>` to make synth retain ports not installed on the system, but maintaining a list of prime ports is a manageable solution. Especially given the `pkg prime-origins` command, which I wasn't familiar with.

Thanks for the help.


----------



## Mayhem30 (Jul 16, 2017)

I'm new to ports-mgmt/synth (just installed) and previously have been using ports-mgmt/portmaster for years.

My question is - when I run `synth status`, it shows :


```
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
These are the ports that would be built ([N]ew, [R]ebuild, Upgrade):
  N => print/indexinfo
  N => devel/gettext-runtime
...
...
  N => www/nginx
Total packages that would be built: 152
```
Why does Synth want to build all 152 ports when they are already installed?

I guess I just don't understand why "/var/synth/live_packages" is even necessary? (I'm assuming that's where the ports will be compiled in to)

Do I just need to run `synth prepare-system` for now since everything is already installed and running?


----------



## marino (Jul 16, 2017)

because synth is a _repository_ builder.
Your proposed repository has no packages.  The "prepare-system" command is basically telling synth to build (if needed) packages given a list generated by what you have installed currently.  So that list is 152 packages long.  You have zero packages.  152 - 0 = 152.

Moreover, just building packages doesn't change your host system.  At some point you'd have to tell pkg(8) to use the new repository to update the system.   Basically your confusion seems to be caused by a fundamental misunderstanding of what Synth does and how it interacts with pkg(8).  Don't worry, it's a common misunderstanding.


----------



## Mayhem30 (Jul 16, 2017)

marino said:


> At some point you'd have to tell pkg(8) to use the new repository to update the system.



Ok, that clears up the confusion. I've only used PKG to delete ports and to check for port dependencies.

I've never used it to update the system, so this is something I'll need to figure out. I stayed away from it because it required messing with the config file (and point to a repository + other things I didn't understand).


----------



## marino (Jul 16, 2017)

I'm certain you've used something like `pkg install emacs` before.
With commands like that, pkg will connect to the remote repository, ingest the catalog thus figuring out which installed packages have available updates, then it will download the requested package and install it, along with any dependencies that have been updated.

So you've definitely used pkg to update your system before.  You just didn't realize what was going on underneath.
In conjunction with synth, the repository pkg(8) connects to is local rather than one of the remote official freebsd ones.  But the first time you run synth, the entire repo has to be built up.  After that it's usually incremental builds.


----------



## Mayhem30 (Jul 17, 2017)

marino said:


> I'm certain you've used something like  pkg install emacs before.



I'm sure I have never used it. I was taught from day one just to use `make config`, `make install clean` and `portupgrade` (later transitioned to `portmaster`). Also, when I first started using FreeBSD, `PKG` wasn't an option, it was only `PKG_*` - I never used those directly either.


----------



## marino (Jul 17, 2017)

hmm?  taught by whom?
FreeBSD has been a binary package first model for several years now.  
Way back when, FreeBSD was known for building everything by scratch, but that's basically discouraged now.  The philosophy now is only build if you need non-standard (and thus poorly tested) options, or if you are one of the tin-hat people that think all freebsd packages are compromised by the NSA (or a small 3rd group that think they tailor CPU settings and end up with "better" executables).

the point is that nobody is saying that building from source is the primary way to acquire 3rd party software.  The official line is "use packages".

pre-pkg(8) (as you indicate is your generation), it was common to build everything because binary packages had lots of issues.  Most of those issues have been eliminated and official packages are trustworthy IMO.


----------



## Mayhem30 (Jul 17, 2017)

A good friend of mine showed me the ropes a long time ago. It just worked for me, so I never looked at anything other method - until now.

I still want to be able to control what gets installed (php extensions, apache modules, etc) plus control the port options myself. I believe Synth is the answer for me, rather then using the official binary packages.

I've always had great dedicated servers, so I never cared about tailoring anything for more speed or better executables.


----------



## Mayhem30 (Jul 17, 2017)

This is really impressive! It only took 21 mins to do all 152 packages. If I remember correctly, it took well over an hour using `portmaster`.

Also, PKG found a bunch of issues. How it spells everything out is really impressive.

I noticed that is was going to remove MySQL 5.5 client & server, but only install mysql5.6 client - so I need to still figure that out.


```
$ sudo pkg upgrade -n
Updating custom repository catalogue...
custom repository is up to date.
All repositories are up to date.
Checking for upgrades (142 candidates): 100%
Processing candidates (142 candidates): 100%
Checking integrity... done (4 conflicting)
  - mysql56-client-5.6.36 conflicts with mysql55-client-5.5.56 on /usr/local/bin/msql2mysql
  - mysql56-client-5.6.36 conflicts with mysql55-client-5.5.56 on /usr/local/bin/msql2mysql
  - jpeg-turbo-1.5.1 conflicts with jpeg-8_7 on /usr/local/bin/cjpeg
  - jpeg-turbo-1.5.1 conflicts with jpeg-8_7 on /usr/local/bin/cjpeg
Cannot solve problem using SAT solver, trying another plan
Checking integrity... done (0 conflicting)
The following 13 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        mysql55-server-5.5.56
        mysql55-client-5.5.56
        jpeg-8_7

New packages to be INSTALLED:
        mysql56-client: 5.6.36
        liblz4: 1.7.5,1
        jpeg-turbo: 1.5.1

Installed packages to be REINSTALLED:
        sphinxsearch-2.2.11_1 (direct dependency changed: mysql56-client)
        postfix-3.2.2,1 (direct dependency changed: mysql56-client)
        phpmailer-5.2.23 (direct dependency changed: php71)
        phpMyAdmin-4.7.2 (direct dependency changed: php71-ctype)
        php71-gd-7.1.7 (direct dependency changed: jpeg-turbo)
        gdbm-1.13_1 (needed shared library changed)
        dovecot2-2.2.31_1 (direct dependency changed: mysql56-client)

Number of packages to be removed: 3
Number of packages to be installed: 3
Number of packages to be reinstalled: 7

The operation will free 66 MiB.
```


----------



## marino (Jul 17, 2017)

you have to define the mysql version in the <profile>-make.conf, otherwise it will build the default version of mysql.
There should be lots of posts about defining default versions, and check the handbook (and maybe synth man page)


----------



## Mayhem30 (Jul 17, 2017)

Anything else need to be added for it to work? I just copy & pasted my /etc/make.conf into /usr/local/etc/synth/LiveSystem-make.conf.

I also ran synth rebuild-repository and it still wants to get rid of 5.5 and use 5.6


```
CPUTYPE?=corei7-avx

# /usr/ports/Mk/bsd.options.desc.mk
OPTIONS_UNSET = X11 CUPS NLS

MAKE_JOBS_NUMBER = 8
WITH_POSTFIX = yes
DEFAULT_VERSIONS += ssl=openssl php=7.1 mysql=5.5

WRKDIRPREFIX = /ram
```


----------



## marino (Jul 17, 2017)

big sin, just copying and pasting that.
HUGE ERROR 1: defining MAKE_JOBS_NUMBER.  get rid of that line immediately
HUGE ERROR 2: defining WRKDIRPREFIX.  get rid of that line immediately
ASKING FOR TROUBLE: defining CPUTYPE.   I would remove it

You've just illustrated one good reason why /etc/make.conf is not used and why each profile has a make.conf.


----------



## marino (Jul 17, 2017)

Mayhem30 said:


> Anything else need to be added for it to work? I just copy & pasted my /etc/make.conf into /usr/local/etc/synth/LiveSystem-make.conf.
> 
> I also ran synth rebuild-repository and it still wants to get rid of 5.5 and use 5.6



Did you actually call `synth prepare-system` first? Anything that is affected by your make.conf changes has to be rebuilt.


----------



## Mayhem30 (Jul 17, 2017)

Ok, I ran `synth prepare-system` after correcting the make.conf changes and it shows :


```
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
dovecot2-2.2.31_1.txz failed dependency check.
postfix-3.2.2,1.txz failed dependency check.
sphinxsearch-2.2.11_1.txz failed dependency check.

The task is complete.  Final tally:

Initial queue size: 3
    packages built: 3
           ignored: 0
           skipped: 0
            failed: 0

Duration: 00:01:24
```

Those are the 3 files that depend on MySQL 5.6. It still wants to use 5.6 though.


```
$ pkg upgrade -n     
Checking for upgrades (142 candidates): 100%
Processing candidates (142 candidates): 100%
Checking integrity... done (4 conflicting)
  - mysql56-client-5.6.36 conflicts with mysql55-client-5.5.56 on /usr/local/bin/msql2mysql
  - mysql56-client-5.6.36 conflicts with mysql55-client-5.5.56 on /usr/local/bin/msql2mysql
  - jpeg-turbo-1.5.1 conflicts with jpeg-8_7 on /usr/local/bin/cjpeg
  - jpeg-turbo-1.5.1 conflicts with jpeg-8_7 on /usr/local/bin/cjpeg
Cannot solve problem using SAT solver, trying another plan
Checking integrity... done (0 conflicting)
The following 13 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        mysql55-server-5.5.56
        mysql55-client-5.5.56
        jpeg-8_7

New packages to be INSTALLED:
        mysql56-client: 5.6.36
        liblz4: 1.7.5,1
        jpeg-turbo: 1.5.1

Installed packages to be REINSTALLED:
        sphinxsearch-2.2.11_1 (direct dependency changed: mysql56-client)
        postfix-3.2.2,1 (direct dependency changed: mysql56-client)
        phpmailer-5.2.23 (direct dependency changed: php71)
        phpMyAdmin-4.7.2 (direct dependency changed: php71-ctype)
        php71-gd-7.1.7 (direct dependency changed: jpeg-turbo)
        gdbm-1.13_1 (needed shared library changed)
        dovecot2-2.2.31_1 (direct dependency changed: mysql56-client)

Number of packages to be removed: 3
Number of packages to be installed: 3
Number of packages to be reinstalled: 7
```


----------



## marino (Jul 17, 2017)

You could be connecting with the freebsd repository, not the one you built.
try `pkg upgrade -r Synth -n`
to explicitly use the local repository


----------



## Mayhem30 (Jul 17, 2017)

Weird .. I thought I had FreeBSD repository disabled. This is what I have done :

/usr/local/etc/pkg/repos/FreeBSD.conf :

```
FreeBSD: { enabled: no }
```
Also, I had to create my own custom.conf in the repos directory (was Synth supposed to do that for me?)

/usr/local/etc/pkg/repos/Synth.conf :

```
custom: {
  url: "file:///var/synth/live_packages/",
  enabled: yes
}
```
Using `pkg upgrade -r custom -n` fixes the issue, but I would like to disable the FreeBSD repo and default it to my custom one.


----------



## marino (Jul 17, 2017)

synth creates one, but I think you have to try to install something through synth, e.g. "synth install dovecot" or "synth upgrade-system" .  After the first installation, you can use "-r Synth".

It will probably overwrite what you did manually.


----------



## Mayhem30 (Jul 17, 2017)

No way to disable the FreeBSD repo and default it to the Synth repo?

That way I don't have to point to the correct repo every time.


----------



## marino (Jul 17, 2017)

You don't need to disable it.  Synth version takes precedence.  It starts with "00_" and the repo config files are processed in alphabetical order. (so your version WON'T be overwritten).
I don't know if you can actually disable it.  I think if nothing is configured it always goes back to the official repo.


added: But I don't trust the precedence so I always specify "-r Synth" to be sure.


----------



## Mayhem30 (Jul 17, 2017)

Ok, `synth install databases/phpmyadmin` created the missing 00_synth.conf file and everything is working great now.

I checked using `pkg -vvv` and only the Synth repo is listed at the bottom - which appears the FreeBSD repo was successfully disabled.

Thanks again for all your help (and patience).


----------



## Mayhem30 (Jul 18, 2017)

Does Synth (or PKG) have any commands that will compare local and remote database - and remove any packages from the repository that is not currently installed?


```
$ pkg stats
Local package database:
        Installed packages: 149
        Disk space occupied: 1 GiB

Remote package database(s):
        Number of repositories: 1
        Packages available: 152
        Unique packages: 152
        Total size of packages: 199 MiB
```


----------



## marino (Jul 19, 2017)

I don't think I understand the question.   Or the ultimate purpose.  
"remove any packages from the repository".
For most people, there's only one repository, the remote one.  Nobody can "remove" packages from the remote repository.
For synth and poudriere users, there is a local repository.  Obsolete packages are removed before the repository is built, so everything is current.
Moreover, what's the purpose of "removing packages from a repository"?  to save disk space?

The answer is "I don't know of any commands" but as I said, I think I don't even understand what you're getting at.


----------



## Mayhem30 (Jul 19, 2017)

I'm sorry, I should have been more clear in my explanation. I noticed the Synth repository contains 152 items. However, I only have 149 packages installed.

Is there any way to compare the 152 items in the repository with the 149 items I have installed - to let me know which (3 files) can be deleted in repository?

I'm thinking over time that number will grow and would be nice to not have any obsolete items sitting in the repository when they are not needed or being used.


----------



## marino (Jul 19, 2017)

the items are not "obsolete".
They are likely build dependencies.
For example, "gmake".  You need "gmake" to build a package, but you don't need it after that.
If you deleted "gmake" package, synth would have to build it again.
Synth deletes truly obsolete packages (older versions) when the repository is generated, so the actual repository is 100% current.

Bottom line: the packages you don't use are not worth worrying about.  I'm surprised the number is so low.  I would expect a lot more than 3 packages out of 152 to be "build-only".


----------



## Mayhem30 (Jul 19, 2017)

Ok, I just did a compare and these are the files left over :

```
jpeg-8_7.txz
nasm-2.11.08_1,1.txz
liblz4-1.7.5,1.txz
```
jpeg is not longer used and was replaced with jpeg-turbo (so that's not a build dependency)

I'm not sure what the other files are.  I won't worry about though.


----------



## marino (Jul 20, 2017)

i didn't understand you were talking about what was installed on the system vs. what got built.


----------



## w1k1n9cc (Jul 20, 2017)

Hy, I'm not at my FreeBSD computer but maybe you have a good suggestion. I try to install py-pytoport. `synth status ports-mgmt/py-pytoport` show me that it is marked with N, so it should be installed. But if I run `synth install ports-mgmt/py-pytoport` , it didn't compiled it and so pkg can't find a packages. Do you need some further informations?
I haven't a custom kernel or anything else only a few ports installed through synth.


----------



## marino (Jul 20, 2017)

There are several logs in the log directory that start with the number zero.  Reviewing these overall build logs usually answers questions like this.  With no real details, I'd say py-pytoport was either skipped or ignored.  The reason why would be available in the logs.


----------



## marino (Jul 20, 2017)

Oh, by the way, there is a detailed HTML report with the complete build history in the "Report" subdirectory of the log directory.  It will also clearly show skips and ignores and is searchable/filterable.  It is always reflecting the ongoing or last build, then overwritten.  If you aren't aware of this web report, you should check it out.


----------



## w1k1n9cc (Jul 21, 2017)

Thanks, with your advice I found the problem really fast. Maybe a verbose output would be great, unfortunately I found no hints in the man page.
But the problem comes from the Makefile itself and from the fact that a package should be built.

```
.if defined(PACKAGE_BUILDING) && !defined(PACKAGE_BUILDING_FLAVORS) && \
    ${PYTHON_VER} != ${PYTHON_DEFAULT}
IGNORE= you have python ${PYTHON_DEFAULT} set as the default, and this needs ${PYTHON_VER}
.endif
```
Would it be useful to set PACKAGE_BUILDING_FLAVORS automatically from synth? According to this https://svnweb.freebsd.org/ports/head/ports-mgmt/py-pytoport/Makefile?view=log the package build is disabled because of poudriere.


----------



## marino (Jul 22, 2017)

it's a known flaw with ports that won't be properly corrected until flavors/subpackages is implemented.  poudriere by now probably implements the rather nasty framework workaround, but I'm not going to.  Without looking, I'm guessing other ports get around this using slave ports.  I don't really have a suggestion for you regarding this known issue.


----------



## Wapcaplet (Jul 27, 2017)

FYI, there's a problem with building Synth after upgrading to 11.1, as it appears lang/gcc6-aux must be rebuilt after the upgrade to 11.1 before Synth will build.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220458


----------



## marino (Jul 28, 2017)

I assume that the modifications to the other gcc ports to block the installation of fixed-headers also needs to be applied to gcc-aux ports.  Anybody can request this; they are unmaintained ports.


----------



## Wapcaplet (Jul 29, 2017)

I took a look at the patches for the other gcc ports, and it looks like the patches just delete the /include-fixed directory during post-stage.  lang/gcc6-aux already does this if the BOOTSTRAP option is turned on, so should that line be taken out of the BOOTSTRAP option requirement and simply made mandatory during post-stage?

This is a bit above my knowledge level, unfortunately.


----------



## marino (Jul 29, 2017)

Wapcaplet said:


> so should that line be taken out of the BOOTSTRAP option requirement and simply made mandatory during post-stage?



That sounds about right.  I haven't looked at it, but yeah, the change would be to remove fixed headers unconditionally.


----------



## Mayhem30 (Jul 31, 2017)

Is this a bug or something else?

I noticed when I updated my last 3 ports (each on separate days), the log file shows that PKG is not found and installs it. PKG has always existed on the system though.

```
--------------------------------------------------------------------------------
--  Phase: pkg-depends
--------------------------------------------------------------------------------
===>   py27-pyinotify-0.9.6 depends on file: /usr/local/sbin/pkg - not found
===>   Installing existing package /packages/All/pkg-1.10.1.txz
Installing pkg-1.10.1...
Extracting pkg-1.10.1: .......... done
===>   py27-pyinotify-0.9.6 depends on file: /usr/local/sbin/pkg - found
===>   Returning to build of py27-pyinotify-0.9.6



--------------------------------------------------------------------------------
--  Phase: pkg-depends
--------------------------------------------------------------------------------
===>   unbound-1.6.4_1 depends on file: /usr/local/sbin/pkg - not found
===>   Installing existing package /packages/All/pkg-1.10.1.txz
Installing pkg-1.10.1...
Extracting pkg-1.10.1: .......... done
===>   unbound-1.6.4_1 depends on file: /usr/local/sbin/pkg - found
===>   Returning to build of unbound-1.6.4_1



--------------------------------------------------------------------------------
--  Phase: pkg-depends
--------------------------------------------------------------------------------
===>   nano-2.8.6 depends on file: /usr/local/sbin/pkg - not found
===>   Installing existing package /packages/All/pkg-1.10.1.txz
Installing pkg-1.10.1...
Extracting pkg-1.10.1: .......... done
===>   nano-2.8.6 depends on file: /usr/local/sbin/pkg - found
===>   Returning to build of nano-2.8.6
```

The file is where it should be :

```
$ ls /usr/local/sbin/pkg
/usr/local/sbin/pkg
```


----------



## marino (Jul 31, 2017)

it's installing pkg(8) in the builder jail (internal build environment).
synth doesn't modify the localhost environment which is where the pkg you're referring to is located.


----------



## Mayhem30 (Jul 31, 2017)

Ah ok, I just wanted to make sure everything was fine. Thanks.


----------



## Wapcaplet (Jul 31, 2017)

marino said:


> That sounds about right.  I haven't looked at it, but yeah, the change would be to remove fixed headers unconditionally.


I just opened PR 221111 to patch lang/gcc6-aux.  Hopefully everything looks OK.  Thanks again for your help!


----------



## Mayhem30 (Aug 3, 2017)

I'm curious if this is working as intended? Maybe I'm just misunderstanding how this works.

There was an update for lang/python27 today.

After running `portsnap fetch update`, I did a `synth upgrade-system` and it rebuilt the following 24 packages.

```
python27-2.7.13_7
python2-2_3
py27-Babel-2.3.4
py27-Jinja2-2.9.5
py27-MarkupSafe-1.0
py27-alabaster-0.7.6
py27-authres-1.0.0
py27-dns-2.3.6_1
py27-docutils-0.13.1
py27-fail2ban-0.9.7
py27-imagesize-0.7.1
py27-ipaddr-2.1.11
py27-postfix-policyd-spf-python-1.3.2_1
py27-pygments-2.2.0
py27-pyinotify-0.9.6
py27-pyspf-2.0.12_4
py27-pytz-2017.2,1
py27-setuptools-36.0.1
py27-six-1.10.0
py27-snowballstemmer-1.2.0_1
py27-sphinx-1.4.8_1,1
py27-sphinx_rtd_theme-0.2.4
py27-sqlite3-2.7.13_7
scons-2.5.1_1
```
After doing so, it only installed "python27-2.7.13_7" and did nothing with the 23 other packages.

```
Number of packages to be upgraded: 1
```
Shouldn't the other 23 packages been forced upgraded - since they depend on the new updated python version that was just installed?


----------



## rigoletto@ (Aug 9, 2017)

Hi, 

1. I have 16GB RAM + 32GB Swap (SSD). I am using 4 builders and 8 jobs and it is working fine, but the maximum swap usage I got were about 14%. Usually it does not goes more than 4% usage. However if I increase the build or jobs numbers I begin to have some ports failing to build.

Is it supposedly to use more swap, allowing me to increase these numbers? Or should I have something mis-configured, possible what?

2. I was thinking if it would be feasible to implement the ability of setting the total jobs number instead of per builder. I mean, (e.g.) I currently have 32 possible jobs in total but if I have just two ports building, with two slots in idle/shutdown state (I understand) ports-mgmt/synth will be using just half of the total possible jobs, 16.

Or, in a event of several ports to be built, but with two idle slots and two working, each working slot could receive 16 jobs. When one of these be built, eventually allowing more three new jobs, the total jobs could be re-distributed to the new ports, then 4 for each one.

Thank you.


----------



## Mayhem30 (Aug 9, 2017)

I also have 16GB of ram, and I use 6 builders and 4 jobs and it's never failed to build a port or touched any swap. It builds all my 150 ports in less than 20 minutes.

Other than that, I'm the last guy you want giving any advice on this


----------



## Eric A. Borisch (Aug 11, 2017)

lebarondemerde said:


> 2. I was thinking if it would be feasible to implement the ability of setting the total jobs number instead of per builder. I mean, (e.g.) I currently have 32 possible jobs in total but if I have just two ports building, with two slots in idle/shutdown state (I understand) ports-mgmt/synth will be using just half of the total possible jobs, 16.



I wholeheartedly second this. At the very least once a builder is shut down, reallocate jobs to the next task that kicks off in a still-running builder. Or when there is a choke-point of dependencies that bring it down to one builder...

Otherwise no complaints; a very happy "customer!"


----------



## rjohn (Aug 11, 2017)

i have installed synth via pkg install not from ports.

when i run `synth status` or `synth system-upgrade` i get


```
$ sudo env http_proxy=http://10.1.1.1:8080/ synth upgrade-system
Password:
Builder mounts detected; attempting to remove them automatically ...
Dismounting successful!
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Stand by, building pkg(8) first ... Failed!!  (Synth must exit)
Unfortunately, the system upgrade failed.
```


----------



## rigoletto@ (Aug 12, 2017)

Mayhem30

With 150 ports I can assume you are talking about a server, what usually does not include the big ones, like www/firefox, www/qt5-webkit, editors/libreoffice etc. And they also tends to be built at the same time...

The big ones are the real bottleneck, otherwise I could increase the numbers considerably.


----------



## Mayhem30 (Aug 12, 2017)

Yes, mine is a server (not desktop).

Those ports you listed above have some rather large dependencies ... now I see why you're trying to tweak things.


----------



## rjohn (Aug 17, 2017)

hi,as mentioned above i did not manage to make synth work ,in `pkg update`

was getting : 
	
	



```
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating Synth repository catalogue...
pkg: Repository Synth load error: access repo file(/var/db/pkg/repo-Synth.sqlite) failed: No such file or directory
pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
repository Synth has no meta file, using default settings
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Unable to update repository Synth
Error updating repositories!
```
i have removed the packages required for synth and synth via : `pkg remove synth` and `pkg remove gcc6-aux ncurses`

but seems its still there ,even in search i get : 
	
	



```
pkg: Repository Synth missing. 'pkg update' required
pkg: Repository Synth load error: access repo file(/var/db/pkg/repo-Synth.sqlite) failed: No such file or directory
pkg: Repository Synth cannot be opened. 'pkg update' required
```

any ideas to complete remove it ?


```
freebsd# sudo pkg install desktop-installer
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating Synth repository catalogue...
pkg: Repository Synth load error: access repo file(/var/db/pkg/repo-Synth.sqlite) failed: No such file or directory
pkg: file:///var/synth/live_packages/meta.txz: No such file or directory
repository Synth has no meta file, using default settings
pkg: file:///var/synth/live_packages/packagesite.txz: No such file or directory
Unable to update repository Synth
Error updating repositories!
```
seem i really messed it ,now cant install pkgs at all .tryed to remove and install pkg from ports with no luck same error .


----------



## tobik@ (Aug 17, 2017)

rjohn Delete the repository configuration Synth creates: `rm /usr/local/etc/pkg/repos/00_synth.conf`


----------



## rjohn (Aug 17, 2017)

thanks!!!!


----------



## topcat (Aug 17, 2017)

I have been reading this thread for a while, and want to take this chance to thank marino for writing ports-mgmt/synth. I like to tweak options and build my ports, and ports-mgmt/synth makes setting up the isolated builds super easy, plus I like the curses interface.


----------



## Mayhem30 (Sep 6, 2017)

What is the best method to reinstall all ports after doing an OS upgrade?

Should I be doing it differently? or am I missing any steps?

```
sudo pkg delete -a
delete all files in: /var/synth/live_packages/All
cd /usr/ports/ports-mgmt/synth
sudo make install clean
sudo synth build /home/me/build.list
```


----------



## mefizto (Sep 6, 2017)

Greetings all,

I had been using mgmt-ports/synth to compile ports, instead of the `make, install, clean`. 

Recently, I decided to install GUI using the guide at https://forums.freebsd.org/threads/35308/.  The instructions recommend to disable sysutils/hal in the x11-servers/xorg-server.  O.K., no problem, I used `make -C /usr/ports/x11-servers/xorg-server config` to set the desirable options before starting the compilation.  However, the instructions recommend to further disable sysutils/hal in _all other ports that might have HAL option._  This gave me a pause because I do not know how to ascertain that one of the dependencies does not have sysutils/hal option.  The `make, install, clean` interrupts the compilation at every dependency with options, so one could check.  However, with mgmt-ports/synth it seems practically impossible to set the configuration for all the dependencies using the `make -C /usr/ports/[category]/[portname] config`.

Thinking about it, the issue is even more complex.  How does one ensure that the option settings is consistent not only for the particular port dependencies, but among all the ports?  Any advice is welcomed. 

Kindest regards,

M


----------



## Mayhem30 (Sep 6, 2017)

In order to disable an option for all ports, you need to edit your /usr/local/etc/synth/LiveSystem-make.conf file and add :

```
OPTIONS_UNSET = HAL
```
If you want to disable multiple options, it needs to be space-delimited.

```
OPTIONS_UNSET = CUPS HAL
```
Also, I usually run `synth prepare-system` afterwards to let Synth rebuild any changes.


----------



## mefizto (Sep 6, 2017)

Hi Mayhem30,

awesome.  Thank you very much.

Kindest regards,

M


----------



## Mayhem30 (Sep 8, 2017)

marino said:


> the items are not "obsolete".
> They are likely build dependencies.
> For example, "gmake".  You need "gmake" to build a package, but you don't need it after that.
> If you deleted "gmake" package, synth would have to build it again.
> ...



I figured out what the issue was. I had to `pkg delete` all the packages off the server and reinstall them. That is an important step if your migrating from portupgrade/portmaster, otherwise you have all sorts of build dependencies installed on the system for no reason.

What I did was :

```
cd /var/synth/live_packages/All
rm -rf *.txz
synth just-build `cat /home/build.list`
synth rebuild-repository
pkg delete -af
/usr/sbin/pkg bootstrap
pkg install `cat /home/build.list`
```
Now it shows :

```
$ pkg stats          
Local package database:
        Installed packages: 103
        Disk space occupied: 567 MiB

Remote package database(s):
        Number of repositories: 1
        Packages available: 142
        Unique packages: 142
        Total size of packages: 193 MiB
```
That's a big difference. Almost 40 packages that were not needed to be installed on the system.


----------



## Matt Joras (Sep 18, 2017)

So I randomly seem to get this error when building ~500 ports (just started using synth):


```
thread <something> exits with resources held!' at line 316 in file /usr/src/lib/libthr/thread/thr_exit.c (errno = 2)
```

Any ideas what might be causing that? It doesn't seem to be related to any particular port. Anything I can do to help debug it? This is on a FreeBSD-CURRENT machine, fwiw.


----------



## Mayhem30 (Sep 18, 2017)

If you don't receive any assistance, you might want to post this bug on GitHub : https://github.com/jrmarino/synth/issues


----------



## rigoletto@ (Sep 18, 2017)

Matt Joras

Have you watched the memory/swap usage?


----------



## Matt Joras (Sep 18, 2017)

lebarondemerde said:


> Matt Joras
> 
> Have you watched the memory/swap usage?



Admittedly not very closely prior to the crash. Watching it now it doesn't look like it gets anywhere close to the 32GB mem + 4GB swap.

Anyway, the message doesn't seem memory-related. Per my cursory reading of libthr it looks like a thread exiting while holding a lock:

```
#if defined(_PTHREADS_INVARIANTS)                                                                                                                                                             
        if (THR_IN_CRITICAL(curthread))                                                                                                                                                       
                PANIC("thread %p exits with resources held!", curthread);                                                                                                                     
#endif
...

#define THR_IN_CRITICAL(thrd)                           \                                                                                                                                     
        (((thrd)->locklevel > 0) ||                     \                                                                                                                                     
        ((thrd)->critical_count > 0))
```


----------



## Eric A. Borisch (Nov 30, 2017)

Sooo.. any hints on what to do post python-flavors change 11/30/2017? Everything python related (and above) fails due to requiring "py27-setuptools>0:devel/py-setyptools@py27" which "does not exist".


----------



## rigoletto@ (Nov 30, 2017)

Eric A. Borisch

For now I think you need ports-mgmt/poudriere to have FLAVOURS support.


----------



## Mayhem30 (Nov 30, 2017)

Why are you recommending Poudriere in a Synth thread as a means of a fix?


----------



## Mayhem30 (Nov 30, 2017)

Eric A. Borisch said:


> Sooo.. any hints on what to do post python-flavors change 11/30/2017? Everything python related (and above) fails due to requiring "py27-setuptools>0:devel/py-setyptools@py27" which "does not exist".



I'm having the same issue and started this thread : https://forums.freebsd.org/threads/63494/


----------



## rigoletto@ (Nov 30, 2017)

Mayhem30 said:


> Why are you recommending Poudriere in a Synth thread as a means of a fix?



It seems quite obvious ports-mgmt/synth still does not support the new port FLAVOURS feature.


----------



## marino (Nov 30, 2017)

In my defense, we were told we would have 6 months to work on the support.
This introduction of flavors predates that by at least 3 months.
Secondly, the flavors developers did not provide a separate flavor branch in order to test new tools.
AFAIK, portmaster would also be caught by this.

support was planned, but introducing it 3 months early catches me by surprise.
On the plus side, at least I now how real ports to test it against.


----------



## rigoletto@ (Nov 30, 2017)

Upstream should be probably very keen of getting rid of those many, many @python slave ports. 

EDIT: I am also surprised BTW. When Poudriere 3.2 come out (the version with FLAVOUR support) I asked around when upstream would start accepting FLAVOURED ports and I got no reply.

So, I assumed it would still take a while.


----------



## Mayhem30 (Nov 30, 2017)

marino said:


> support was planned, but introducing it 3 months early catches me by surprise.
> On the plus side, at least I now how real ports to test it against.



Do you think it'll take a long time add support for FLAVOURED ports?


----------



## marino (Nov 30, 2017)

dunno, probably a few days.  I recommend synth users lock in tree at specific revision, no more than rev 455209.  That's the one before ports are converted to flavors.  I'm assuming the previous flavors commits don't break the tree yet.  In any case, the last "good" revision is sometime earlier today.


----------



## marino (Dec 3, 2017)

new version released for flavors support, see Thread 63525


----------



## Mayhem30 (Dec 3, 2017)

You've been busy! I noticed on GitHub that you were constantly working on this day and night to get it done.

Thank you for all your hard work on this.


----------



## priyadarshan (Dec 3, 2017)

marino said:


> new version released for flavors support, see Thread 63525


Thank you for this and all the work you have been offering to BSD community. It is quite inspiring.


----------



## jdp_03 (Jan 12, 2018)

I am trying to bring my system up to date after a while away due to illness.  After running `synth prepare-system` request, I get:


```
Stand by, updating external repository catalogs ... done.
Scanning existing packages.
Scanning existing packages.  
Queue integrity lost!   (Synth must exit)
Unfortunately, the system upgrade failed.
```

Please could you advise on what I need to do next?


----------



## marino (Jan 12, 2018)

You didn't specify what version of synth you're using.  If it's not 2.02, then upgrade that first.


----------



## jdp_03 (Jan 12, 2018)

marino said:


> You didn't specify what version of synth you're using.  If it's not 2.02, then upgrade that first.


 
Thanks - that was my problem.


----------



## Snurg (Jan 13, 2018)

Is it possible to use synth for speedier buildkernel/buildworld, too?

(Sorry if this has been asked before... this thread is just too long to read all)


----------



## Eric A. Borisch (Jan 13, 2018)

Snurg said:


> Is it possible to use synth for speedier buildkernel/buildworld, too?
> 
> (Sorry if this has been asked before... this thread is just too long to read all)



No, but put `WITH_META_MODE=1` in src-env.conf(5). (The value doesn't matter; just needs to be set.)


----------



## Mayhem30 (Jan 28, 2018)

Is there any way to tell what is causing synth to build lang/python27 (+ 17 other py27- packages) as a build dependency?

They are never installed, so I can't query the package database to find out.

I've recently moved from lang/python27 to lang/python36 and the ports that depended on it (security/py-fail2ban, devel/py-pyinotify and mail/postfix-policyd-spf-python) all have been rebuilt to the py36 flavors.


----------



## marino (Jan 28, 2018)

are you using `synth prepare-system` or something like that?  (as opposed to a static build list?)
If so, it could be that an old package relies on python27, even if the newer versions don't.

Also, you state they are being built, but not if they are being installed.  Python27 can easily be a build dependency for any number of ports.


----------



## Mayhem30 (Jan 28, 2018)

Yes, I am using `synth prepare-system` and that is what triggered it.

I was just curious on what package depended on it, because the only reason I installed lang/python27 to begin with is in order to use those 3 ports mentioned above.

It was never a build dependency or installed until I installed those 3 ports.


----------



## marino (Jan 28, 2018)

when in doubt, use the procedure to remove all packages from the system and reinstall them from a local repository.  That gets rid of all the cruft that builds up.  You may have manually installed python27 in which case pkg(8) is telling synth to rebuild it because it's a primary package.


----------



## Mayhem30 (Jan 28, 2018)

marino said:


> when in doubt, use the procedure to remove all packages from the system and reinstall them from a local repository. That gets rid of all the cruft that builds up



Will do, thanks!


----------



## Mayhem30 (Mar 2, 2018)

When synth is regenerating the flavor index, does it use tmpfs or does it write directly to disk?

I have tmpfs enabled in the synth config, but my HD shoots up to 40%-50% load when the index is regenerating. I have plenty of free ram and swap is not being used.


----------



## marino (Mar 3, 2018)

it reads the mounted ports tree (every port) with all available CPUs and writes directly to disk.  The writing is not an issue.  If it takes 10 minutes to scan the ports directory with 4 CPUs, it would probably take an hour to do it with 1 CPU.   With recent (long overdue) improves in compiler features testing, the scan time might reduce considerably though.

In any case, the server is very busy during portstree scans and a huge load jump is normal.


----------



## Mayhem30 (Mar 3, 2018)

To reduce the disk io, would it not be possible to build/store the flavor index in tmpfs until it's complete and then dump it to disk?

I don't notice any load on my SSD's on my production server, but my home machine (which has a standard HDD) is hovering right around 50% for that 15 minutes or so.


----------



## marino (Mar 3, 2018)

as I said, *building* is not the issue.  The actual creation of the index takes less than 1 second.  Dozens of instances of make are trying to read various directories, each one spawning hundreds of forks.  You can thank the ports collection makefile design for that.  tmpfs isn't going to help any of that.


----------



## Mayhem30 (Mar 4, 2018)

marino said:


> With recent (long overdue) improves in compiler features testing, the scan time might reduce considerably though.



Do you know when these improvements will be released? Do you need to change anything in Synth to take advantage of them?


----------



## marino (Mar 4, 2018)

They are already in the tree I think.  You should already be seeing improvements in the scan time (I'm guessing; all I saw was the code, I don't know for a fact that it significantly helps FreeBSD, but I hope so)


----------



## Mayhem30 (Mar 4, 2018)

I didn't notice any improvement on my home machine. Maybe it hasn't been added yet.


----------



## marino (Mar 4, 2018)

then i would guess that it's code that makes poudriere faster that isn't active for any other system.  If poudriere is using tricks to store the compiler features, it would be nice if that trick was exposed for all systems.  The change has been in place for about a week now.


----------



## Mayhem30 (Mar 5, 2018)

marino said:


> then i would guess that it's code that makes poudriere faster that isn't active for any other system



I really hope that isn't the case!

I'm definitely not seeing any improvement over here. I just timed it and it took 21 minutes to regenerate the flavor index on my home machine. It's only used to receive backups from my dedicated server.

It's an Intel i5 CPU with 8 GB ram. Here is the "Top" stats when synth is regenerating the index.

```
last pid: 14470;  load averages:  1.16,  0.93,  0.54
31 processes:  2 running, 25 sleeping, 4 zombie
CPU 0: 28.0% user,  0.0% nice,  6.5% system,  0.5% interrupt, 65.0% idle
CPU 1: 12.6% user,  0.0% nice,  4.7% system,  0.0% interrupt, 82.7% idle
CPU 2: 15.0% user,  0.0% nice,  6.1% system,  0.0% interrupt, 79.0% idle
CPU 3: 22.4% user,  0.0% nice,  6.5% system,  0.0% interrupt, 71.0% idle
Mem: 32M Active, 368M Inact, 945M Wired, 536M Buf, 6471M Free
Swap: 3852M Total, 3852M Free
```

And my config :

```
[I] Num. concurrent builders   4
   [J] Max. jobs per builder      4
   [K] Use tmpfs for work area    true
   [L] Use tmpfs for localbase    true
```


----------



## marino (Mar 5, 2018)

here's the commit:
https://svnweb.freebsd.org/ports?view=revision&revision=463259

maybe you can find a reference to where _CCVERSION_hash and _CXXINTERNAL_hash are getting defined.  I was guessing that poudriere outputs these.

However, it means it's possible to add these caches to synth's [profile]-make.conf file to leverage this new code.  I just don't know what the values are for FreeBSD release 11, 12, etc.  Somebody could figure it out and post it.

The idea is that the values are determined ONCE and written down in [profile]-make.conf, rather than be determined hundreds of times a second, all with the same identical result.


----------



## Mayhem30 (Mar 5, 2018)

This is beyond my expertise. Someone needs to ask about this on the FreeBSD mailing list.


----------



## xtaz (Mar 6, 2018)

Out of curiosity, when you say the flavor index takes 21 minutes to generate, how many ports do you have installed? Or is this literally recursing through every single port in the port tree? I'm building around 200 ports.

The reason I'm curious is because I switched to Poudriere when Synth temporarily broke because of gcc-aux not building and I've not tried Synth again since. But Poudriere doesn't spend any time generating long indexes so I'm wondering why Synth needs to take that long. I don't even use ZFS so it spends time copying files around on UFS but a bulk run with four jails still only takes a single minute. If Synth takes 20 times longer than that on every run then that's not a good improvement on when I was using it before.


----------



## Mayhem30 (Mar 6, 2018)

I don't have many ports installed at all. This machine is only used to receive backups every night from my dedicated server.

```
[backup@localhost ~]$ pkg stats
Local package database:
        Installed packages: 15
        Disk space occupied: 37 MiB

Remote package database(s):
        Number of repositories: 1
        Packages available: 32
        Unique packages: 32
        Total size of packages: 101 MiB
```


This is the log I saved from last night so I have something to compare to if / when things get sorted out with Synth. As you can see, it took 21 minutes to rebuild the flavor index.

```
[backup@localhost ~]$ date && sudo synth status && date                     
Mon Mar  5 17:52:58 PST 2018
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt
Mon Mar  5 18:14:00 PST 2018
[backup@localhost ~]$
```


----------



## marino (Mar 6, 2018)

xtaz said:


> But Poudriere doesn't spend any time generating long indexes so I'm wondering why Synth needs to take that long.



Because the ports tree design sucks.  And this is a misleading question.  The same exact operation on DragonFly takes about 2-3 minutes.  Yes, DragonFly Dports is 10x faster than FreeBSD ports because most of the inefficient crap that is in the ports tree was removed/fixed on dports.

poudriere doesn't use indices, but it's not like it's not doing the same calculations.  Synth does it ONCE per tree update, poudriere does it hundreds or thousands of times repetitively.  It determines via a make query each and every time.  The main difference seen by a local user is that even thought it's inefficient, they might scan only a fraction of the tree for any given run, but synth does the entire tree because it doesn't know what will be required between then and the next tree update.


----------



## xtaz (Mar 6, 2018)

Ahhh that's a good answer. Thanks. OK. But I would question the wisdom of this. As Synth is kind of being used as a supposedly less resource intensive alternative to poudriere for people to build local ports with. But for that use it's now recursing the entire port tree every time you use it which negates this purpose.

If you're building the entire ports tree like the dragonfly or freebsd pkg builders would be then fair enough, I totally see your point. But if like most users you're only building a 100 ports or so it seems intensive.

Every 24 hours I update my ports tree and run a bulk build on the ports I have listed. Assuming no new ports get built poudriere is taking just over a minute. Lets say Synth takes 20 minutes I don't want to be doing that every 24 hours.


----------



## marino (Mar 6, 2018)

> But for that use it's now recursing the entire port tree every time you use it which negates this purpose.


False.  It's scanning the tree when the tree changes.  People don't update the tree, build one port, update the tree again, build another port.
I assume most people have the port upgrade and index creation on a cron job and do it unattended.

The real issue is that ports doesn't provide this information in the INDEX provided by portsnap.
It all goes back to a poor design.  You can blame synth for how it works around this obvious deficiency, but I wasn't going to completely rewrite synth to work in the hacky way poudriere works.

edit: and obviously, if ports manager would fix the real issues rather than obscuring performance tweaks inside poudriere, then every tool could benefit from a more efficient tree.  If I had time, I could post a make.conf that would speed up scan times by 75% easily, which is basically doing the same tricks poudriere is doing.  Oh, and maybe they could arrange the index to contains flavors (or a second index) so the index generation is done by the freebsd cluster once and doesn't have to be recreated by every user.


----------



## Mayhem30 (Mar 6, 2018)

I have `portsnap fetch update` & `synth status` on cron, but I still run them manually before I upgrade any ports to ensure I have the latest versions. So I'm seeing that 21 minute delay each time.

That's why I am hoping someone can figure out what we need to do with Synth to work with the new changes to the ports system and speed things up.


----------



## Mayhem30 (Mar 10, 2018)

So that's it? We need to switch to poudriere if we want to have the performance tweaks?

Is no one even going to attempt to see what is needed to be done with Synth (or make.conf file) in order to bring it up to speed?


----------



## topcat (Mar 10, 2018)

Perhaps some compromise could be reached? I find running ports-mgmt/portmaster directly to build is too risky (the path followed is not deterministic, depending on ports already on the system, and it installs dependencies even when the main build fails, which is sub-optimal). ports-mgmt/poudriere is nice, but I really enjoy the relative simplicity of ports-mgmt/synth's operation.


----------



## marino (Mar 13, 2018)

Mayhem30 said:


> So that's it? We need to switch to poudriere if we want to have the performance tweaks?
> 
> Is no one even going to attempt to see what is needed to be done with Synth (or make.conf file) in order to bring it up to speed?



I've been reluctant to point this out, but you've already said that you update ports via cron on a daily basis and update the flavor index at that time, but then invalidate that work by doing it manually because you *must* have the very latest ports tree and not one 6 hours old.   If you "settled" for a resolution of a daily tree update, you personally wouldn't have the issue.

There is ZERO quality-assurance on the ports tree.  Any given commit can break the index or cause many upstream ports not to build (because the port maintainer didn't bother to check for upstream breakage and just upgraded without regard, for example).  So in reality, the "up to the last second" update may very well be worse than one a few hours old.

The best system would be a ports tree updated every 3 days after an integrity test (with possible exceptions for major security fixes) but that would never happen unfortunately.


----------



## Mayhem30 (Mar 13, 2018)

marino said:


> I've been reluctant to point this out, but you've already said that you update ports via cron on a daily basis and update the flavor index at that time, but then invalidate that work by doing it manually because you *must* have the very latest ports tree and not one 6 hours old.



I have good reason to do this though. Almost *always* after a port has been updated, a patch is added an hour or 2 later tagged "fixed issue with ..." or something important. These updates are being done on a production server, so I need to make sure the port I'm updating is up to date.

The issue I'm having is finding time to do the updates, and when I do - I have to wait 20 minutes for the flavor index to be regenerated before I can get started.


----------



## Eric A. Borisch (Mar 13, 2018)

Mayhem30 said:


> Almost *always* after a port has been updated, a patch is added an hour or 2 later tagged "fixed issue with ..." or something important



Yes, but unless you’re a fortune teller, you have no way of knowing if there is an upcoming “fixed issue” patch, or a new version, etc. Put another way, what about the patch that will come an hour or two from “now”? Are you going to go through it all again then? I’d guess not - with the potential exception for a security fix, depending on your environment and exposure.

And if it’s a production system, you should consider a build and test on another system (or jail/VM) before applying the latest and “greatest” anyway.


----------



## rigoletto@ (Mar 13, 2018)

There are very little activity in ports around 2:00 (UTC), usually more like NONE.


----------



## marino (Mar 13, 2018)

Eric A. Borisch said:


> Yes, but unless you’re a fortune teller, you have no way of knowing if there is an upcoming “fixed issue” patch, or a new version, etc. Put another way, what about the patch that will come an hour or two from “now”? Are you going to go through it all again then? I’d guess not - with the potential exception for a security fix, depending on your environment and exposure.
> 
> And if it’s a production system, you should consider a build and test on another system (or jail/VM) before applying the latest and “greatest” anyway.



I was going to say pretty much exactly this, but you did it better.


----------



## Mayhem30 (Mar 14, 2018)

This has gone completely off topic. All I wanted to know was :

Is anything going to be done with Synth (or make.conf file) in order to bring it up to speed with poudriere?


----------



## marino (Mar 14, 2018)

Nothing will be done to synth.  The "workaround" is strictly a [profile]-make.conf adjustment.  I personally don't have time to work out the adjustment and apparently nobody else does either.  If you want to take the initiative, ask the poudriere maintainer nicely what poudriere is doing to activate that compiler cache code.


----------



## xtaz (Mar 14, 2018)

I have poudriere build logs and can see stuff like this in them. Is this what you mean?


```
_CCVERSION_921dbbb2=FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin

_ALTCCVERSION_921dbbb2=none

_CXXINTERNAL_acaad9ca=FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin "/usr/bin/ld" "--eh-frame-hdr" "-dynamic-linker" "/libexec/ld-elf.so.1" "--hash-style=both" "--enable-new-dtags" "-o" "a.out" "/usr/lib/crt1.o" "/usr/lib/crti.o" "/usr/lib/crtbegin.o" "-L/usr/lib" "/dev/null" "-lc++" "-lm" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "/usr/lib/crtend.o" "/usr/lib/crtn.o"

_OBJC_CCVERSION_921dbbb2=FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin

_OBJC_ALTCCVERSION_921dbbb2=none
```


----------



## Mayhem30 (Mar 14, 2018)

marino said:


> Nothing will be done to synth. The "workaround" is strictly a [profile]-make.conf adjustment.



The Poudriere maintainer had this to say about it :

While it's true that it is something that needs to be piped to make.conf, it should not be done by the user as the values may change in every execution. It needs to be done by the tool. You could wrap synth with a script that sets up a make.conf like this though until the tool learns about it.

https://github.com/freebsd/poudriere/blob/master/src/share/poudriere/common.sh#L2251-L2275


----------



## marino (Mar 15, 2018)

The hash might change but the values themselves should not.  
I'd bet taking the output xtaz provided (as an example) and defining the hash as "921dbbb2" (in that example) in the make.conf would work just fine.

The issue you'd see with that approach is when freebsd system is updated.  Then the old values continue (as they are now hardcoded).  

Maybe the linked tool will provide output for the profile make.conf, dunno.  I never used it.


----------



## Mayhem30 (Mar 15, 2018)

marino said:


> Maybe the linked tool will provide output for the profile make.conf, dunno. I never used it.


Yes, synth will need to be programmed to run /Mk/Scripts/ports_env.sh and export it in to the make.conf file.

Here is the output that code produces on my machine :

```
_CCVERSION_921dbbb2=FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin
_ALTCCVERSION_921dbbb2=none
_CXXINTERNAL_acaad9ca=FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin "/usr/bin/ld" "--eh-frame-hdr" "-dynamic-linker" "/libexec/ld-elf.so.1" "--hash-style=both" "--enable-new-dtags" "-o" "a.out" "/usr/lib/crt1.o" "/usr/lib/crti.o" "/usr/lib/crtbegin.o" "-L/usr/lib" "/dev/null" "-lc++" "-lm" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "/usr/lib/crtend.o" "/usr/lib/crtn.o"
CC_OUTPUT_921dbbb2_58173849=yes
CC_OUTPUT_921dbbb2_9bdba57c=yes
CC_OUTPUT_921dbbb2_6a4fe7f5=yes
CC_OUTPUT_921dbbb2_6bcac02b=yes
CC_OUTPUT_921dbbb2_67d20829=yes
CC_OUTPUT_921dbbb2_bfa62e83=yes
CC_OUTPUT_921dbbb2_f0b4d593=yes
CC_OUTPUT_921dbbb2_308abb44=yes
CC_OUTPUT_921dbbb2_f00456e5=yes
CC_OUTPUT_921dbbb2_65ad290d=yes
CC_OUTPUT_921dbbb2_b2657cc3=yes
CC_OUTPUT_921dbbb2_380987f7=yes
_OBJC_CCVERSION_921dbbb2=FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) Target: x86_64-unknown-freebsd11.1 Thread model: posix InstalledDir: /usr/bin
_OBJC_ALTCCVERSION_921dbbb2=none
ARCH=amd64
OPSYS=FreeBSD
_OSRELEASE=11.1-RELEASE-p8
OSREL=11.1
OSVERSION=1101001
PYTHONBASE=/usr/local
HAVE_COMPAT_IA32_KERN=YES
CONFIGURE_MAX_CMD_LEN=262144
HAVE_PORTS_ENV=1
```

Here is the bash script I used to get the information :

```
#!/usr/local/bin/bash

export SCRIPTSDIR="/usr/ports/Mk/Scripts"
export PORTSDIR="/usr/ports"
export MAKE="/usr/bin/make"
/bin/sh ${PORTSDIR}/Mk/Scripts/ports_env.sh |
grep '^export [^;&]*' |
sed -e 's,^export ,,' -e 's,=",=,' -e 's,"$,,'
```


----------



## Mayhem30 (Mar 15, 2018)

I rebooted the machine, threw the output in to the LiveSystem-make.conf file. It took 4 mins and 35 seconds to rebuild the flavor index.

This is from my machine running with a single non-ssd 1TB drive :

```
$ date && sudo synth status && date
Thu Mar 15 09:00:14 PDT 2018
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt
Thu Mar 15 09:04:49 PDT 2018
```


This is from my machine running dual SSD's in raid 0 :

```
$ date && sudo synth status && date
Thu Mar 15 09:12:02 PDT 2018
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
Total packages that would be built: 0
The complete build list can also be found at:
/var/synth/synth_status_results.txt
Thu Mar 15 09:15:44 PDT 2018
```

It only took 3 mins and 42 seconds. This is a huge performance boost.


----------



## marino (Mar 15, 2018)

Where is the hash getting defined?  ("921dbbb2")
I'd expect some variable to have that value.


----------



## Eric A. Borisch (Mar 15, 2018)

It's from Make's ${X:hash} when X=cc


----------



## marino (Mar 15, 2018)

man, talk about obscure.  I'm pretty knowable in bmake and I didn't even recognize that as a builtin make function.

P.S. I bet there's another 2 or 3 simple make.conf tricks that would cut off another 50+% of the scan time for freebsd ports still left to find.


----------



## marino (Mar 15, 2018)

maybe with some more research, maybe these tricks could be incorporated into synth, but I'd like to see the requirements laid out about what is needed to happen.


----------



## Mayhem30 (Mar 15, 2018)

Isn't this enough to incorporate it in to Synth now? Who knows if/when anymore tricks will be found.


----------



## marino (Mar 15, 2018)

A) I know for a fact there are more of these.  I fixed them for dports, remember?
B) I don't have the exact requirements, the step-by-step "do A, B, C".  To get that requires a lot of study on my part and right now I'm personally swamped, so I'm look for hand-holding here.  If people wait for me to take the initiative when I'm not personally affected, I don't know how long they'll wait.


----------



## marino (May 2, 2018)

Mayhem30@, I just released Synth 2.03 which automatically caches the (mostly compiler) variables that weren't already cached.  Once that is published and you upgrade to it, you can remove your make.conf modifications.


----------



## Mayhem30 (May 2, 2018)

Ok great, thank you very much for adding that in.


----------



## loki2012 (May 20, 2018)

Hello Marino,
I am having an issue with synth, trying to install python27 and python36. I can build both ports directly, but when trying a `synth just-build lang/python36`, it is failing with the following error:

```
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
===>  Script "configure" failed unexpectedly.
Please report the problem to python@FreeBSD.org [maintainer] and attach the
"/construction/xports/lang/python36/work/Python-3.6.5/config.log" including
the output of the failure of your make command. Also, it might be a good idea
to provide an overview of all packages installed on your system (e.g. a
/usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

Stop.
make: stopped in /xports/lang/python36
```

I am not sure why in the jail synth is having issue building the port.

Does it ring a bell for you?

The issue came recently after I deleted all packages and tried to reinstall from scratch all packages on my system.


----------



## marino (May 20, 2018)

you have to use ENTERAFTER=stage environment variable and go find the config.log when you get the prompt back.  view the config.log file with vi and look at the error that says the compiler doesn't work.   The log will contain the answer.

obviously synth builds python for everyone else.  There would be a lot of anger if everyone had this issue.


----------



## loki2012 (May 20, 2018)

Thanks Marino. As I said, python used to compile fine with synth on my system. It is only recently I have some troubles. I could spot three errors in the config.log file. I am not very familiar with compilation, but here it is:


```
configure:3900: cc -V >&5
cc: error: argument to '-V' is missing (expected 1 value)
cc: error: no input files
configure:3911: $? = 1
configure:3900: cc -qversion >&5
cc: error: unknown argument: '-qversion'
cc: error: no input files
```

and


```
configure:4083: $? = 0
configure:4090: ./conftest
Shared object "libintl.so.8" not found, required by "conftest"
configure:4094: $? = 1
configure:4101: error: in `/construction/xports/lang/python36/work/Python-3.6.5':
configure:4105: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
```

I checked that /xports/devel/gettext-runtime/work/gettext-0.19.8.1/gettext-runtime/intl/.libs/libintl.so.8 exists.
So it looks libraries are not configured correctly in the jail? What could I try?


----------



## marino (May 20, 2018)

loki2012 said:


> Thanks Marino. As I said, python used to compile fine with synth on my system.



This fact could not be more irrelevant.
edited: sorry, I misread it thinking you were still saying it built fine outside of synth.



> Shared object "libintl.so.8" not found, required by "conftest"



That's the problem.



> What could I try?



I would try deleting all packages in the synth package directory and rebuild everything.
added: and when you install it, use pkg update -f to force reinstallation of everything.


----------



## loki2012 (May 21, 2018)

So I tried to delete all the packages in the synth package directory, and also rebuild everything in synth, and still get the same issue and error message. I then tried to uninstall all packages, delete again all packages in the synth package directory, start from scratch with a new /usr/local  directory, and installed synth with pkg, then I upgrade synth with synth and now trying again to build python still leads to the same issue. I guess at some point I will have to reinstall the system from scratch, which I would like to avoid.


----------



## marino (May 21, 2018)

Loki, you don't have the "download from official packages if available" option set do you?
Are you sure you're building *everything* from scratch?

Another question, is your system is some kind of weird state, like a partial upgrade from an earlier release?


----------



## Simon Wagner (May 24, 2018)

Hi,

I also started getting failures from a few packages:

ftp/curl with
“
checking run-time libs availability... failed
configure: error: one or more libs available at link-time are not available run-time. Libs used at link-time: -lnghttp2  -lssl -lcrypto -L/usr/lib -lgssapi -lgssapi_krb5 -lheimntlm -lkrb5 -lhx509 -lcom_err -lcrypto -lasn1 -lwind -lheimbase -lroken -lcrypt -pthread -lz -lkrb5 -lgssapi -lgssapi_krb5 -lkrb5 -lgssapi -lgssapi_krb5 -L/usr/local/lib
“

security/sudo, lang/python27 and lang/python36 with
“
checking whether we are cross compiling... configure: error: in `/construction/xports/lang/python27/work/Python-2.7.15':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
“

The environment: Server with FreeBSD on metal, months ago synth built all tools including virtualbox without any hiccup. The learning, testing and writing of documentation happens in FreeBSD VMs so I can start from scratch and branch at will.

The setup: 
FreeBSD 11.1-RELEASE
ZFS (I do not care about data loss in these VMs, I want to learn/document setup- and maintenance steps for using it on the final metal)
portsnap auto
svnlite checkout https://svn.freebsd.org/base/release/11.1.0/ /usr/src/
freebsd-update fetch&install
Make ccache (the fails also happen without)
Make synth
Synth configured with defaults (except ccache when installed)
synth status/prepare-system/upgrade-system

Many other ports build without problems.

If memory servers correct, these builds inside VMs worked during the last weeks, but I can’t pin it on something that changed.
I can’t believe that it’s a general problem, as John already pointed out. But seeing Loki’s message, I feel good that it’s not something that only I fritzed up.

I test with setenv ENTERAFTER stage, and I also see libintl.so.8 ‘missing’ messages in config.log. But why only for these ports?
Inside the interactive mode, I see the file:
# ls -l /usr/local/lib/libintl.so.8.1.5
-rw-r--r--  1 root  wheel  55114 May 22 11:58 /usr/local/lib/libintl.so.8.1.5

Right before the libintl message, I also see the following:
“
configure:3486: cc -V >&5
cc: error: argument to '-V' is missing (expected 1 value)
cc: error: no input files
configure:3497: $? = 1
configure:3486: cc -qversion >&5
cc: error: unknown argument: '-qversion'
cc: error: no input files
“

I don’t know if this is “bad things” or to be expected.

As this is a built from scratch, the hints with deleting, rebuilding or partial upgrades seem moot to me.

By now I’m a little bit out of my league. To my (freebsd-)untrained eye, it looks like somethings is wrong with a library path or so, but why would other ports build flawlessly?
What information should I gather to point me in the right direction?

Thanks a lot
And
Best regards

Simon

PS: renamed config.log so it is accepted by the uploader


----------



## marino (May 24, 2018)

Since this is the 3rd report on python (caused by missing libintl.so.8) I would say somebody broke the ports tree but it only affects freebsd.
For some reason, people seem to blame synth whenever the ports tree get broken.
In other words, none of this are synth bugs.   
The person that last changed gettext  a few days should definitely be involved via email or bug report.


----------



## marino (May 24, 2018)

I booted in FreeBSD 11.1 and was able to build python36 without issue.  Gettext was built correctly.
Somebody is going to have to break into the build (e.g. env ENTERAFTER=stage synth test lang/python36 or env ENTERAFTER=install synth test devel/gettext-devel) and go over and inspect libintl.so.  Look at symlinks, look at readelf -a (e.g. Library soname) and see why python configure script is failing.

It's working for me.  I can't reproduce any of this on freebsd hardware.


added: Are you guys on FreeBSD 12.0-CURRENT?  If so, I would suspect a bleeding edge bug in the rtld.


----------



## Simon Wagner (May 24, 2018)

Hello John,
Not blaming synth (sorry if it came across like that), but working the onion from the outside. First I have to figure out how to parse synth output to find out where to look next.
And I'm back to square one because the last change I see in gettext would be from 2016. And python27 from 2017. While I absolutely agree with you, I have no clue about what to do next.
But I also do not expect you to lecture me, I'll just wait by the sidelines and see how this develops. Perhaps someone will post back here so 'we' know in the future how to chase synth error messages and catch the real culprit.

Grabbing the popcorn
simon

PS:
Someone found a bug in ftp/curl that looks familiar, but they claim that dns/libpsl is to blame. I would read the error message differently as gettext being the bad guy, but that's just me.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227909


----------



## marino (May 24, 2018)

That bug report says "rebuild all packages" which was the very first thing I suggested.
But to confirm, you are not on FreeBSD 12.0 ?
Oh, you're right on gettext.  I thought it was updated 2 weeks ago, but it was 2016.

added: if rebuilding all packages is needed, it means somebody failed to bump downstream libraries when a dep library was updated.  It's a maintainer error.
added2: we posted at same time, did you see the comment before?


----------



## loki2012 (May 24, 2018)

So I have some finding now.
The building of python started to fail after I switch from libressl back to openssl by commenting this line in my /usr/local/etc/synth/LiveSystem-make.conf

```
#DEFAULT_VERSIONS+= ssl=libressl
```
So with this line commented, python27 doesn't build. If I uncomment this line, libressl is being built and python is building as well.

The reason why I tried to come back to openssl rather than libressl was because since the update to libressl 2.7, py-cryptography is not building any more and apparantly it is a known bug.
I am under Freebsd 11.1


----------



## marino (May 24, 2018)

loki2012@, with a change like that, you have to remove all packages from synth package directory and rebuild everything.  You also cannot have configuration item[FONT=Courier New] [N] Fetch prebuilt packages[/FONT] set to "true".

Check your configuration and make sure [N] is set to false, and remove *ALL* packages from the directory indicated by [FONT=Courier New][ B ] packages directory[/FONT].
I built python just fine with default settings, which is openssl.


----------



## Simon Wagner (May 24, 2018)

[FONT=Courier New]root@zfs-default:~ # freebsd-version 
11.1-RELEASE[/FONT]

No 12 here ;-)

John, I'll try the enterafter digging tomorrow at the office, let's see how far I get. Perhaps I'll shove another SSD into the server and do a vanilla install there to rule out virtualisation (with thorough protocol to enable replay by other parties).

Loki, can you build security/sudo and ftp/curl without errors? Perhaps better to tackle it with a 'lesser' tool than python...


----------



## rigoletto@ (May 24, 2018)

TLDR; to build ftp/curl with security/libressl the TLS_SRP should be disabled.


----------



## marino (May 24, 2018)

yep.  in dports we set TLS_SRP off by default for this very reason.


----------



## Simon Wagner (May 24, 2018)

Not using libre. It's a vanilla 11.1 install using openssl in ftp/curl.


----------



## zirias@ (May 25, 2018)

loki2012 said:


> since the update to libressl 2.7, py-cryptography is not building any more and apparantly it is a known bug.


And in the corresponding PR, there's a patch attached -- worked fine for me, so, no reason to switch back to openssl.


----------



## loki2012 (May 26, 2018)

Zirias said:


> And in the corresponding PR, there's a patch attached -- worked fine for me, so, no reason to switch back to openssl.


Thanks for that, indeed that works.



marino said:


> loki2012@, with a change like that, you have to remove all packages from synth package directory and rebuild everything.  You also cannot have configuration item[FONT=Courier New] [N] Fetch prebuilt packages[/FONT] set to "true".
> 
> Check your configuration and make sure [N] is set to false, and remove *ALL* packages from the directory indicated by [FONT=Courier New][ B ] packages directory[/FONT].
> I built python just fine with default settings, which is openssl.


I did as you suggested. [N] was always set to false since the begining. But the issue was still there. Using libressl instead of openssl, python is building but l had issues building other ports, such as glib20, postfix (once I sorted-out the py-cryptography issue). So I guess something is still wrong with my system. I will go on trying to understand what could be wrong, but I don't remember having a special setup. Just a 11.1-RELEASE updated from previous releases, and the issues appeared a few months after I upgraded to 11.1. I will post if I can find what was the issue. Thanks for you help Marino anyway, I learned a few things in the process.


----------



## loki2012 (May 26, 2018)

Simon Wagner said:


> Loki, can you build security/sudo and ftp/curl without errors? Perhaps better to tackle it with a 'lesser' tool than python...


let me try to build these two ports.

Edit: These two ports don't build as well on my side.


----------



## Mayhem30 (May 29, 2018)

Could you please add a changelog to your GitHub repo so we know what's been updated?

It's hard to tell what's new in 2.05 by looking at the code.


----------



## loki2012 (May 30, 2018)

I am happy to report that my issues are gone with synth-2.05. Looking at GitHub there was probably some patch on the way ldconfig was called in the jail, in order to address a recent change in the Makefile in ports, if I read it correctly.


----------



## tobik@ (May 31, 2018)

Mayhem30 said:


> Could you please add a changelog to your GitHub repo so we know what's been updated?
> 
> It's hard to tell what's new in 2.05 by looking at the code.


There is a very brief Changelog here:
https://github.com/jrmarino/synth/releases/tag/v2.05

You can also always use GitHub's compare function:
https://github.com/jrmarino/synth/compare/v2.04...v2.05


----------



## Simon Wagner (Jun 1, 2018)

loki2012 said:


> I am happy to report that my issues are gone with synth-2.05.


I get birds.... meToo!

Upgraded synth, everything compiles.


----------



## hrenznaet (Jun 23, 2018)

Does synth's action 'upgrade-system' respect 'pkg lock's?
I locked my firefox-esr, but I fear that synth upgrade-system will update it anyways.


----------



## marino (Jun 23, 2018)

"upgrade-system" just calls the pkg command, so yes, it has to respect locks.
(the use of locks is another topic entirely.  be prepared for issues because of it)


----------



## hrenznaet (Jun 23, 2018)

marino said:


> "upgrade-system" just calls the pkg command, so yes, it has to respect locks.
> (the use of locks is another topic entirely.  be prepared for issues because of it)


Thanks for the answer.
Could you name those issues, please?
It's just I'm on the edge with FreeBSD already and if it will update firefox-esr irregardless of my wish not to do so - I'll just finally end my experience with FreeBSD for good.


----------



## marino (Jun 23, 2018)

it's an unreasonable expectation.  If you use binary packages, those packages are going to update from time to time.  All the dependencies will update.  If you lock a package but allow everything else to update, the library linkages will get broken.   Basically locking doesn't really work, certainly not like a newbie user might expect.

Somebody that wants to freeze package versions needs to build from source.  They either freeze the entire source tree at some point, or they manually keep a port at a specific version and use poudriere/synth to keep the local package repository working together.


----------



## hrenznaet (Jun 23, 2018)

I use only the packages I built from ports.
I would like not to also lock packages, that firefox-esr is based on.
I guess what I want is statically built firefox/firefox-esr, but now that ports tree has more recent versions than I need - I guess I need to download an older instance of the ports tree first.
Oh, god, another quest...


----------



## marino (Jun 24, 2018)

if that's the case, you have no need for package locking.  You just need to lock the source port.  Right?


----------



## hrenznaet (Jun 24, 2018)

I was not aware of such an option.
But I'd still need to build a static version of port, because I don't want to lock dependencies.


----------



## marino (Jun 24, 2018)

Who said anything about locking dependencies?  If you build it yourself, the issue that binary packages have disappears.


----------



## hrenznaet (Jun 26, 2018)

marino said:


> Who said anything about locking dependencies?  If you build it yourself, the issue that binary packages have disappears.


Are you saying is that all I need in my case is just lock 1 port? Could you, please, tell how to do that? I've googled that, but all I've found was pkg lock.
Also, if I lock a port - will it stop synth from building more recent versions of it (luckily, it doesn't attempt to install them, but it still wastes time building something I won't ever need)?


----------



## marino (Jun 26, 2018)

You're just not getting this concept.
If you build from source, and install those local packages, they're always going to work together.  The only thing you have to "lock" is the source directory (in your case firefox-esr).  So you use subversion to update the ports source tree (not portsnap) and just make sure that the firefox-esr port is downgraded and build everything you want.  Don't use official packages at all.  Then you don't have to mess with pkg locks at all.


----------



## Crivens (Jun 26, 2018)

I seem to remember that you can also build a PBI from ports. That would be firefox and all dependencies in one directory. That way, you might end up with dozens of copies of f.e. libc, but only if you have dozens of pbi's. Maybe you need to check in that direction?


----------



## hrenznaet (Jul 3, 2018)

marino said:


> The only thing you have to "lock" is the source directory (in your case firefox-esr).  So you use subversion to update the ports source tree (not portsnap) and just make sure that the firefox-esr port is downgraded and build everything you want.  Don't use official packages at all.  Then you don't have to mess with pkg locks at all.


I have 0 experience with subversion, I'd like not to mess with it.
Also, from what I googled: 'SVN does not allow directories to be locked. You can lock individual files, but not an entire directory tree.'

I think there should better be a way for synth to ignore (not attempt to build them) specific ports when doing 'upgrade-system' apart from manipulating files under /usr/ports/.
I'd rather manually manage some exclusion list in some file.

Or, after all, why not teach 'synth upgrade-system' to treat locked packages as a signal that user doesn't want synth to attempt build them? I think that'd be quite a logical behavior.


----------



## hrenznaet (Jul 3, 2018)

Crivens said:


> I seem to remember that you can also build a PBI from ports. That would be firefox and all dependencies in one directory.


I'm not familiar with that abbreviation.


----------



## rigoletto@ (Jul 3, 2018)

The svn(1) `lock` is not what you are thinking it is. 

To do what you want, you need to switch to the svn tree then 'update' the desired port (directory) to the svn revision version you want. But be aware after some time that would require to be 'updating' several other things to keep the desired package building due to dependecies, and then it will start breaking other stuff due to the same reason...


----------



## marino (Jul 4, 2018)

i wasn't using "lock" literally.  lebarondemerde is right in that you manually set the port to be frozen.  There are many ways to do it.  One is with svn at a directory level.  Another is just copy files over.  But some things in the ports tree get changed/deprecated and even frozen source ports start generating errors because the original functionality isn't supported anymore.

It's basic ports-fu.  Once you start talking about not using the default binary packages, then you're basically committing to learn source ports (and how to manipulate the tree via subversion)


----------



## hrenznaet (Jul 6, 2018)

Well, then it's a shame that there's no proper tool to manage ports in FreeBSD.
For now I just found firefox52 port sources online, downloaded it and now I do `portsnap fetch update && rm -rf /usr/ports/www/firefox-esr/ && cp ~/backup/firefox-esr /usr/ports/www/` and only then I can use a tool to mass upgrade ports.


----------



## marino (Jul 6, 2018)

What you are doing is not standard nor recommended.  Why would there be a tool dedicated to do something that is sure to eventually fail?  Plus more than half your issue is that you're not close to being an expert with customizing ports, yet you're trying to do exactly that, then blaming something vague for not having tools to do it.

Plus I already told you that using portsnap is not the way to go if your trying to lock down sources.  The next time that port is updated, it wipes out your changes.

plus plus, why do you need to find random "sources online" when you have SVN and can acquire exact versions from specific dates?  If you use the proper tools, at least the non-recommended expert tasks will be easier to do.


----------



## abishai (Jul 7, 2018)

The are no proper tools in Unix, there are possibilities, you can
1. Fork port tree and sync with your own repo
2. Restore customized ports with rsync or simple sh script.
3. portsnap and ports-mgmt/portdowngrade


----------



## hrenznaet (Jul 8, 2018)

You seem to pay too little attention, marino.


marino said:


> What you are doing is not standard nor recommended.  Why would there be a tool dedicated to do something that is sure to eventually fail?


Makes no sense. I want to preserve a port in an exact version. For reasons. That's a simple thing to ask for, because otherwise it's a dictate (the system tells me what I should be using instead of what I want to use).
Are you telling me there is no Freedom in FreeBSD?



marino said:


> Plus more than half your issue is that you're not close to being an expert with customizing ports, yet you're trying to do exactly that, then blaming something vague for not having tools to do it.


I don't customize firefox, I don't know where you've taken this from, so I just ignore this point of yours.



marino said:


> Plus I already told you that using portsnap is not the way to go if your trying to lock down sources.  The next time that port is updated, it wipes out your changes.


That's why there are 
	
	



```
rm
```
 and then 
	
	



```
cp
```
 operations after the 
	
	



```
portsnap
```
 one.



marino said:


> plus plus, why do you need to find random "sources online" when you have SVN and can acquire exact versions from specific dates?  If you use the proper tools, at least the non-recommended expert tasks will be easier to do.


Because I'm rational: it was an easier way, because I didn't need to build&install a tool to work with SVN and I didn't need to learn how to work with that tool. The online source with the port is owned by FreeBSD anyways.


----------



## rigoletto@ (Jul 8, 2018)

hrenznaet said:


> Are you telling me there is no Freedom in FreeBSD?



Create your own ports tree and maintain it. Nobody is holding you.


----------



## marino (Jul 8, 2018)

hrenznaet said:


> You seem to pay too little attention, marino.
> 
> Makes no sense. I want to preserve a port in an exact version. For reasons. That's a simple thing to ask for, because otherwise it's a dictate (the system tells me what I should be using instead of what I want to use).
> Are you telling me there is no Freedom in FreeBSD?



As lebarondemerde implies, this is a troll statement, one that completely ignores the reality how of the ports system works.  It's interconnected with a framework.  When the framework changes, the entire ports tree is modified.  Each version of the tree works together.  If you "freeze" any leaf of that tree, it will eventually be incompatible with the core framework.   You might as well say car manufacturers are taking away your freedom because the don't offer 400 miles to the gallon mileage.




> I don't customize firefox, I don't know where you've taken this from, so I just ignore this point of yours.



Hello!  If you freeze the port so its differs from portsnap, you've customized it.  It's not a very hard concept to grasp.




> That's why there are rm and then cp operations after the portsnap one.



Or just could just, I don't know, learn the basics of svn which should take about half an hour.
Plus its loads faster than portsnap usually.



> Because I'm rational: it was an easier way, because I didn't need to build&install a tool to work with SVN and I didn't need to learn how to work with that tool. The online source with the port is owned by FreeBSD anyways.



If you don't want to put in the time to gain expertise, then don't do expert-level work.   I've paid all the attention I need to this subject.  I grasp it very well.


----------



## Crivens (Jul 8, 2018)

Gentlemen, please behave. I really don't want to put a peg into this long thread, but if the bickering gets out of hand...


----------



## marino (Jul 8, 2018)

it should probably be locked anyway.  The original thread was about announcing synth, but now it's basically used to ping my attention since it's clear I get email notifications when somebody posts to it.  Ideally all these new posts should be individual threads.


----------



## hrenznaet (Jul 8, 2018)

lebarondemerde said:


> Create your own ports tree and maintain it. Nobody is holding you.


I don't want to. I just want to freeze a particular leaf of the tree, but marino says it's a bad idea (read below on that).



marino said:


> It's interconnected with a framework.  When the framework changes, the entire ports tree is modified.  Each version of the tree works together.  If you "freeze" any leaf of that tree, it will eventually be incompatible with the core framework.


Freezing a leaf is just a trick to fool synth into not building firefox, because otherwise it compiles firefox of newer version, which I DON'T WANT. And this already resulted once into firefox stopping to work (it built newer version of firefox, didn't install it [AFAIU because I pkg lock'ed it], didn't uninstall the old version, but the old version refused to start due to some lib missing).
I wouldn't have to trick it if I knew how to tell it to not build firefox upon `upgrade-system`.

What if I follow your idea to create a separate ports tree (I don't need the whole tree, I need just a leaf of it) that will not be tied - will synth finally stop attempts to rebuild firefox upon upgrade-system?



marino said:


> Hello!  If you freeze the port so its differs from portsnap, you've customized it.  It's not a very hard concept to grasp.


I disagree with the wording (I didn't customize the port, but you may say that I customized ports tree), but I got the idea.



Crivens said:


> Gentlemen, please behave. I really don't want to put a peg into this long thread, but if the bickering gets out of hand...


I don't see anyone misbehaving, we are having a discussion and figuring and fixing out the misunderstanding (mostly mine). Although it may seem as an unpleasant talk - in fact it isn't and these guys are very civil.



marino said:


> it should probably be locked anyway.  The original thread was about announcing synth, but now it's basically used to ping my attention since it's clear I get email notifications when somebody posts to it.  Ideally all these new posts should be individual threads.


In my opinion it's better to keep 1 thread per port discussion. I did seek your attention, indeed. I didn't know replies here bothered you much, but if the emails really are that bothering - I think you may unsubscribe from this thread if the forum supports it. Otherwise create a rule for emails on this topic.


----------



## Crivens (Jul 8, 2018)

hrenznaet maybe this is what you need. The PBIs I mentioned are no longer supported, I'm afraid.


----------



## rigoletto@ (Jul 8, 2018)

Crivens

There is a Haskell Nix implementation too, HNix (BSD3CLAUSE). 

Now we would just need marino to cheer up to write a Ada one.


----------



## Eric A. Borisch (Jul 10, 2018)

hrenznaet said:


> And this already resulted once into firefox stopping to work (it built newer version of firefox, didn't install it [AFAIU because I pkg lock'ed it], didn't uninstall the old version, but the old version refused to start due to some lib missing).



This. This is the point. You didn’t upgrade Firefox (you had it locked) but one of the dynamic libraries it depends on changed underneath it, and the old Firefox would no longer run. This is exactly what everyone is saying when they talk about the ports tree describing a framework that needs to be kept in sync as a whole.

A new build of Firefox (after the underlying library changed) would compile and link against the new library / interface.

If you want to stick at a particular version of an application that depends on other libraries, you can’t allow the libraries under it to change, either, if you want to ensure it still works.


----------



## hrenznaet (Jul 10, 2018)

Oh, so I'm doomed to either
- not upgrade the ports in my system at all;
or
- upgrade selected ports hoping they won't cause a break in firefox again via [firefox dependencies, firefox dependencies dependencies, firefox dependencies dependencies dependencies, ...] and so on.

Neither of options is optimal for me.

I hoped there's a way to build a package statically (building it in such a way that it'd include every dependency, so it becomes fat, but completely independent from the packages in the system).
Or like a 'portable app' (as an ex-windows user I refer to apps like the ones distributed by portableapps.com).


----------



## rigoletto@ (Jul 11, 2018)

Crivens 

Btw, since a couple weeks I've seen many people (IRC mostly) talking about desire of a FreeBSD [H]Nix implementation. Curious.


----------



## marino (Jul 11, 2018)

hrenznaet said:


> Oh, so I'm doomed to either
> - not upgrade the ports in my system at all;
> or
> - upgrade selected ports hoping they won't cause a break in firefox again via [firefox dependencies, firefox dependencies dependencies, firefox dependencies dependencies dependencies, ...] and so on.
> ...



This means you didn't comprehend what I was telling you.  If you lock a binary package, you're going to have issues with dependencies updating underneath it.  If you build from source (by controlling the source tree, e.g. with svn) then you never have the issue of the dependencies being mismatched.  The issue you have is the locked source port falls out of sync with the tree.  Two totally different consequences to two totally different attempts of freezing a port.

For the latter, you can update the frozen port to keep working.  But you're doing port maintenance at that point.

And nobody said "don't update ports at all".  At best you could interpret this as "don't use firefox if you insist on an obsolete version".
or alternatively, use ravenports that will have firefox 52 for a long time.


----------



## vit. (Jul 13, 2018)

hrenznaet said:


> to either
> -


Effectively, you're talking about a container  where all app dependencies are shipped together with the app. Like Ubuntu snaps or Windows side-by-side. With the app package being completely isolated from the rest of the system. Which is doable but requires one to maintain a custom port builds tree / deployment location. In other systems, that custom deps tree is implicitly maintained by somebody else, like the app developers themselves, but it does exist. 
On FBSD you'll have to maintain it yourself. Then you can deploy it in a jail, as one of the options.


----------



## Mayhem30 (Jul 24, 2018)

I'm having an issue and not sure how to move forward with it.

```
$ sudo synth status
Regenerating flavor index: this may take a while ...
Scanning entire ports tree.
 progress: 35.51%             
culprit: emulators/xen
  Scan aborted because 'make' encounted an error in the Makefile.
  emulators/xen (return code = 1)
Flavor index generation failed: ports scan
```
The /usr/ports/emulators/xen directory does not exist. What should I do now?


----------



## marino (Jul 24, 2018)

see: https://www.freshports.org/emulators/xen
open bug report immediately and make sure royger@FreeBSD.org is alerted to the fact that he did not properly remove the xen port.
see line 179: https://svnweb.freebsd.org/ports/head/emulators/Makefile?revision=471283&view=markup&pathrev=475253

There is a procedure to remove/rename ports and that procedure includes editing the category makefile (emulators/Makefile).  

Just another case of Synth catching a maintainer mistake.


----------



## Mayhem30 (Jul 24, 2018)

Thank you.


----------



## nov1c3 (Aug 7, 2018)

Hello,

Would it be ok if the version of FreeBSD where Synth is installed is higher than the FreeBSD version of the servers I'm building pkg's for? For example the server where Synth is used runs 11.2-RELEASE and all other servers run 11.1-RELEASE.

Thank you.


----------



## marino (Aug 7, 2018)

it should work at least 99% of the time.  Ideally the builder would be the older machine though.  It's guaranteed that packages from older minor releases work on newer minor releases, but not the other way around.


----------



## mfoacs (Aug 13, 2018)

Probably an old topic but I couldn't find an answer: every time I `synth upgrade-system`, it ignores sysutils/lsof and sysutils/htop. There are no logs to look into - that's what "ignored" really means I guess -, so I move to a manual install with `synth install sysutils/lsof`. This results in "The most recent version of packages are already installed", which again, is true.

With sysutils/htop however, there is something else:  `synth install sysutils/htop` or any of "build, force" gives me "pkg: No packages available to install matching 'htop-2.2.0' have been found in the repositories". It seems to make sense since htop requires lsof and nothing was built, so synth is happy, but I am confused...

I had no issues installing both of the packages it with pkg, portmaster or make install.

And that begs the question: if the package is already installed with the latest available version why is synth selecting it for an upgrade? and since lsof it's a dependency from another package - also already installed and up to date, shouldn't synth ignore them both completely?


----------



## marino (Aug 13, 2018)

the log starting with 00_* will explain why its getting ignored.  go to logs directory, find it, open it, and look for for lines involving lsof and htop.


----------



## mfoacs (Aug 15, 2018)

marino said:


> the log starting with 00_* will explain why its getting ignored.  go to logs directory, find it, open it, and look for for lines involving lsof and htop.



Hmm, yes that makes sense: lsof requires the kernel sources and I've installed the kernel sources on a non-standard location. Lesson learned.


----------



## rigoletto@ (Aug 15, 2018)

Just to remember you can also set a small web server and use the Synth web interface.


----------



## Mayhem30 (Aug 15, 2018)

lebarondemerde said:


> Just to remember you can also set a small web server and use the Synth web interface.



Great suggestion! I use it myself and it's very handy


----------



## rigoletto@ (Aug 25, 2018)

I am wondering if with a fsevents/inotify facility ports-mgmt/synth could use it be aware of the ports tree changes...


----------



## free-and-bsd (Aug 26, 2018)

Hello, I'm having this funny message upon the completion of a `synth install` command:

```
The task is complete.  Final tally:
Initial queue size: 2
    packages built: 2
           ignored: 0
           skipped: 0
            failed: 0
Duration: 00:01:35
The build logs can be found at: /var/log/synth
Stand by, recursively scanning 2 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
pkg: Unknown repository: Synth
Unfortunately, the system upgraded failed
```
Interestingly, this last pkg error prevents pkg from installing whatever has been built by synth.
Interesting, too, that when I serve the same repository via httpd to other computers, the local pkg find no problems updating from it.
It's pkg-1.10.5_1 and synth-2.05.

Here is my repo.conf which pkg pretends not to see (although it did see it until now):

```
# Automatically generated.
Synth: {
  url      : "file:///var/synth/11.1-packages",
  priority : 0,
  enabled  : yes
}
```


----------



## free-and-bsd (Aug 26, 2018)

free-and-bsd said:


> Hello, I'm having this funny message upon the completion of a `synth install` command:
> 
> ```
> The task is complete.  Final tally:
> ...


OK, I've fixed it  Just mixed up a bit the repo.conf files. Now they look fine and everything is running again.
Sorry for the false alarm... just when I get nervous I become distracted and can mess up things a bit.


----------



## Mayhem30 (Sep 14, 2018)

Hi Marino, I've run in to a weird problem.

I've updated my make.conf file to use the security/openssl111 port and Synth only want to build the new OpenSSL version, and not any of the 70 ports that depend on it.

Any ideas what I am doing wrong?

/usr/local/etc/synth/LiveSystem-make.conf :

```
OPTIONS_UNSET = X11 CUPS

DEFAULT_VERSIONS+=ssl=openssl111 php=7.1 mysql=5.6
```


```
$ sudo synth status
Querying system about current package installations.
Stand by, comparing installed packages against the ports tree.
Scanning existing packages.
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  N => security/openssl111
Total packages that would be built: 1
```


----------



## aht0 (Sep 14, 2018)

I think these 70 ports may need adjustments because OpenSSL internals changed and just forcing a new version on ports would not just work. So they are still hardwired to use older openssl version. Just wait until port maintainers perform their magic.


----------



## SirDice (Sep 14, 2018)

openssl111 is not a valid option:

```
# Possible values: base, openssl, openssl-devel, libressl, libressl-devel
```
See /usr/ports/Mk/bsd.default-versions.mk


----------



## kpa (Sep 14, 2018)

It's not listed as a valid option there but it does work as a custom setting for selecting an exact OpenSSL port to be used. Lot of ports seem to break though at the moment with that setting, I was lucky to have only one broken port and I don't really need that one.


----------



## SirDice (Sep 14, 2018)

Right, I had to dig through /usr/ports/Mk/Uses/ssl.mk to see what's happening. The code itself doesn't actually check what's put in, it simply uses security/${SSL_PORT}. So it automagically works for anything you put in. It actually allows bad input too, setting SSL_DEFAULT=gnutls for example, will actually install security/gnutls but then fail later during dependency checks. Interesting.


----------



## mfoacs (Sep 20, 2018)

Another one: synth is failing to install databases/mongodb36.
`/usr/bin/ld: final link failed: No space left on device
c++: error: linker command failed with exit code 1 (use -v to see invocation)
scons: *** [build/opt/mongo/mongoperf] Error 1
scons: building terminated because of errors.
build/opt/mongo/mongoperf failed: Error 1
*** Error code 2

Stop.
make: stopped in /xports/databases/mongodb36

--------------------------------------------------
--  Termination
--------------------------------------------------
Finished: Thursday, 20 SEP 2018 at 18:13:42 UTC
Duration: 01:00:57`

At some point I see synth using 99.9% of the 9G swap available, but only roughly 10% of RAM.
The machine has 12 cores, 20G RAM and fast enough disks so I guess my synth configuration is not optimal.
I currently have 6 builders, 3 jobs per builder and I could, according to the man pages, go up to 12*0.75=9 builders and 4 jobs per builder.
What else can I do? disable tempfs?


----------



## marino (Sep 20, 2018)

The rule of thumb is swap should be ideally be set at 4 times the ram, so 80G in your case.
If you build mongo by itself, you should be okay.  one builder, 3 jobs, 20G ram, I would be surprised if you touched swap at all.  
Don't disable tmpfs.


----------



## Mayhem30 (Sep 20, 2018)

mfoacs - if you run "top", what does your memory usage show when Synth is not building?

I have 6 builders, 4 jobs per builder and it only uses ~4GB ram (no swap).


----------



## mfoacs (Sep 21, 2018)

Mayhem30 said:


> mfoacs - if you run "top", what does your memory usage show when Synth is not building?
> 
> I have 6 builders, 4 jobs per builder and it only uses ~4GB ram (no swap).



Here you have it:

`Mem: 6160K Active, 2536M Inact, 2315M Wired, 1526M Buf, 14G Free
Swap: 9670M Total, 9670M Free`


----------



## freebsdinator (Nov 1, 2018)

Forgive me if this is the wrong place for my question, but it appears Synth allows you to have different profiles. Would you be able to have a profile for server and another for desktop (same arch)?

Also, this tool builds the entire ports tree, correct? If so, what size are we talking about here? 3TB, 6TB, 12TB? I don't suppose you can select part of the ports tree to build? IE: Say you want to omit games/xxxxxx ports.

I apologize if these were asked earlier. I tried to lookup size + synth on search and couldn't find much context aside from people asking how this tool is different from pkg or portmaster over and over.

edit: I think I found my answers at: http://www.kaba1ah.org/2017/07/31/managing-ports-for-multiple-freebsd-servers-2/ I understand I can just feed synth a list of ports to keep track of. I would be curious if anyone knew the size of every compiled package. I'm also trying to understand if synth is dynamic enough to build a package for a client that requests a port to be installed or if I have to go to synth first.


----------



## cbrace (Nov 4, 2018)

Hi all,

Is there are recommended way to use synth to upgrade the version of php in use? My site is currently running php v7.0 and I'd need to move to v7.1 or higher.

In the past, using ports-mgmt/portmaster etc I kept running into problems doing this, ie, it took several attempts to upgrade versions of PHP and Perl. It is one of those tasks that I as a very part-time sysadmin only undertake from time to time; I forget how to do it properly and end up breaking my Joomla, Roundcube, and Nextcloud setup.

Yesterday I tried simply installing lang/php71 using synth, hoping it would update the dependencies according, but yet again I ran into problems, and had to revert to php70 to get everything working again.

Thanks for pointing me in the right direction.


----------



## rigoletto@ (Nov 4, 2018)

You need to set the php default version in make.conf, then upgrade the system.

`DEFAULT_VERSIONS+=php=72`


----------



## cbrace (Nov 6, 2018)

I've added that line to /etc/make.conf, thanks.

Reinstalling nextcloud, I'm prompted for the flavor, so I choose nextcloud@php72, obviously.

But when I try to reinstall joomla3, there are no flavors, and synth deletes all the php72 dependencies and reinstalls the php70.   

Once again, I have reverted to php70 to run Joomla.


----------



## Mayhem30 (Nov 6, 2018)

When using synth, you need to add `DEFAULT_VERSIONS+=php=72` in your /usr/local/etc/synth/LiveSystem-make.conf file.

That should fix your issue.


----------



## free-and-bsd (Nov 10, 2018)

freebsdinator said:


> I'm also trying to understand if synth is dynamic enough to build a package for a client that requests a port to be installed or if I have to go to synth first.


My, but that would be too damn "dynamic", wouldn't you think so? I'm afraid your help will be needed for Synth to understand which particular port the client requests.

Seriously, though, the degree of the supposed "dynamism" can be imagined. For example, does it make any sense building a separate package for an unknown machine without building the dependencies? So, the dependencies WILL be needed. But then, up to what depth? What packages/versions are already installed on the target client machine? 

So, you create a corresponding PROFILE and keep the needed packages in sync with the ports tree -- then you can add another package to that repository. Then you make that repository available to your client machine and then install whatever packages that machine would need. Your client's pkg command will know which dependencies from the repo it needs and which are already installed. I hope that covers it more or less.


----------



## Sevendogsbsd (Nov 11, 2018)

So, was a bit of work, but set up a build server so I can custom build ports --> packages for my personal workstation. Initially used ports-mgmt/*poudriere* but I find ports-mgmt/*synth* to be faster at builds and easier to manage. So far so good, works like a charm! Got my repository set up and my workstation happily uses it. Only bad part was the initial set up of figuring out which packages I had used, configuring the ones I needed to, then building a file with the list so ports-mgmt/*synth* could build them and create the repo. Now it's just maintenance so no worries.


----------



## Mayhem30 (Nov 23, 2018)

When you run `synth status`, does the display order show what the dependencies are?

For example :

```
These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  N => security/openssl111
  R => lang/python27
  ...
  ...
  N => security/openssl
  R => security/py-fail2ban@py36
  ...
  ...
```

Does the above mean that security/py-fail2ban depends on security/openssl port?

I'm curious why fail2ban won't compile using security/openssl111 instead.


----------



## Sevendogsbsd (Dec 8, 2018)

Quick question: does ports-mgmt/synth support cryptographic verification of the repository?


----------



## cbrace (Dec 11, 2018)

After updating my VPS to v12.0-RELEASE today using freebsd-update, I am trying to update the ports, but synth is choking on something:


> $ sudo synth upgrade-system
> Builder mounts detected; attempting to remove them automatically ...
> Dismounting successful!
> Regenerating flavor index: this may take a while ...
> ...


Any ideas about what is going wrong here?


----------



## freebsdinator (Dec 13, 2018)

https://github.com/jrmarino/synth/issues/145



> it may be that ntpd needs to be added add to the default users....
> 
> You can probably modify /etc/mtree/BSD.var.dist to remove reference to ntpd to get around it for now.


----------



## aht0 (Sep 8, 2021)

Is ports-mgmt/synth still receiving maintenance-love?

A) It's repo conf file in /usr/local/etc/pkg/synth is being ignored by FreeBSD 13-RELEASE. You'd need to write it's content manually into conf file in /etc/pkg for synth repo to be visible for OS.

B) Used to be that only first package builds (upgrade-system or some package install) were all-inclusive and after updating ports and issuing it again - only ports installed and with any changes were rebuilt. Now ports-mgmt/synth seems to build same packages over and over completely uncaring whether packages of same version&options already exist in repo or not.


----------



## free-and-bsd (Sep 9, 2021)

aht0 said:


> Is ports-mgmt/synth still receiving maintenance-love?
> 
> A) It's repo conf file in /usr/local/etc/pkg/synth is being ignored by FreeBSD 13-RELEASE. You'd need to write it's content manually into conf file in /etc/pkg for synth repo to be visible for OS.
> 
> B) Used to be that only first package builds (upgrade-system or some package install) were all-inclusive and after updating ports and issuing it again - only ports installed and with any changes were rebuilt. Now ports-mgmt/synth seems to build same packages over and over completely uncaring whether packages of same version&options already exist in repo or not.


A)I guess it IS ignored. The correct path is /usr/local/etc/pkg/repos/. There you place your Synth.conf and FreeBSD.conf.

B)Actually, it rebuilds _changed_ ports + ANY ports that _could possibly be_ affected. There is some policy about it which seems to be common to all package site build tools. It has been so from the start (many complained about it).
Though I had recently this problem, that it rebuilt ALL my packages. But since then it's working as expected.

EDIT: And yes, its maintainer seems to be still working on it, last comment was 21 day ago. Though he doesn't quickly answer to problems not related to Synth...


----------



## aht0 (Sep 9, 2021)

Yeah, I remembered path name to conf file wrong, I was typing that post from phone at the time.
I installed synth from the binary package and for some reason "13" didn't use it's automatically created configuration file until I manually intervened. Then ran into issue where synth tried to rebuild packages over and over, despite them already having been built before without any changes in port options.

Ended up deleting all packages from system, deinstalling synth and installing everything from zero using only binary packages.


----------



## Mayhem30 (Mar 23, 2022)

When upgrading packages, Synth does not remove the old symbolic links. I see someone has already reported this issue on Github in January, and it was never replied too.

Until we get a fix, you use the following to remove all orphaned symlinks in the the /var/synth/live_packages/All folder.

`find -L /var/synth/live_packages/All -type l -print0 | xargs -0 rm`


----------



## free-and-bsd (Mar 30, 2022)

Mayhem30 said:


> When upgrading packages, Synth does not remove the old symbolic links. I see someone has already reported this issue on Github in January, and it was never replied too.


That leaves one with a very insecure feeling... so I finally stopped using Synth.


----------

