# Synth v1.10 has been released (fast repo generation)



## marino (Mar 2, 2016)

I had planned release 1.1 even before the 1.0 release to address the noticibly long repository generation times, especially on older hardware with mechanical disks.  I think the latest version should make people happy but definitely feedback is welcome.


```
ports-mgmt/synth: Upgrade version 1.03 => 1.10

This release addresses unacceptably long repository rebuild times for the
worst cases (FreeBSD [1], slow CPU, slow mechanical disk).  Until now,
rebuilding the repository required a full tree scan (nearly 26k ports).
While I only saw around 4 minutes on a 4-year DragonFly machine equipped
with a SSD, others reported times exceeding 20 minutes.

This new method scans existing packages twice -- first to eliminate those
packages where the port was removed and also those with a mismatching
version (parallel).  This sets up a second pass to serially and
recursively scan the ports of the remaining packages.  That leads to the
final validation (same as previously done) and the actual repository
generation.  Now the repository generation time is much shorter, but
corresponds to the number of build packages in the packages directory.

The long repository generation times were identified prior to the 1.0
release, but I targetted 1.1 for the enhancement.
```

I forgot to put it in the commit message, but [1] was going to refer to the horrible USES=compiler:features implementation that makes freebsd slow at scanning the ports tree.


----------



## fernandel (Mar 2, 2016)

On my FreeBSD 10.2-RELEASE (amd64), 8GM of RAM is not much faster as before.
Thank you.


----------



## fernandel (Mar 3, 2016)

I running synth-upgrade system now about 20 minutes and building is going very slow. It is still on package for print/texlive-texmf (16 minutes). Before I had in 20 minutes at least 10 port or more done.
Now is done: 17 minutes.
And I didn't change anything in the settings.


----------



## marino (Mar 3, 2016)

irrelevant. 

texlive-texmf is a huge port that takes a long time to build
nothing changed with building
it's a coincidence at best.


----------



## fernandel (Mar 3, 2016)

I understand... I was a little in panic.
Thank you it works great.


----------



## marino (Mar 3, 2016)

I am wondering if you misunderstand what the enhancement is.
To be clear, it makes the command `synth rebuild-repository` generate the repository from a set of already-built packages in seconds rather than many minutes.

You seem to be talking about time to build individual packages, hence I'm wondering if you are thinking it's supposed to build packages faster.  That's not what the change was about.


----------



## marino (Mar 3, 2016)

```
ports-mgmt/synth: Upgrade version 1.10 => 1.11

This fixes a regression in building ports that have dependences that
install kernel modules.  When DTrace support was added by providing a
read-only mount of /boot to the builder, the kernel modules could no
longer be installed at /boot/modules by pkg(8).

Previously, although successful, module installs would have caused a file
system violation on test mode checks.  Since /boot is now excluded from
checks (since DTrace support), leftovers in /boot/modules will not be
detected in test mode.  The fix is too elaborate and FreeBSD-specific
to worry about (plus there's the philosophy question about why the ports
framework is even allowed to modify the base but that's out of scope).
```


----------



## xtaz (Mar 3, 2016)

I've just timed how long it took to rebuild-repository on my system. It took 31 minutes before. With this version it takes 1 minute 31 seconds. So I'm a very happy bunny  Thanks!


----------



## xtaz (Mar 3, 2016)

Something screwy is going on actually. I just tried to prepare-system after changing a port option in openssl. It rebuilt 67 packages, rebuilt the repository, and then all the ports it rebuilt failed a dependancy check and it deleted all the existing packages from the repo.


```
The task is complete.  Final tally:
Initial queue size: 67
  packages built: 67
  ignored: 0
  skipped: 0
  failed: 0

Duration: 00:50:44
The build logs can be found at: /var/log/synth
Stand by, prescanning existing packages.
Stand by, recursively scanning 212 ports serially.
Scanning existing packages.
py27-setuptools27-20.0.txz failed dependency check.
py27-pytz-2015.7,1.txz failed dependency check.
py27-Babel-2.2.0_1.txz failed dependency check.
... and so on
```

I tried it again, and same thing, it built 67, then deleted them again. From running ls in the repo directory I can see that all the built packages exist right up until the final "Scanning existing packages." part, then they vanish.


----------



## marino (Mar 3, 2016)

it smells like a port error somewhere.
However, it's usually the first port on the list that causes the cascade, so that would mean py27-setuptools27 is the culprit. (or the first victim of your change), but that port will wipe out a lot of others.

What *exactly* did you change?


----------



## xtaz (Mar 3, 2016)

I changed the MAN3 option to false in the openssl port. But even changing it back to true does exactly the same thing so I don't tihnk it's related. My port tree was out of date though as portsnap appears to have not had any updates all day, I guess there was some issue. It appears to be updated again now though so I've just managed to upgrade it to synth 1.11. I'm just giving it another run with the latest port tree, see what happens.


----------



## marino (Mar 3, 2016)

seems that lang/python27 is the only dependency of py-setuptools27.

You might try dumping the build list, using "just-build" option, and then "status" so it doesn't actually detect the packages next time.  (and set WHYFAIL=yes in the environment before running `status` command)


----------



## xtaz (Mar 3, 2016)

Heh OK. Either updating to synth 1.11 or updating the ports tree has fixed it. It's not deleted anything this time. Portsnap updated between Thu Mar  3 06:45:00 GMT 2016 and Thu Mar  3 22:08:51 GMT 2016. So might have been something between that time. Oh well working now


----------



## protocelt (Mar 4, 2016)

Just adding this update is working well for me and is quite fast across the board again. No problems yet to speak of.


----------



## yukiteruamano (Mar 6, 2016)

marino@ said:


> irrelevant.
> 
> texlive-texmf is a huge port that takes a long time to build
> nothing changed with building
> it's a coincidence at best.



Hi marino@

I don't think so...I have four hours in "checksuming" the texlive-texmf ports, and nothing. Synth stuck in this task, right now I'm reboot my computer, delete all distfiles and rebuild again this port, more information incoming. 

My CPU is a Pentium G645 and 4 Gb RAM, two hard disk 320 GB in stripe zpool.


----------



## marino (Mar 6, 2016)

how is it relevant?
The release didn't change building at all, and fernandal was talking about building issues (and latter retracted it).

You realize that one of the texlive-texmf tarballs is close to 2Gb is size, right?  I'd say that would give a pentium some exercise to checksum.


----------



## marino (Mar 6, 2016)

(note that it also takes a long time to fetch a 2Gb file, and it's possible the checksum phase is including the fetch)


----------



## yukiteruamano (Mar 6, 2016)

Fetch file isn't a issue here, but checksumming is other history, in this Pentium (sorry for my "old" hardware) I checksumming files for 4 GB and more using SHA1, without issues and don't take long time.


----------



## kpa (Mar 6, 2016)

Run the fetch, checksum, extract and patch phases manually on the port and measure the time taken by them with time(1). This takes the package builder out of the picture. You will be surprised by the results.


----------



## marino (Mar 6, 2016)

okay, then given that this machine is the only known machine that fails to build texlive-texmf, what is it you would like me to do?
Nobody else is reporting this. Did you check the build logs?


----------



## marino (Mar 6, 2016)

also, I would look at your memory and swap situation.  Maybe you saturated swap?  you aren't using tmpfs with low memory are you?


----------



## yukiteruamano (Mar 6, 2016)

I have already said, I am restarting the reconstruction of the package, I deleted everything again, distfiles, waiting is a failure by some strange and random reason, I'll leave it running long enough until it is built, and then'll post the result. Then do the same using the time option, obviously erasing the distfile and see what happened.

On the log, I have not seen, nor package actually came to build the process stopped about 5 hours of checksumming, the rest, synth has worked perfectly.

Perhaps because of what you have told me marino@, the checksummig maybe that package is a good exercise for my Pentium,


----------



## marino (Mar 6, 2016)

okay, well, you didn't address memory (starting with how much the machine has and the tmpfs settings) or what the "swap" measurement was.  I'd look at memory in this case.


----------



## fernandel (Mar 6, 2016)

It happened to me today:
security/gcr: package Lines: 2043
, time 00:01.047
www/webkit2-gtk3 configure, lines 1076, time 00:00:49

```
swapinfo
Device  1K-blocks  Used  Avail Capacity
/dev/ada0p6  3773548  312736  3460812  8%
```

top shows:

```
PID USERNAME  THR PRI NICE  SIZE  RES STATE  C  TIME  WCPU COMMAND
15021 root  1  84  0  155M  135M CPU1  1  0:05  41.36% c++
15014 root  1  84  0  167M  145M CPU7  7  0:05  39.60% c++
15024 root  1  82  0  156M  132M CPU3  3  0:04  36.28% c++
15030 root  1  80  0  123M  103M CPU4  4  0:03  26.95% c++
15026 root  1  52  0 16820K  3432K wait  2  0:00  2.39% ccache
15029 root  1  52  0 36272K 14140K wait  3  0:00  2.39% c++
10662 fernandel  59  27  0  934M  416M uwait  5  1:45  1.76% firefox
15023 root  1  52  0 36272K 14140K wait  6  0:00  1.76% c++
15010 root  1  52  0 16820K  3228K wait  0  0:00  1.66% ccache
15018 root  1  52  0 16820K  2972K wait  7  0:00  1.66% ccache
15019 root  1  52  0 36272K 14140K wait  1  0:00  1.66% c++
15013 root  1  52  0 36272K 14140K wait  5  0:00  1.46% c++
 1896 root  1  52  0  338M  330M piperd  6  0:58  1.37% gmake
15006 root  1  52  0 16820K  3332K wait  3  0:00  1.37% ccache
52875 fernandel  8  20  0  657M  146M kqread  0  14:21  0.98% gnome-she
 1274 fernandel  2  20  0  210M 37820K uwait  0  32:02  0.00% Xorg
```
It looks building is going (gcr is build and is in/var/synth/live_packages/All). Maybe is something wrong with ncurses?
Thank you.


----------



## marino (Mar 6, 2016)

"It happened to me today:"
What happened today?  I don't understand your description of what the problem is.

Are you saying something didn't build?


----------



## fernandel (Mar 6, 2016)

I have long time on the screen what I wrote in the previous post but when I check Synth directory the packages were build but what I saw on the screen was all the time the same - didn't change. Now again working correct. Is it problem with my system?
There are nothing in the logs.


----------



## marino (Mar 6, 2016)

The first time you were alarmed that something took a long time to build, longer than you expected, but it built.

I still don't know what you are concerned about. It seems the issue is that sometimes it takes a lot longer to build than you expect?  If packages are building and things are correct, what concerns you?  it is not clear.


----------



## fernandel (Mar 6, 2016)

marino@ said:


> The first time you were alarmed that something took a long time to build, longer than you expected, but it built.
> 
> I still don't know what you are concerned about. It seems the issue is that sometimes it takes a lot longer to build than you expect?  If packages are building and things are correct, what concerns you?  it is not clear.



I concern because the package security/gcr and Webkit were build and in the directory but Synth did show that are still building and it was more than one hour and many other packages were build in this time. After one hour Synth screen changed and it start shows again what is building.


----------



## marino (Mar 6, 2016)

it sounds like are running synth in a resizable virtual terminal and then you resized the window.  Or something else affected ncurses from working, dunno.


----------



## fernandel (Mar 6, 2016)

marino@ said:


> it sounds like are running synth in a resizable virtual terminal and then you resized the window.  Or something else affected ncurses from working, dunno.


Yes, I thought that is something with ncurses. If I am running GNOME than I am checking what Synth doing (ctrl-alt-F?) and going back to GNOME.
But important is that is working .
Thank you.


----------



## marino (Mar 6, 2016)

I don't know gnome.  Are you saying you are switching between X and syscons with control-alt-F?

If you primarily use a desktop, I think I'd recommend opening xterm and running synth in a window there (perhaps within _screen_ or _tmux_) and just stick that window on one of the desktops so you can check it whenever you want.  Switching modes like that is probably ncurses doesn't like.


----------



## fernandel (Mar 6, 2016)

marino@ said:


> I don't know gnome.  Are you saying you are switching between X and syscons with control-alt-F?
> 
> If you primarily use a desktop, I think I'd recommend opening xterm and running synth in a window there (perhaps within _screen_ or _tmux_) and just stick that window on one of the desktops so you can check it whenever you want.  Switching modes like that is probably ncurses doesn't like.


Yes, I do because sometimes I start GNOME and more often Fluxbox.  I will do. Thank you and it was very interesting and educational for me, listening your interview.


----------



## tankist02 (Mar 7, 2016)

fernandel Perhaps you could disable ncurses display (option M)? I use text display and while it may not be as pretty as ncurses it works great.


----------



## marino (Mar 7, 2016)

using the text display would certainly fix the issue.  If he wants to continue switching between desktop and syscons and use ncurses, I'd recommend that he run synth inside the *screen* program and just attach as necessary.  I know some people use *tmux* but I've heard ithat is more senstive to window resizing than screen is.  I use *screen *and it works for me.


----------



## tankist02 (Mar 7, 2016)

Some people using synth noticed that libc++ gets rebuilt every time because it fails dependency check. I created a PR 207756 to help address the issue. Here is the latest reply from Dimitry who is maintainer of libc++:


```
Why on earth is your ports tree trying to use the devel/libc++ port, when you are on FreeBSD 10.3-PRELEASE?  This release has libc++ in the base system, and the port should use that instead...
```

Maybe Marino and Dmitry can contact each other directly to resolve the issue?


----------



## kpa (Mar 7, 2016)

tankist02 said:


> Some people using synth noticed that libc++ gets rebuilt every time because it fails dependency check. I created a PR 207756 to help address the issue. Here is the latest reply from Dimitry who is maintainer of libc++:
> 
> 
> ```
> ...



Some ports depend on newer libc++ than is available on the base system, same thing as with clang/llvm where the base system ones are not enough.


----------



## marino (Mar 7, 2016)

tankist02 said:


> Some people using synth noticed that libc++ gets rebuilt every time because it fails dependency check. I created a PR 207756 to help address the issue. Here is the latest reply from Dimitry who is maintainer of libc++:
> 
> Maybe Marino and Dmitry can contact each other directly to resolve the issue?



we've talked about it a couple of times already.
We left it at that I would look at it and fix it, but I never got around to it.
Dmitry doesn't fully understand the situation; he's thinking about it from a different POV and he's deferring to bapt who wasn't all that clear (for dim)


----------

