# Parallel Port Building Script



## cakersq (Jun 3, 2011)

I was annoyed when building ports used only one of my servers processor cores, and annoyed at having to install Perl/Ruby/Python just to make the INDEX-#.

After an afternoon of coding, I wrote a handy SH script to install ports in parallel, accounting for dependencies, and not requiring installing another language.

It's a little rough, but I got it working to install X and Gnome.

What do you think (See Attached)?


----------



## wblock@ (Jun 3, 2011)

How old is the ports tree on that machine?  The ports system has supported multiple jobs for quite a while now.  See
`% less -p JOBS /usr/ports/Mk/bsd.port.mk`


----------



## cakersq (Jun 3, 2011)

I'm aware of the JOBS support in Ports, but in my experience, most ports do not have that enabled, and forcing it usually doesn't work, especially when dependencies are required.

This way parallel building is supported for EVERY port.


----------



## wblock@ (Jun 3, 2011)

Bringing this up on the ports mailing list might be good.  Maybe the advantages of both can be combined.


----------



## cakersq (Jun 3, 2011)

Updated to version 0.2
 - Fixed falsely removing dependencies from to-be-installed list
 - Fixed default number of jobs to CPUs+1
 - Fixed port count display
 - Output pkg_add to /dev/null (Quiet)


----------



## cakersq (Jun 3, 2011)

Updated to version 0.3
 - Fixed checking for already installed packages
 - Improved usage help message
 - Improved output messages (removed some, corrected others)

That's the last update for a while, next will focus on speeding up the process further:
 - Speed up dependency gathering phase
 - Build larger packages first
 - Background distfile download and extraction


----------



## Alt (Jun 3, 2011)

Can you make a port of your script? It's a bit ugly this way - to share scripts in a text file, I think. Anyway I'll try it on next build. Can it work with ccache/portmaster?


----------



## cakersq (Jun 3, 2011)

@Alt



> Can you make a port your script?



The concept of installing a port to install ports never sat well with me.  There are also 1000's of port management ports out there.



> Can it work with ccache/portmaster



My goal was to create this using a minimum amount of additional software, so that it can be used on any system.  What I will probably try to do is to integrate it into the Ports system (bsd.pports.mk or something), that way you can go into a port, and specify *make parallel-package* or *make parallel-install clean* and have it go.

That being said, this uses the standard *make package clean* method of installing ports, so if you install ccache, and add its configuration to the /etc/make.conf configuration file, it should still work.


----------



## cakersq (Jun 4, 2011)

Upgraded to version 0.4
 - Continuously update the ports dependencies list, to find ports that can be built
    - Greatly reduces (to almost none) the number of ports that are installed serially.


----------



## cakersq (Jun 5, 2011)

Version 0.4.1 - Bug fix - dependencies were being installed out of order ... sorry.


----------



## Seeker (Jun 5, 2011)

Can you clarify ..., in a case, when more then 1 origin is specified, i.e;

```
pports-0.4.1.sh ftp/wget lang/php5
```
Will it firstly build ftp/wget and simultaneously build all its dependencies, while lang/php5 is waiting OR will it be simultaneously building both origins and simultaneously all their dependencies?

Thanks!

Ok, I've been impatient, waiting longer then one minute for a reply, so I've taken a peak in your code and I say: it builds ftp/wget first and simultaneously builds all its dependencies, while lang/php5 is waiting.


----------



## cakersq (Jun 5, 2011)

@Seeker

Excellent question!  The answer is:


> ...it be simultaneously building both origins and simultaneously all theirs dependencies



The code gathers the dependencies for all of the origins specified serially (one at a time), but will then build all dependencies together.  So it is possible the second origin completes before the first origin.

The algorithm goes something like this:
- Run *make config-conditional* for all specified origins, unless *-b* is specified
- Gather list of dependencies for all specified origins (one at a time), store in "index"
    - Gather list of dependencies for all dependencies, store in "index"
    - Repeat until all possible dependencies are indexed
- Find all ports in the above "index" that are leaves (have no dependencies of their own), store in "install"
    - Build all leaves listed in the "install" file, up to *-j* amount at a time.
    - Rebuild "index" as needed, removing listed dependencies for an origin if the dependency was installed (will reveal more "leaves" that have no further dependencies)
    - Repeat until all ports are installed.

Every now and then, there will be a time where the system is just building one port at a time, this is normal.  It happens when a major port (such as devel/gettext or devel/libiconv) is being built, and everything requires it.  I've made an effort to reduce the amount of time only one port is being built, but it will still happen.

The code's not easy to read (for that I apologize).  This is more of a proof-of-concept than anything else.  As I mentioned, I'll probably try to get this code added to the Ports system properly, in the /usr/ports/Mk/ files.  That way a user can specify [cmd=]make -C /usr/ports/lang/php52 parallel-package[/cmd] and it's done, much faster than the normal serial building method.


----------



## Seeker (Jun 5, 2011)

You have a problem! Installing ports into DESTDIR doesn't work!


```
===>  Chrooted make in /tmp/md.sh.cy9DSHlJ succeeded
===>  Cleaning up...
Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory
make: chdir ===>: No such file or directory
.make: chdir Creating: No such file or directory
make: chdir Creating: No such file or directory
.make: chdir some: No such file or directory
make: chdir some: No such file or directory
.make: chdir important: No such file or directory
make: chdir important: No such file or directory
10make: chdir subdirectories: No such file or directory
make: chdir subdirectories: No such file or directory
.make: chdir Starting: No such file or directory
make: chdir Starting: No such file or directory
.make: chdir chrooted: No such file or directory
make: chdir chrooted: No such file or directory
.make: chdir make: No such file or directory
make: chdir make: No such file or directory
.make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory
make: chdir /tmp/md.sh.cy9DSHlJ...: No such file or directory
.make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory
make: chdir /tmp/mountpoint.vPMIYS/devel/gmake: No such file or directory
.make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory
make: chdir /tmp/mountpoint.vPMIYS/dns/libidn: No such file or directory
.make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory
make: chdir /tmp/mountpoint.vPMIYS/lang/perl5.12: No such file or directory
.make: chdir Chrooted: No such file or directory
make: chdir Chrooted: No such file or directory
.make: chdir succeeded: No such file or directory
make: chdir succeeded: No such file or directory
20make: chdir Cleaning: No such file or directory
make: chdir Cleaning: No such file or directory
.make: chdir up...: No such file or directory
make: chdir up...: No such file or directory

Gathering Dependencies for lang/php5..make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/devel/autoconf: No such file or directory
.make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/devel/pcre: No such file or directory
.make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/devel/pkg-config: No such file or directory
30make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory
make: chdir /tmp/mountpoint.cX1UL3/textproc/libxml2: No such file or directory

Gathering Dependencies for sysutils/fusefs-ntfs..make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/converters/libiconv: No such file or directory
.make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/devel/libtool: No such file or directory
.make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/devel/libublio: No such file or directory
.make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-libs: No such file or directory
40make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory
make: chdir /tmp/mountpoint.PO8Tw2/sysutils/fusefs-kmod: No such file or directory

[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
[: /var/db/pkg/===>: unexpected operator
grep: /tmp/pports-2119/index3: No such file or directory
grep: /tmp/pports-2119/index3: No such file or directory
rm: /tmp/pports-2119/index3: No such file or directory
Done!
```

You have to fix this!


----------



## cakersq (Jun 5, 2011)

Well, Seeker, I have no idea what you did to cause that!   The script requires a full base system to run, especially /usr/ports, /var/db/pkgs, etc.

If you want to build the ports in a full chroot environment, including ports tree, you can copy the script into the chroot, and run
[cmd=]chroot /mnt /root/pports.sh lang/php52[/cmd]

If you just want to install the ports to the mountpoint, I would run the script and build the ports on the main system, then use the packages created in /usr/ports/packages (created via my script.)
[cmd=]pkg_add -P /mnt /usr/ports/packages/All/php-5.2.tbz[/cmd]

Anyway, good luck!


----------



## Seeker (Jun 6, 2011)

cakersq said:
			
		

> Well, Seeker, I have no idea what you did to cause that! ...


My DESTDIR, has a fresh FreeBSD installation.

Your script should behave in a same way as ports d, that is, if I:

```
cd /usr/ports/ftp/wget
make install clean
```
it will *chroot* into DESTDIR and successfully install the port, *EVEN* if DESTDIR doesn't have a /usr/ports tree, because it mounts into the chrooted DESTDIR, via nullfs, what it needs.

As you intend to integrate this, into the /usr/ports/Mk/ files, you have to fix this, as your script has to handle this, in the same way as the ports system does.

So good luck to you, fixing your script.


----------



## cakersq (Jun 7, 2011)

Just for you Seeker:

Version 0.4.3 2011/06/06
 - Works with DESTDIR for ports (specified in /etc/make.conf)
 - Added DestDir option to script
 - pkg_add honors DestDir variable
 - Only checks for already installed ports once
 - Fixed error handling when a specified origin doesn't exist


----------



## Seeker (Jun 7, 2011)

DESTDIR now works!
However, your script will hang endlessly, because of a daemonized dialogs. That is:

```
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'

pports.sh $ports
```

Dialogs will be thrown at the start, for each origin, before builds occur. However, once any dependency is started to be built, its dialog will hang endlessly, as it won't be thrown to user.


```
last pid: 41730;  load averages:  2.07,  2.19,  2.17                                                    up 0+03:09:45  11:10:32
83 processes:  3 running, 80 sleeping
CPU: 52.1% user,  0.0% nice, 46.4% system,  1.5% interrupt,  0.0% idle
Mem: 58M Active, 309M Inact, 114M Wired, 2508K Cache, 111M Buf, 513M Free
Swap: 2014M Total, 2014M Free

  PID USERNAME        THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
18232 root              1 107    0  3668K  1576K RUN     53:27 49.76% dialog
 5822 root              1 107    0  3668K  1576K RUN     55:53 48.49% dialog
```

I had to kill the dialog processes to continue builds..


```
18232  ??  R     54:22.88 /usr/bin/dialog --checklist Options for m4 1.4.16,1 21 70 15 LIBSIGSEGV Use libsigsegv for better
```

You have to fix this.

I would recomend, hammering a human with ALL dialogs, before building starts, via:

```
# make config-recursive
```
So human can take a walk, doing its "human stuff".


----------



## Seeker (Jun 7, 2011)

Additionally ...
I need to run this code twice:

```
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'
# It only BUILDS, so DESTDIR ends up empty
pports.sh $ports
```


```
ports='ftp/wget lang/php5 sysutils/fusefs-ntfs'
# This installs ...
pports.sh $ports
```

Also *devfs* remained mounted in chroot


----------



## cakersq (Jun 7, 2011)

This seems to be a problem with Ports system not honoring the BATCH flag when DESTDIR is specified.  I'll have to go digging and figure out why.


----------



## cakersq (Jun 7, 2011)

Couldn't fix the BATCH issue when DESTDIR is specified so I wrote my own DESTDIR algorithm for this script.

Check it out, tell me what you think.


----------



## cakersq (Jun 8, 2011)

Fixed script, was not checking for ports currently being installed correctly...


----------



## cakersq (Jun 8, 2011)

I fixed a bug before you even found it!

Version 0.5.2
 - Fixed that port dependencies are inaccurate when DESTDIR is used and *make config* changes options from the defaults.


----------



## graudeejs (Jun 8, 2011)

cakersq said:
			
		

> I fixed a bug before you even found it!
> 
> Version 0.5.2
> - Fixed that port dependencies are inaccurate when DESTDIR is used and "make config" changes options from the defaults.



If you want I can provide Mercurial repository for your project at http://hg.bsdroot.lv also bugzilla via bugs.bsdroot.lv
Or you can use github


----------



## Seeker (Jun 8, 2011)

```
===>  Directory /tmp/md.sh.zBpkvMWn does not exist
===>  Please set DESTDIR to a valid jail environment.
*** Error code 1

Stop in /usr/ports/ftp/wget.
===>  Directory /tmp/md.sh.zBpkvMWn does not exist
===>  Please set DESTDIR to a valid jail environment.
*** Error code 1

Stop in /usr/ports/lang/php5.
Gathering Dependencies for ftp/wget..make: chdir ===>: No such file or directory
make: chdir ===>: No such file or directory
.make: chdir Directory: No such file or directory
make: chdir Directory: No such file or directory
.make: don't know how to make package-name. Stop
make: chdir /tmp/md.sh.zBpkvMWn: No such file or directory
.make: chdir does: No such file or directory
make: chdir does: No such file or directory
.make: chdir not: No such file or directory
make: chdir not: No such file or directory
.make: chdir exist: No such file or directory
make: chdir exist: No such file or directory
.make: chdir Please: No such file or directory
make: chdir Please: No such file or directory
.make: chdir set: No such file or directory
make: chdir set: No such file or directory
10make: chdir DESTDIR: No such file or directory
make: chdir DESTDIR: No such file or directory
.make: chdir valid: No such file or directory
make: chdir valid: No such file or directory
.make: chdir jail: No such file or directory
make: chdir jail: No such file or directory
.make: chdir environment.: No such file or directory
make: chdir environment.: No such file or directory
.make: chdir boot.txz: Not a directory
make: chdir boot.txz: No such file or directory
.make: chdir Error: No such file or directory
make: chdir Error: No such file or directory
.make: chdir code: No such file or directory
make: chdir code: No such file or directory
.make: chdir 1: No such file or directory
make: chdir 1: No such file or directory
.make: chdir Stop: No such file or directory
make: chdir Stop: No such file or directory
.make: chdir in: No such file or directory
make: chdir in: No such file or directory
20make: chdir /usr/ports/ftp/wget.: No such file or directory
make: chdir /usr/ports/ftp/wget.: No such file or directory

Gathering Dependencies for lang/php5..make: chdir /usr/ports/lang/php5.: No such file or directory
make: chdir /usr/ports/lang/php5.: No such file or directory

[1/2] Installing ftp/wget
[2/2] Installing lang/php5
Done!
```
Last two ports are installed, because there was a previous run, from yesterday. DESTDIR is valid, as was used by other parts of MY script, before and after calling your script.


----------



## cakersq (Jun 9, 2011)

How are you setting DESTDIR?

Edit:  Never mind, I figured it out.  You're setting it in the environment, via *export* or *setenv*. ... Fix will be provided shortly.


----------



## cakersq (Jun 9, 2011)

Version 0.5.3
- Fix for when DESTDIR is set in Environment instead of /etc/make.conf


The cause was when DESTDIR is set in the environment, when I run the *chroot* command, DESTDIR was set in the *chroot* environment as well, causing the port install to attempt to be in another level of *chroot*.

For example if
`# setenv DESTDIR /jail/php ; ./pports.sh lang/php5`
is used, the port attempted to be installed in /jail/php/jail/php


----------



## AngelescuO (Aug 10, 2011)

Hello,

I have discovered an bug and I have sent you an e-mail with title pports.sh.

Thank you.


----------



## cakersq (Aug 12, 2011)

In honor of AngelescuO, I have added error checking to the script.  Now, if a build of a package fails, the script will finishing building anything in progress, then terminate, alerting the user to which port(s) had errors during building.


----------



## AngelescuO (Aug 12, 2011)

Hi 

I have sent you another feedback  and a patch via e-mail.
Now error checking works excellent.

Thank you


----------



## cakersq (Aug 12, 2011)

> As I told you that script it's amazing.
> Please take a look at the attached files.
> I have discoveed another bug .
> Running pports.sh with -b or without it sometimes  if the ports options were not configured pports stall because it waits for answer for dialog ( make configure ) .
> ...


@AngelescuO:

The script is working as designed, and the code patch you provided will actually cause the script to hang.

What happens is that the script will prompt the installer for the configuration of the specified ports, if one does not exist (make -C $port config-conditional)  The only difference that the Batch -b flag does is that this step is skipped.

When the packages are being built, they are all done BATCH style, with make -C $port -D BATCH package, so will not prompt the user for the configuration.

With your code, the port will prompt everyone for the configurations, and if the -b flag is not specified, the script will hang on the configuration pages.

Where were you seeing my script get hung on configurations?

Thanks for your interest.


----------



## AngelescuO (Aug 12, 2011)

Hi,

Sometimes it hangs at configurations.
I will retest without patch on a fresh install.

Greetings


----------



## AngelescuO (Aug 12, 2011)

Sorry,

I have sent you an bad file.
You have right with to not use  make -C portsname without BATCH flag.
I don't agree with the way of configuring packages.

I expect to do an make config-recursive when not using -b flag.
When building using make config-recursive main port's option are set and also for depends.
Using BATCH flag all ports will be compiled with chosen options. 

I expect to use make config when using -b flag.
When building using only make config will create only main port's options and since the packages are built with -BATCH flag the depends are built with default options.

In both cases previous config options are overwritten.

If we'll use config-conditional main port options file is created first time and it will be reused if we need to rebuild.

Greetings,
PS: The way of configure packages it's a taste matter.


----------



## cakersq (Aug 12, 2011)

Updated Script to version 0.5.6, to fix a bug with error checking.

Ports tended to exit with an error status, even when installed correctly, so I am now checking whether it is installed.

Also, for *AngelescuO*, I am now running config instead of config-conditional.

I suggest adding any package you want to configure to the command line, otherwise it will use the defaults as you noticed.

Enjoy!


----------



## AngelescuO (Aug 13, 2011)

Hello,
You have an pm and an e-mail with same content of course related to pports.sh.
Thank you.


----------



## cakersq (Aug 13, 2011)

Minor bug fixes in Error checking

This is probably the last version of this script, I want to start integrating it into Ports tree soon.


----------



## AngelescuO (Aug 20, 2011)

Hi,

I have done tests and it works great.
An idea/suggestion.
Can you add conflict resolution in it ?
Something like searching in all depends ,finding conflicts and printing them to screen before starting an build.
As an example ntfsprogs conflict with fusefs-ntfs. If I choose both the build will fail.
If the conflict will be printed to screen before starting the build it will be excellent.

Thank you.


----------



## peerst (Aug 27, 2011)

*0.5.7 doesnt work for me*



			
				cakersq said:
			
		

> Minor bug fixes in Error checking
> 
> This is probably the last version of this script, I want to start integrating it into Ports tree soon.



Tried out version 0.5.7 on FreeBSD 8.2 amd64 and just got this:


```
Gathering dependencies .... etc etc
....
[  1/327] Building devel/libtool
[  2/327] Building devel/m4

Error building port(s):
devel/m4 Error Level 1
```
When I try to make and install devel/m4 manually it just works


----------



## smj (Dec 12, 2011)

Thanks cakersq - this script has been a big help with getting a new 8 core server built faster. Nice to be able to give it VirtualBox, VNC, and X11 components, deal with a few screens, and then just watch it keep the load as high as possible.

Can't wait to see where this and the parallel build support coming into ports will go over the next few months.


----------

