# using "make -jx"  with ports



## ph0enix (Mar 21, 2009)

I have a quad-core system so I've tried using "make -j5 install distclean" with a few ports but most of them fail to compile that way.  They compile fine with just "make install distclean".  Is there a reason for this?  

Thanks! 

J.


----------



## DutchDaemon (Mar 21, 2009)

You could try make -j4 && make install clean. I think the make install part panics when some jobs are finished before others. So use the extra cpu power for compiling only, and leave the installing part to a 'serial process'. I'm not entirely sure how this pans out when the ports starts pulling in dependencies, so test.


----------



## tingo (Mar 21, 2009)

ph0enix said:
			
		

> I have a quad-core system so I've tried using "make -j5 install distclean" with a few ports but most of them fail to compile that way.  They compile fine with just "make install distclean".  Is there a reason for this?


Yes - the reason is: parallell make (make -j) is not supported for ports.

Think about it - there are well over 19000 (almost 20000 now) ports in the ports collection, and they use many different methods for building and installing. Even if we (FreeBSD users and developers) hda all the manpower we wanted, getting the ports to support parallel building and install would be almost impossible.

I suggest you do something more productive or fun the next time you compile a port; play a game, read a book or surf the net.


----------



## ph0enix (Mar 22, 2009)

tingo said:
			
		

> Yes - the reason is: parallell make (make -j) is not supported for ports.
> 
> Think about it - there are well over 19000 (almost 20000 now) ports in the ports collection, and they use many different methods for building and installing. Even if we (FreeBSD users and developers) hda all the manpower we wanted, getting the ports to support parallel building and install would be almost impossible.
> 
> I suggest you do something more productive or fun the next time you compile a port; play a game, read a book or surf the net.



Thanks for the great suggestions but I already played 3 games, read 7 books and surfed every single page on the internet.  The code is still compiling


----------



## ale (Mar 22, 2009)

http://www.freebsd.org/cgi/getmsg.cgi?fetch=59555+0+current/cvs-ports


----------



## Brandybuck (Mar 22, 2009)

You can set MAKE_ARGS to the arguments you want to use. The port  makefile itself won't use them, but the actual make to build will. The problems arise when you try to use -jn with the port itself, or during install. So here is what I added to the bottom of my /etc/make.conf:


```
.if !(make(*install) || make(package))
MAKE_ARGS+=-j4
.endif
```

So far this was worked great for me. Just remember though that some apps will still bomb with parallel builds.


----------



## ph0enix (Mar 22, 2009)

Should I use -j4 or -j5 for a quad-core system?  I always thought that the formula was:
number of CPUs +1

Is this not right?


----------



## vermaden (Mar 22, 2009)

ph0enix said:
			
		

> Should I use -j4 or -j5 for a quad-core system?  I always thought that the formula was:
> number of CPUs +1
> 
> Is this not right?



Depends, you need to measure it, I would set it to 2 x CORES count, I remember I measured once time buildworld with -j and I got best results (fastest) with 4-5 jobs at dual core system, so I will set 8-9 at quad core.


----------



## ph0enix (Mar 22, 2009)

vermaden said:
			
		

> Depends, you need to measure it, I would set it to 2 x CORES count, I remember I measured once time buildworld with -j and I got best results (fastest) with 4-5 jobs at dual core system, so I will set 8-9 at quad core.



Can I test with buildkernel instead of buildworld?  It'll be "a little" faster.


----------



## vermaden (Mar 22, 2009)

ph0enix said:
			
		

> Can I test with buildkernel instead of buildworld?  It'll be "a little" faster.



Yes, of course.


----------



## oliverh (Mar 22, 2009)

@ph0enix, you could even use ccache and dramatically reduce build time.


----------



## danger@ (Mar 22, 2009)

yup indeed. the issue is that only a few ports support it currently


----------



## ph0enix (Mar 22, 2009)

vermaden said:
			
		

> Yes, of course.



I'd say the test results are inconclusive with buildkernel.  I did two runs with each setting.  It looks like -j3 is out but that's about it.  I'll try buildworld:

-j8:
real    5m45.118s
user    10m8.636s
sys     1m18.063s

real    5m29.173s
user    10m8.689s
sys     1m17.048s

-j6:
real    5m24.774s
user    10m8.773s
sys     1m17.537s

real    5m26.855s
user    10m8.329s
sys     1m16.085s

-j5:
real    5m27.501s
user    10m9.188s
sys     1m16.038s

real    5m21.730s
user    10m7.112s
sys     1m16.250s

-j4:
real    5m38.467s
user    10m5.442s
sys     1m14.540s

real    5m20.220s
user    10m5.708s
sys     1m14.321s

-j3:
real    5m45.420s
user    10m0.737s
sys     1m10.541s

real    5m51.748s
user    10m0.168s
sys     1m11.131s

-j1 is about 11 minutes.


----------



## Mel_Flynn (Mar 24, 2009)

*Official MAKE_JOBS hit ports tree*

pav@ committed initial support for parallel builds. See this PR for details.

At present it works as opt-in on a per port basis, meaning the port maintainer has to set MAKE_JOBS_SAFE in the Makefile to enable parallel builds. When set, the number of jobs is set to the number of cores in the machine, specifically the output of sysctl -n kern.smp.cpus.

Usage of MAKE_ARGS is now discouraged. Adventurous users can set FORCE_MAKE_JOBS to enable parallel builds globally or on a per port basis, using Makefile.local or constructs in make.conf like so:

```
.if !empty(.CURDIR:M*/x11/kdebase3)
FORCE_MAKE_JOBS=yes
.endif
```

An opt-out approach can be done like so:

```
FORCE_MAKE_JOBS=yes
# These I know to be unsafe
UNSAFE_PORTS =audio/libsndfile
UNSAFE_PORTS+=audio/nas
UNSAFE_PORTS+=audio/xfce4-mixer
UNSAFE_PORTS+=devel/doxygen
UNSAFE_PORTS+=devel/gperf
UNSAFE_PORTS+=devel/libthai
UNSAFE_PORTS+=devel/nasm
UNSAFE_PORTS+=devel/pth
UNSAFE_PORTS+=devel/dbus-qt4
UNSAFE_PORTS+=editors/vim
UNSAFE_PORTS+=games/falconseye
UNSAFE_PORTS+=graphics/libart_lgpl
UNSAFE_PORTS+=lang/lua50
UNSAFE_PORTS+=lang/ocaml
UNSAFE_PORTS+=misc/e2fsprogs-libuuid
UNSAFE_PORTS+=net-mgmt/net-snmp
UNSAFE_PORTS+=print/ghostscript8
UNSAFE_PORTS+=security/libgpg-error
UNSAFE_PORTS+=security/nss
UNSAFE_PORTS+=textproc/xerces-c2
UNSAFE_PORTS+=www/lynx
UNSAFE_PORTS+=x11-clocks/kdetoys3
# Apply disabling make jobs
.for port in ${UNSAFE_PORTS}
.if !empty(.CURDIR:M*/${port})
DISABLE_MAKE_JOBS=yes
.endif
.endfor
```

Hope this helps.


----------



## Seeker (Feb 1, 2010)

Mel_Flynn said:
			
		

> pav@ committed initial support for parallel builds. See this PR for details.
> 
> At present it works as opt-in on a per port basis, meaning the port maintainer has to set MAKE_JOBS_SAFE in the Makefile to enable parallel builds. When set, the number of jobs is set to the number of cores in the machine, specifically the output of sysctl -n kern.smp.cpus....



To translate this...
It is implemented in a way that _we, the retards_, don't even have to think about it, as if port is capable of parallel build, then it will automatically build with -j option.
And not only that, it will also choose N (-jN) number.., again on it's own, so once again, we don't even have to think about it :stud


----------



## cakersq (Jun 4, 2011)

I realize this thread is a few years old, but if you are still looking around for a way to utilize your quad-core better, I just wrote a script to install the dependencies for a port in parallel.  Each port being built will still use the 1 core, but will have multiple ports building at once, while being considerate of dependencies.

I wrote this in SH and it does not require the /usr/ports/INDEX file, so this will work on the base system without additional languages or ports.

http://forums.freebsd.org/showthread.php?p=136569#post136569

Let me know how it works, and if you are interested in other features.


----------

