# Suggested New Make Targets in Ports



## Ruler2112 (Mar 10, 2011)

I'm in the process of setting up a new box for myself to use as a desktop.  In the process, I've had a few ideas for new *make* targets/options that would significantly help reduce the overall build time and hassle.


1.  *make configdeps* | *make configusedefaults*

To compile x11/kde4 on a new system, there are a whole slew of dependencies that need to be built/installed - not a big deal.  However, whenever one of these dependencies pops up a config screen to set options for that port, the compile stalls until I pick the options and hit OK.  I constantly have to click over to the server on my KVM throughout the day to set options and continue the compile.  Given the number of dependencies and the fact that I let it run overnight, this is a problem; it is now Thursday afternoon, I've been compiling since Friday, and it is still not done.  It's almost a certainty that the computer has spent the vast majority of that time waiting for me to confirm what options to use; the vast majority of the time, I simply hit [TAB][ENTER] to accept the defaults and the computer continues compiling.  (I don't know what the majority of the options even are or what they do, so most of the time I just accept the defaults.)

Why not have a make target of configdeps that would go through each of the build/run dependencies and run *make config* on them, one after another, without compiling them?  This way, I could run *make configdeps* and pick all the options for every port, then just run a simple *make*.  The computer would be busy and stay busy until it was done.

Another option that might be of use is *make configusedefaults*.  This would simply bypass the config screen and use the default options, exactly as if you were to use [TAB][ENTER] on each config screen.  I'm sure people change options here and there so it wouldn't be good for everybody, but it would certainly be applicable for many people.


2.  *make rmdistafter* / *make cleanafter* | *make savespace*

I allocated only 10 gig for my /usr partition.  While more than large enough for my mail server, I did not foresee that I'd run short of space on a kde compile.  When it happens, I simply run *make clean* in the x11/kde4 directory and the system cleans all the work files out of the dependency directories.  I then re-run *make* and it picks up where it left off.  (Thank God.  )  The distfiles are also consuming a significant amount of space.  These ideas are designed to eliminate these from causing problems.

*make rmdistafter* would simply remove the distfile for a given port after the install is complete.

*make cleanafter* would simply run make clean in a given port after the install is complete.

*make savespace* would simply enable both rmdistafter and cleanafter.

While both of these are easy enough to accomplish manually after building a port, the advantage would come when there are a large number of dependencies involved.  After each dependency is installed, the system would run *make clean*, remove the distfile, and continue.  These would probably be better as options rather than make targets, but could be either.


3.  *make concurrent*

This is another item that might be better served as an option.  It's also the only one I can see that might be problematic to implement.

When a port is built, the distfile is fetched, then extracted, patched, and compiled.  Some of the distfiles are large and take a while to download.  Even with my 6 meg internet connection, some take several minutes to finish downloading, and many people have slower Internet than I.  This results in the system waiting for the download to finish, then pegging the CPU, basically waiting for it to finish compiling.  These resources are not tied together and could be used concurrently, but the present structure does not allow for this.

This option would start a separate thread for up to n dependencies, running each concurrently.  It would more effectively utilize the CPU/network resources of the machine because while one port is waiting for the network, another could use the CPU.  However, if a dependency is dependent on another port that is also a dependency of the original, things could get hairy with the timing of things.  I do not believe this to be an insurmountable problem, but it is more complicated that any of the others presented here.

Another option that may work better is to have the main *make* process simply download all the distfiles for the dependencies of the port you're in, one after the other.  After the first is done, a thread is spawned and the compile process for the port is started.  As soon as the thread finishes, control is returned to the main/download process.  If another port's distfile is done, a new thread is immediately started to compile/install that one.  If the distfile isn't done downloading, the system waits until it is and then fires it.  This structure would serve to concurrently utilize both the CPU and network resources so that time waiting for either while the other is idle is kept to a bare minimum.  The timing problem when compiling/installing multiple ports would no longer be a concern with this design because only one compile thread would be active at any given time, even though the next port's distfile is being downloaded at the time.


4.  *make rmolddist*

This one is applicable to any system that's run for any length of time that has had the ports updated.  All the distfiles for the old port versions are stored.  I don't know how the specifics of the ports system work and therefore don't know how involved it would be, but a *make rmolddist* target to simply remove all the old distfiles for the current port, keeping the current one, would be very nice.  Of course one can always *rm -rf /usr/ports/distfiles/** and just let the distfiles re-download as needed, but this would be a much more elegant solution.


I believe that these additions would benefit the software installation process.  Quite frankly, not many people are going to try BSD if they have to wait as long as I have just to build/install KDE.  I know that binary packages are available, but then I'd be in a similar situation as what I'm in with my Slackware box right now; I'm moving to BSD because I don't like relying on other people to generate binary packages for me and don't have the patience to jump through all the hoops needed to make them myself.  (I've done a few and it seems like every piece of software that exists has it's own process that is different from every other piece of software.  Also, most require some specific tweaking to work at all.  Even some of the binary packages don't work correctly...)  After using ports on my mail server, I decided that BSD would be more optimal for my desktop as well.

I don't know if this is the correct place to make these suggestions, but since it's on freebsd.org I'm guessing that developers sometimes look at the forum.  If there's a better place, please let me know.


(Apologies in advance to DD - couldn't find a way to make the cmd tag not line break before/after.  Because it would really break up the above, I used the file tag instead.)


----------



## wblock@ (Mar 10, 2011)

Ruler2112 said:
			
		

> 1.  make configdeps | make configusedefaults
> 
> To compile x11/kde4 on a new system, there are a whole slew of dependencies that need to be built/installed - not a big deal.  However, whenever one of these dependencies pops up a config screen to set options for that port, the compile stalls until I pick the options and hit OK.  I constantly have to click over to the server on my KVM throughout the day to set options and continue the compile.



`# make config-recursive`
does this.  You can combine make targets to do the config and then the build:
`# make config-recursive build`

It's also been pointed out to me that make does not necessarily build the targets in order, so a possibly better way would be
`# make config-recursive && make`
("build" is the default.)



> Another option that might be of use is make configusedefaults.  This would simply bypass the config screen and use the default options, exactly as if you were to use [TAB][ENTER] on each config screen.



I think this is what setting the "BATCH" variable does.
`# make BATCH=yes`

But I don't use BATCH, so check that.



> 2.  make rmdistafter / make cleanafter | make savespace
> 
> I allocated only 10 gig for my /usr partition.  While more than large enough for my mail server, I did not foresee that I'd run short of space on a kde compile.



Running KDE on a mailserver is probably not necessary.  If you must have X on a server, consider a lighter window manager, i.e., anything other than KDE.

The "distclean" target combines a clean with removing the distfile:
`# make install && make distclean`

It might be possible to delete distfiles after each every dependent port is built, but that will run into problems where multiple ports use the same distfiles.  emulators/virtualbox-ose and emulators/virtualbox-ose-kmod, for example.



> 3.  make concurrent


(download while building)

Interesting.  Don't know how tricky it would be to do, but maybe not a lot different from the MAKE_JOBS_NUMBER concurrency.



> Another option that may work better is to have the main make process simply download all the distfiles for the dependencies of the port you're in, one after the other.



You can fetch all the distfiles in a step, although not concurrent with building.
`# make fetch-recursive`

Well, you can make it build in another shell, but there's no real safety.



> 4.  make rmolddist
> 
> This one is applicable to any system that's run for any length of time that has had the ports updated.  All the distfiles for the old port versions are stored.  I don't know how the specifics of the ports system work and therefore don't know how involved it would be, but a make rmolddist target to simply remove all the old distfiles for the current port, keeping the current one, would be very nice.



ports-mgmt/portupgrade (or -devel) includes portsclean(1) which can do that and more.

`# portsclean -CLD`
cleans out old work directories, libraries, and distfiles.


----------



## Ruler2112 (Mar 10, 2011)

Wow... so basically, everything but the building while downloading already exists - I just didn't know about it.  Thanks for the info.

Sorry - should have been more clear.  This machine isn't going to be a mail server, it's going to be my PC.  I have another machine that I've run FreeBSD on for several years that is a mail server.  There's no X on that one.

I picked KDE because it's what I'm used to on my Slackware box... You're the second person who has told me it's overly big/slow - is there a better option available, something that doesn't sacrifice the functionality of KDE for speed?  (This is really a general question though.)

Is there a list somewhere of all the make options like this?


----------



## wblock@ (Mar 10, 2011)

The most common port options are in ports(7).  But there are others that can be discovered by reading the files in /usr/ports/Mk.  The comments at the start of bsd.port.mk describe a lot of the variables and targets.


----------



## Ruler2112 (Mar 10, 2011)

*holds my face in my hands and shakes my head*

It's in the man page... I feel like such an idiot!


----------



## wblock@ (Mar 10, 2011)

AFAIK, KDE really is the biggest "desktop environment".  If it runs fast enough for you, use it.  I just don't have the patience to wait for the menus to appear, or for all the dependencies to build, so I use x11-wm/xfce4.  It's sort of a GNOME-lite.  There are lots of window managers in ports.  You can find some information at http://xwinman.org/, http://www.gilesorr.com/wm/table.html, and http://en.wikipedia.org/wiki/Comparison_of_X_Window_System_desktop_environments.  There's a nice page somewhere at freedesktop.org, but I can't find it again.


----------



## wblock@ (Mar 10, 2011)

Also:

ports-mgmt/portmaster and ports-mgmt/portupgrade-devel both can automate a lot of the processes.  For example, portmaster automatically gives you all the config screens before building, and deletes old distfiles afterwards.


----------



## DutchDaemon (Mar 11, 2011)

When using ports-mgmt/portmaster you can set almost all of these options in /usr/local/etc/portmaster.rc:


```
$ cat /usr/local/etc/portmaster.rc.sample                                                                                
# $FreeBSD: user/dougb/portmaster/files/portmaster.rc.sample.in 211231 2010-08-12 19:15:21Z dougb $
#
# Sample portmaster rc file.
# Place in $HOME/.portmasterrc or /usr/local/etc/portmaster.rc
#
# Please read the portmaster man page for more information on
# these options.
#
# Do not create temporary backup packages before pkg_delete (-B)
# NO_BACKUP=Bopt
#
# Always save the backup packages of the old port (-b)
# BACKUP=bopt
#
# Make and save a package of the new port (-g)
# MAKE_PACKAGE=gopt
#
# Do not preclean the port's build directory (-C)
# DONT_PRE_CLEAN=Copt
#
# Do not clean the port's build directory after installation (-K)
# DONT_POST_CLEAN=Kopt
#
# Never search for stale distfiles to delete (-D)
# DONT_SCRUB_DISTFILES=Dopt
#
# Always delete stale distfiles without prompting (-d)
# ALWAYS_SCRUB_DISTFILES=dopt
#
# Do not run 'make config' for ports that need updating (-G)
# PM_NO_MAKE_CONFIG=Gopt
#
# Hide the build and install processes in a log file (-H)
# HIDE_BUILD=Hopt
#
# Arguments to pass to make (-m)
# PM_MAKE_ARGS='-DFORCE_PKG_REGISTER'
# 
# Recurse through every dependency, and child dependencies (-t)
#
# NOTE:	USE OF THIS OPTION IN YOUR CONFIG FILE IS NOT RECOMMENDED
#	IT IS ONLY NEEDED IN SPECIAL CIRCUMSTANCES, AND SHOULD BE
#	USED FROM THE COMMAND LINE.
#
# RECURSE_THOROUGH=topt
# 
# Be verbose (-v)
# PM_VERBOSE=vopt
# 
# Save copies of old shared libraries (recommended) (-w)
# SAVE_SHARED=wopt
# 
# Install a package if available (-P or --packages)
# PM_PACKAGES=first
# 
# Only install packages (-PP or --packages-only)
# PM_PACKAGES=only
# 
# Install packages for build-only dependencies (--packages-build)
# PM_PACKAGES_BUILD=pmp_build
# 
# Delete build-only dependencies when finished (--delete-build-only)
# PM_DEL_BUILD_ONLY=pm_dbo
# 
# Use packages if they are newer than installed (--packages-newer)
# PM_PACKAGES=newer
# PM_PACKAGES_NEWER=pmp_newer
# 
# Always fetch new package files (--always-fetch)
# PM_ALWAYS_FETCH=pm_always_fetch
# 
# Specify a local package repository (--local-packagedir)
# LOCAL_PACKAGEDIR=<path>
# 
# Only use packages from --local-packagedir (--packages-local)
# PM_PACKAGES_LOCAL=pmp_local
# 
# Delete packages after they are installed (--delete-packages)
# PM_DELETE_PACKAGES=pm_delete_packages
# 
# Suppress the build confirmation message (--no-confirm)
# PM_NO_CONFIRM=pm_no_confirm
# 
# Do not update the xterm title bar (--no-term-title)
# PM_NO_TERM_TITLE=pm_no_term_title
# 
# Do not fetch the INDEX file (--no-index-fetch)
# PM_NO_INDEX_FETCH=pm_no_index_fetch
# 
# Use only the INDEX file to check if a port is out of date (--index)
# PM_INDEX=pm_index
# 
# Use the INDEX file first, then check /usr/ports (--index-first)
# PM_INDEX=pm_index
# PM_INDEX_FIRST=pm_index_first
# 
# Use the INDEX file instead of /usr/ports (--index-only)
# PM_INDEX=pm_index
# PM_INDEX_ONLY=pm_index_only
#
#
# The following options are available for the rc file only:
#
# Log actions taken by portmaster with a date/time stamp
# PM_LOG=/full/path/to/file
#
# Do not prompt the user for failed backup package creation
# PM_IGNORE_FAILED_BACKUP_PACKAGE=pm_ignore_failed_backup_package
```


----------



## jb_fvwm2 (Mar 11, 2011)

You can mount an extra disk somewhere, and provided it is not on USB which crashes too often here,

```
/bin/rm -rf work && make install WRKDIRPREFIX=/mnt && yell
```
if you've mounted it on /mnt and you have the port (yell) installed. 

(without too much difficulty... can lessen the space problem asked about above)


----------



## phoenix (Mar 17, 2011)

Everything you want, including fetching files in the background, can be done using ports-mgmt/portmaster.


----------

