# Creating/Maintaining custom ports



## jbo (Apr 8, 2022)

I have a lot of tools & libraries that I created over the years which I use heavily for development. Everything that is deemed useful is already publicly available as open source. However, there's a fair chunk of utilities & libraries that won´t be really useful to anybody other than myself (at least in the current state).
I find myself manually pulling, compiling & installing those on several of my machines all the time. For ports, I'm already running a poudriere instance.

My question would be: What is the appropriate way of creating a "custom ports repository" (if that is the correct terminology) and having those built by poudriere as well? Essentially creating packages that I can install via pkg(7) without actually publishing them to the official ports tree.
How would one go about this?


----------



## Menelkir (Apr 8, 2022)

I didn't find anywhere appropriate, but here is the review.
You can add something like this to your /etc/make.conf:

```
OVERLAYS= /usr/local/ports
```
And add your ports to /usr/local/ports, mind the proper categories.


----------



## SirDice (Apr 8, 2022)

Put your custom ports tree in git (you probably already have that), just your own ports and keep the `<category>/<portname>` structure. Then add it as a named ports tree with poudriere-ports(8). With poudriere-bulk(8) you add it, besides the `-p` ports tree, with `-O`:

```
-O overlay
              Specify an extra ports tree to use as an overlay.  Multiple -O
              overlay arguments may be specified to stack them.

              These overlays should be setup with poudriere-ports(8).

     -p tree  Specify on which ports tree the bulk build will be done.
              (Default: “default”)
```


----------



## jbo (Apr 8, 2022)

Hmm, I might have to re-read both that review and the corresponding poudriere manual to fully understand this. My main question at this point would be: Where exactly is that notion of an "overlay" coming from? From my understanding of the overall system architecture this would just be an "additional" ports tree.


----------



## grahamperrin@ (Apr 9, 2022)

jbodenmann said:


> … Where exactly is that notion of an "overlay" coming from? …



For me, <https://forums.freebsd.org/posts/553456> was where the concept of overlays became more understandable.

Some thought-provoking discussion (<https://github.com/freebsd/poudriere/issues/881#issuecomment-892037601> in particular):









						Overlayed ports not reflected in ports tree config · Issue #881 · freebsd/poudriere
					

Prerequisites Have you checked for an existing issue describing your problem? Are you running the latest version? Is your ports tree recent? Is your FreeBSD Host on a supported release? Describe th...




					github.com


----------



## Erichans (Apr 10, 2022)

jbodenmann said:


> [...] My main question at this point would be: Where exactly is that notion of an "overlay" coming from? From my understanding of the overall system architecture this would just be an "additional" ports tree.


Maybe I'm missing the essence of your question but, if you see it as 'just be an "additional" ports tree' how would you easily integrate your private ports tree with the standard ports tree, other than via this overlay option?  The `-O` option enables you to append your private ports tree during a bulk build.

The principle of Overlay (programming)* allows for working with multiple units or segments treated as one—more or less (restrictions apply). Overlay programming was needed when main memory is too small to have a program completely loaded; this was before the era of virtual memory. For environments without virtual memory management this still applies today, especially in embedded applications.

I don't know to what extent poudriere implements overlays however, I think I can say that you don't have to bother yourself with possible memory restrictions when adding one or more separate private ports trees as overlays.  

A specification of overlays defines how segments of a program are related to each other and is usually passed to the linker; overlays can be defined recursively. The specification is used to allow for an overlay segment to be loaded automatically from secondary memory (=disk) to main memory when execution must continue in that segment. Responsible for this automatic loading at runtime is an overlay manager. Thus, after the puzzle of selecting all segments and integrating them into an overlay design, the programmer does not have to inject extra code that explicitly loads segments at runtime when they are needed. On the other hand, when an overlay manager is absent, it's the programmer's responsibility to explicitly call an overlay, for example from FORTRAN source code.

The picture below shows an example of the logical structure of individual segments. Because of memory constraints, segments A, B and C can not to be in main memory at the same time. In this tree structure the main program functions as root. As the program is running, execution moves  from one segment to another; this is visualised by the bended green arrows which represent calls (returns from calls are not shown). When execution starts in the root and following successive calls from one segment to the next, the sequence of segments from the root to the segment where the code is executing is called a path. During execution, calls must be made along those paths defined by the logical structure. A call across the tree (an exclusive call), for example from segment A to segment B is not possible. The layer of segments can be extended by another level as can be seen by segments D & E and, as before, the segments D & E cannot be in main memory at the same time.




This layout means that when segment C is loaded in main memory A & B are not.  When a call is made from segment C into either D or E, the overlay manager loads the appropriate segment into main memory, if it is not already loaded. Suppose we have the situation that code of segment D is already executing. That means that segments C and D are in main memory, in addition to the main program. When the call returns from segment D to C and when, at some point in time, a call is made into segment E then E is loaded into main memory. Of course segment D is discarded from main memory by this action.

The specification defines how segments are related to each other and defines the overlay structure. I now do not consider  segments D and E any further. By defining segments A, B and C as _Overlay 1_, _Overlay 2_, _Overlay 3_ respectively, the overlay structure is shown (from Overlay (programmierung)) in the picture below (the overlay manager is not shown).




It is important to realise that this tree structure formed by overlays is not the tree structure inside either the standard ports tree or your private ports tree.

Suppose you have the ports accessibility/sct & www/firefox and in addition to that you have your private port `tools/tool1` in your `privateportstree`, structured in the same way as the FreeBSD ports tree as SirDice suggested.
Without overlays:

```
# poudriere bulk accessibility/sct www/firefox -p default; poudriere bulk tools/tool1 -p privateportstree
```
With overlays:

```
# poudriere bulk accessibility/sct www/firefox tools/tool1 -p default -O privateportstree
```
I think the advantage will be more obvious when this `-O` option is used in combination with the `-f` option to specify the ports in a separate file.

Note that if a port in your `privateportstree` depends on a port in the default ports tree, the first option will not work. For that you have to integrate the two trees, for example via git; but then you'll have created a new ports tree. Another tree that you'll have to maintain as the default ports tree gets updates, as well as your `privateportstree`, when you produce more code.  

___
* Have a look at the Wikipedia entries for other examples and references of overlays. The subject of overlays is related to linkers and loaders.  For an in depth look at linking (for example dynamic linking in C++) and loading, I'd suggest you have a look at _Linkers and Loaders_ by John R. Levine. From the summary of Chapter 8 _Loading and Overlays_:


> [...] [Overlays] require a great deal of manual programmer work to design and specify the overlay structure, generally with a lot of trial and error "digital origami", but they were a very effective way to squeeze a large program into limited memory.
> Overlays originated the important technique of "wrapping" call instructions in the linker to turn a simple procedure call into one that did more work, in this case, loading the required overlay. Linkers have used wrapping in a variety of ways. The most important is dynamic linking, which we cover in chapter 10, to link to a called routine in a library that may not have been loaded yet. Wrapping is also useful for testing and debugging, to insert checking or validation code in front of a suspect routine without changing or recompiling the source file.


----------



## jbo (Apr 10, 2022)

Erichans Thank you very much for having gone through the efforts of creating that elaborate post. This was helpful!

I'm aware that creating just a "thank you" post is against forum rules but I felt like this deserved more than just clicking the _"Thanks"_ button.
To bend the rules a bit I'm going to add this information: I'm familiar with the concept of overlays in terms of programming. I failed to make the link/connection of this to ports.


----------



## bakul (Apr 11, 2022)

I have carried my own set of libraries and programs for decades (to run on a number of different Unix variants). I just make sure it is all as self contained as possible. Needs nothing more than make and relevant compilers.


----------

