# Local cache for PKGs?



## balanga (May 9, 2017)

Is it possible to set up a local cache/repository of FreeBSD pkgs?


----------



## Beastie (May 9, 2017)

Packages that you install are already cached. See pkg.conf(5):


> PKG_CACHEDIR: string
> Specifies the cache directory for packages. Default: /var/cache/pkg


You can change the location if you want:
`# echo "PKG_CACHEDIR: /local/package/repository" >> /usr/local/etc/pkg.conf`.

Packages can also be installed locally if you already have all required dependencies in the same location. Check this thread.

I've never tried it myself, but I guess you should be able to set up and use a local repository the same way you access a remote one by editing a conf file in /usr/local/etc/pkg/repos/ to point to the local repository instead, that is the "url" option in the file will have to be set to a local URI, e.g. file:///local/repository/.


----------



## SirDice (May 9, 2017)

And if you want to set up your own repository use ports-mgmt/poudriere or ports-mgmt/synth.


----------



## ShelLuser (May 9, 2017)

If you want to re-use your cache you can also always share / provide it to other servers and let them use it as their primary source. Some ports can sort this out automatically, but this is also very easily set up manually. Personally I think it's even easier (but then again, I'm using ports-mgmt/portmaster which also sets up some kind of "cache" (in the large sense of the word)).


----------



## balanga (May 9, 2017)

Thanks for all the replies... Now if I wanted to reinstall FreeBSD 11.0-RELEASE having already installed it, how would I specify Mirror selection and how do I know what needs to be at that location if I want to use my own ftp server?


----------



## SirDice (May 9, 2017)

Note, you can't use /var/cache/pkg as a repository but you can use those packages to pkg-add(8) them. For example `pkg add /var/cache/pkg/tmux-2.3_2.txz`. Also remember that a pkg-clean(8) will remove the cached packages.

Repositories require a certain directory structure and catalog (see pkg-repo(8)).


----------



## balanga (May 18, 2017)

Just wondered if it is possible to `pkg 'download' pkgname` save this to a file and subsequently `pkg install 'from download'`...

The system I want to install the pkg on does not have Internet access.


----------



## SirDice (May 18, 2017)

pkg-fetch(8)


----------



## balanga (May 23, 2017)

ShelLuser said:


> If you want to re-use your cache you can also always share / provide it to other servers and let them use it as their primary source. Some ports can sort this out automatically, but this is also very easily set up manually. Personally I think it's even easier (but then again, I'm using ports-mgmt/portmaster which also sets up some kind of "cache" (in the large sense of the word)).



I'm wondering if I can somehow utilise SQUID on my pfSense box as a local cache.... I have no idea whether this is possible or whether I'm retrieving files from SQUID's cache when I do a `fetch`.


----------



## sko (Nov 14, 2017)

Just in case somebody else stumbles over this thread as I did when searching for some details on how to simplify/enhance my local caches, I'll add my current solution to this topic.

Firstly: There is no special package available for FreeBSD/pkg like e.g. apt-mirror on debian-based Linux systems. This is due to the fact, that a local caching mirror for even multiple pkg-based systems (this was what I was searching information for...), varying release versions and also freebsd-update is _really_ easy to set up with nginx.

Instead of multiple caches/repositories, I'm now using a single nginx installation within a jail to provide caches for all our local FreeBSD (pkg + freebsd-update) and TrueOS installations. The various caches/proxies are distinguished by their subdomain, e.g. freebsd-update.localdomain.lan; trueos-pkg.localdomain.lan etc.... It might be possible to distinguish by the location/path; but this way it would be an easy task to separate them if needed without the need to update configs on 100+ systems/jails.

The nginx.conf server entry for a cache looks like this:


```
server {
                listen          80;
                server_name     freebsd-pkg.dmz.localdomain.lan freebsd-pkg.lan.localdomain.lan freebsd-pkg.localdomain.lan;

                location / {
                        root                    /var/cache/packages/freebsd/;
                        try_files               $uri @cache;
                }
                location @cache {
                        root                    /var/cache/packages/freebsd/;
                        proxy_store             on;
                        proxy_pass              https://pkg.freebsd.org;
                        proxy_cache_lock        on;
                        proxy_cache_lock_timeout        20s;
                        proxy_cache_revalidate  on;
                        proxy_cache_valid       200 301 302 24h;
                        proxy_cache_valid       404 10m;
                }
        }
```

The directory specified in both "root" entries has to be created and owned by the user nginx is running as (e.g. www).
To add another caching proxy, just add another server entry for another subdomain (or use different "location" directives) and create the cache directory. E.g. for freebsd-update:


```
server {
                listen          80;
                server_name     freebsd-update.dmz.localdomain.lan freebsd-update.lan.localdomain.lan freebsd-update.localdomain.lan;

                location / {
                        root            /var/cache/freebsd-update/;
                        try_files       $uri @cache;
                }
                location @cache {
                        root                    /var/cache/freebsd-update/;
                        proxy_store             on;
                        proxy_pass              https://update.freebsd.org;
                        proxy_http_version      1.1;
                        proxy_cache_lock        on;
                        proxy_cache_lock_timeout 20s;
                        proxy_cache_revalidate  on;
                        proxy_cache_valid       200 301 302 24h;
                        proxy_cache_valid       404 10m;
                }
        }
```


How it works:
If a host with this server in its pkg/freebsd-update configuration uses one of the specified server_names, nginx will look at the "root" directory set for that location if it already cached the requested files (e.g. packages). If true, it serves them directly from disk. If it isn't present it proxies the request to the URL specified in "proxy_pass", downloads the file to the local cache and (simultaneously) serves it to the local client. Every client asking for that file later will get it from the local cache.
To ensure the files are updated and/or expired, the proxy_cache_revalidate and proxy_cache_valid directives are set to check if there were changes on the origin server.

Using different locations might be the simpler solution if all caches will always be only on one system - as said I wanted to be able to easily migrate/switch single caches to another machine/jail if needed (e.g. to the same machine the build server is running on).
With the shown configuration this task would only require an update to the nameserver entries, not to all clients.


For further reading on fine-tuning the cache, the nginx blog and the admin-guide section contain some very useful articles:
https://www.nginx.com/blog/nginx-caching-guide/
https://www.nginx.com/resources/admin-guide/reverse-proxy/

Also have a look at the ngx_http_proxy_module:
https://nginx.org/en/docs/http/ngx_http_proxy_module.html


----------



## balanga (Feb 12, 2018)

I was wondering if it's possible to put this repository on my pfSense box which happens to run nginx...

Has anyone tried?


----------



## PacketMan (Feb 13, 2018)

Something a fair bit different, and is not caching, but might interest you.  I set up a server to be a dedicated ports-mgmt/synth builder, and then used net-p2p/btsync to distribute the newly built packages to the target machines.

See Thread 58708


----------



## jhfoo (May 29, 2020)

sko said:


> Just in case somebody else stumbles over this thread as I did when searching for some details on how to simplify/enhance my local caches, I'll add my current solution to this topic.
> 
> Firstly: There is no special package available for FreeBSD/pkg like e.g. apt-mirror on debian-based Linux systems. This is due to the fact, that a local caching mirror for even multiple pkg-based systems (this was what I was searching information for...), varying release versions and also freebsd-update is _really_ easy to set up with nginx.
> 
> ...



sko is the above strategy still valid for 12.1? Are there changes needed to make pkg recognise the local repo url?


----------

