# What is proper method to create a local mirror of pkg package repositories now that pkg.freebsd.org returns 401 when attempting to sync



## auralox (Sep 18, 2020)

Hi Freebsd Forum community, 

We run a large installation of Freebsd 11 and 12 servers that have historically utilized a local FreeBSD package repository mirror on-premise that we sync periodically. The method we have been using for years --  since the official repositories were poisoned a few years ago -- no longer works due to the HTTP policy change on the *pkg.freebsd.org *servers.  We have been trying to understand the new policy but I've not been able to find much. The best quote of the "non-answer" I've found of the about the changed policy is below (source):



> Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages. For sites with lots of machines, it might be advantagous to run a caching HTTP proxy for the pkg(8) process. Alternatively specific packages and their dependencies can be fetched by running something like the following:



We have a pretty diverse environment of FreeBSD hosts and generations and the current recommendation to manually download and sync all the packages we use like this with the `pkg` tool is not a scalable approach. [In fact the only way I can conceive of to do this would be to analyze our configuration management system and build a list of packages from there?) Although we have a working poudriere environment I'd prefer not to build the whole ports tree myself.  The method I see in the current manual as the recommended -- untenable -- approach included with the anwer quoted above is to do the following for every package to get it and its dependencies:


```
% pkg fetch -d -o /usr/local/mirror vim
```

Does anyone have a solution for us other than either running the above command for every package we need to install or building the whole ports tree with Poudreire? Or am I missing something? I'm open to ideas and would really love to hear any other solutions folks have working. 

Thanks in advance,
Josh


----------



## chrbr (Sep 19, 2020)

Hello auralox,
in post #7 of https://forums.freebsd.org/threads/re-use-downloads-for-freebsd-update.76916/ you can find my current solution using www/squid as a caching proxy. Please be careful, I am a hobbyist only. The better approach might be a machine which downloads all required packages and its dependencies and offers the files by a web server. Something which is described in the setups using ports-mgmt/poudriere.


----------



## T-Daemon (Sep 19, 2020)

auralox said:


> Does anyone have a solution for us other than either running the above command for every package we need to install



There is no need to run the command for every package separately, you can compile a list (prime-list) of packages as you mentioned it earlier and let pkg(8) read that list, e.g.: `pkg fetch -y -d -o /usr/local/mirror `cat prime-list``.

The list can be expanded by running `pkg prime-list > prime-list` on machines having packages installed. You need also two separate systems, one for 11 and 12. The packages are downloaded according the systems ABI.



auralox said:


> or building the whole ports tree with Poudreire?


Do you need the whole ports tree, categories like arabic, astro, chinese, games, etc.? Other solution except those already mentioned are none with the official repositories.


----------



## auralox (Sep 21, 2020)

Thanks chrbr and T-Daemon, I appreciate the feedback. However, neither of your answer's are particularly encouraging.



> my current solution using www/squid as a caching proxy



Although a SQUID proxy would be helpful if we were concerned about bandwidth, the technical requirement here is to have a *static package repository* not to use local caching or bandwidth limitations. In fact, the reason we don't use the public repositories to begin with is that we don't want hosts that were built or patched at different times to be running different minor versions of software across our fleet. I don't see how we can even "fake" having static package versions with this approach, although it would get us a local copy of the packages at least.



> There is no need to run the command for every package separately, you can compile a list (prime-list) of packages as you mentioned it earlier and let pkg(8) read that list, e.g.: pkg fetch -y -d -o /usr/local/mirror `cat prime-list`.



Yeah, I can see why this feature got added to pkg, but this is the antitheses of a fully baked "Enterprise" solution. It adds so many steps to installing a package in a well managed environment! Let me walk you through this: If we want to add packages to a host via configuration management we'd first have to sync the package manually to the package server AND re-generate the metadata before we even try to install it? I'm flummoxed how anyone thought this was a sustainable solution for enterprise clients.

I mean I guess we have to move to a caching proxy, but this is the kind of stuff that makes me want to just stop running FreeBSD entirely. 

Disappointed,
Josh


----------



## crashcoredump (Feb 5, 2022)

auralox said:


> this is the antitheses of a fully baked "Enterprise" solution
> 
> but this is the kind of stuff that makes me want to just stop running FreeBSD entirely.
> 
> ...



Josh, have you ever resolve any of your issues or did you decide to drop FreeBSD as a platform after all?


----------

