# Ports transitioned to git.



## SirDice (Mar 31, 2021)

Last SVN commit to the ports tree:






						[ports] Revision 569609
					






					svnweb.freebsd.org
				




So make sure to change your local ports trees if you've been using subversion to update it. portsnap(8) users should not be impacted by this change.


----------



## zirias@ (Mar 31, 2021)

Note that we will have a few days without any ports upgrades whatsoever:


			Git - FreeBSD Wiki


----------



## jb_fvwm2 (Mar 31, 2021)

news.freshports.org is also impacted by this change.


----------



## drhowarddrfine (Apr 1, 2021)

fwiw, I tried updating the ports tree using git a couple of weeks ago and had no issues doing so.


----------



## richardtoohey2 (Apr 1, 2021)

For those of those not yet up to speed, will "portsnap fetch update" work or I need to do something different?


----------



## zirias@ (Apr 1, 2021)

richardtoohey2 said:


> For those of those not yet up to speed, will "portsnap fetch update" work or I need to do something different?


portsnap is deprecated, but AFAIK, this doesn't relate directly to git. I don't know for how long it will still work.

My recommendation would be to use net/gitup instead, givig very similar functionality, but directly accessing a git repository for it.


----------



## jmos (Apr 1, 2021)

Zirias said:


> portsnap is deprecated



Then the handbook should be updated:









						Chapter 4. Installing Applications: Packages and Ports
					

FreeBSD provides two complementary technologies for installing third-party software: the FreeBSD Ports Collection, for installing from source, and packages, for installing from pre-built binaries




					docs.freebsd.org
				




Portsnap also comes with the base system, and there's no hint to that in the manpage or "--help" option. Shure that is official & written in stone - and not a dream of some fans of git?


----------



## SirDice (Apr 1, 2021)

richardtoohey2 said:


> For those of those not yet up to speed, will "portsnap fetch update" work or I need to do something different?





SirDice said:


> . portsnap(8) users should not be impacted by this change.


----------



## zirias@ (Apr 1, 2021)

jmos https://lists.freebsd.org/pipermail/freebsd-ports/2020-August/119098.html

portsnap is still in 13 and will continue to work, but it will be removed at some point in time, so why not use another tool right now? gitup is even a bit easier to use than portsnap and will always get the very latest (while portsnap relies on spashots created from time to time).


----------



## _martin (Apr 1, 2021)

It seems my FreeBSD knowledge is getting stale, I'll need to catch up on these svn to git migration changes.


----------



## SirDice (Apr 1, 2021)

_martin said:


> I'll need to catch up on these svn to git migration changes.


Not much to catch up on, the ports tree is the last to get migrated. Source and docs have been migrated already. Source for releng/11.4 and releng/12.2 can still be found on subversion, from releng/13.0 onward it's only available through git. But this really only impacts people that use the source to update/upgrade, nothing's changed for freebsd-update(8) users.


----------



## _martin (Apr 1, 2021)

SirDice I don't use git too often and when I do I always have to google how to set up the repository.  I need to get familiar with the git, how to summarize the different commits, etc.
I was lazy to do so, I guess now I have an extra push and motivation to do so. 
I always used cvs and later svn. I do like that approach and prefer it to freebsd-update(8). It's only a personal preference, nothing more behind it.


----------



## Deleted member 30996 (Apr 1, 2021)

It wasn't updated when I checked yesterday or this morning when I checked first thing:

`root@bakemono:/ # portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Ports tree hasn't changed since last snapshot.
No updates needed.
Ports tree is already up to date.`

It was updated in time to get the fixed version of security/nettle 3-30-21 after I contacted the Maintainer. That's the last update I got or expect till the change is finalized.


----------



## jmos (Apr 1, 2021)

Zirias said:


> gitup is even a bit easier to use than portsnap



`pkg version -vl '<'` took a blink of an eye. Now i tested a `gitup ports`, and that pkg command takes now ~40 seconds till the ~200 packages are listed that the ports tree is newer than the quarterly binary packages I've got installed.

Your link tells me that git saves some disk space - but it seems to me that it was "usefull wasted" before  Is there a better (faster) command now to get a list of the updateable ports? "/var/db/pkg/repo-FreeBSD.sqlite" seems no longer to be modified, so directly asking that database is also not an option anymore…

(No one asked, but: I don't think that anyone would not contribute because there's svn and not git (that was named as the reason for this switch). Git is just the next cool must have to play with. It has zero benefit to me, but is a additional burden to deal with.)


----------



## zirias@ (Apr 1, 2021)

jmos said:


> `pkg version -vl '<'` took a blink of an eye. Now i tested a `gitup ports`, and that pkg command takes now ~40 seconds till the ~200 packages are listed that the ports tree is newer than the quarterly binary packages I've got installed.


Both tools give you a plain tree of files. With portsnap, you have metadata in /var/db, with git in a _single_ subdirectory in /usr/ports. This doesn't change anything about the speed of traversal on the filesystem level.



jmos said:


> Your link tells me that git saves some disk space - but it seems to me that it was "usefull wasted" before


No. No other tool except portsnap itself ever uses portsnap's metadata.

*edit:* Ha, looking a bit deeper into this: pkg DOES use an index if present in the ports tree. That's _not_ portsnap's metadata, but portsnap fetches it together with the snapshots. You can fetch it yourself with `make fetchindex`. Drawback: could be slightly out of date, similar to the snapshots portsnap fetches. But at least only a few hours, while porsnap snapshots can be a few days old.

`make index` will build it locally, which, of course, requires traversal again.



jmos said:


> (No one asked, but: I don't think that anyone would not contribute because there's svn and not git (that was named as the reason for this switch). Git is just the next cool must have to play with. It has zero benefit to me, but is a additional burden to deal with.)


I get it doesn't have any value for YOU. Well, bad luck, but the rest is utter nonsense.


----------



## Deleted member 30996 (Apr 1, 2021)

When you first run `# portsnap fetch extract` the snpshot it downloads is approximately 76MB in size, possibly one or two larger. I didn't pay that close attention to it, having seen it in the past, but will be sure to take note next laptop I build. 

Very soon.


----------



## SirDice (Apr 1, 2021)

jmos said:


> "/var/db/pkg/repo-FreeBSD.sqlite" seems no longer to be modified, so directly asking that database is also not an option anymore…


The only tool that updates /var/db/pkg/repo-FreeBSD.sqlite is pkg-update(8) itself.


----------



## SirDice (Apr 1, 2021)

Zirias said:


> Ha, looking a bit deeper into this: pkg DOES use an index if present in the ports tree.


Yes, that's why I always explicitly call `pkg version -vR` or `pkg version -vI` because the local INDEX file may or may not be up to date and this could skew the results. 


```
-I [index, --index [index]]
                 Use index file for determining if a package is out of date.
                 If no index file name is specified, uses the default index
                 file.  This is the default, if the index file exists.
```


```
-R, --remote
                 Use repository catalogue for determining if a package is out
                 of date.  This is the default if neither the ports index nor
                 the ports tree exists.
```



Zirias said:


> `make index` will build it locally, which, of course, requires traversal again.


I prefer `make index` because it also takes my `DEFAULT_VERSIONS` settings into account. A `make fetchindex` will get a remote INDEX that uses the standard defaults. 
I use this to update my ports tree:

```
cd /usr/ports && git pull && make index
```



_martin said:


> I always used cvs and later svn. I do like that approach and prefer it to freebsd-update(8). It's only a personal preference, nothing more behind it.


If you're already familiar with CVS and SVN then GIT shouldn't be too difficult to understand. Some commands are slightly different but they all serve the same or a similar purpose.


----------



## PMc (Apr 1, 2021)

WARNING: the cgit-beta ports tree is NOT stable!

There is currently no ports tree visible on the cgit server. As was suggested here on the forums, one can use the cgit-beta which does provide a ports tree.
If you do so (to test your local update scripts, like I am doing currently), there may be surprizes.

Shorthand: when you see "forced update" during fetch, the upstream may have changed in unexpected ways (as obviousely people are also testing there  ).


----------



## zirias@ (Apr 1, 2021)

Yes, rewriting history is perfectly possible with git and the single worst no-no for anything public, for reasons  Of course, on a beta testbed, you might have to expect such things


----------



## grahamperrin@ (Apr 2, 2021)

jb_fvwm2 said:


> news.freshports.org is also impacted by this change.



From 



__ https://twitter.com/i/web/status/1375914698009419779_View: https://twitter.com/DLangille/status/1375914698009419779_
:



> … the most significant change to the FreshPorts codebase since it started over 20 years ago. …



In addition to FreshPorts:

FreshSource login is broken on new git code · Issue #232 · FreshPorts/freshports


----------



## tuaris (Apr 2, 2021)

Does this mean I need to resubmit any open PR's for new/updates to ports?


----------



## grahamperrin@ (Apr 2, 2021)

tuaris said:


> Does this mean I need to resubmit any open PR's for new/updates to ports?


Use of FreeBSD Bugzilla https://bugs.freebsd.org/bugzilla/ remains unchanged. 

Mirror https://gitlab.com/FreeBSD/ports offers issues, but the section is unused. 

Read-only mirror https://github.com/freebsd/freebsd-ports/ does not offer issues.


----------



## befreesd (Apr 3, 2021)

I've been using svn ever since it replaced cvsup. What's the procedure to check out the source and ports trees from git and keep them updated going forward? Are there repositories to `git clone`? I'm skimming the handbook but 24.5.3 Updating the Source and 4.5 Using the Ports Collection both still discuss using subversion, and point to svn repos.


----------



## grahamperrin@ (Apr 3, 2021)

befreesd said:


> … 24.5.3 Updating the Source and 4.5 Using the Ports Collection …


Both links are from an outdated edition of the Handbook.

*Documentation in general *

Outdated links are very frequently seen (not your fault).

Folks, please habitually:

page down to the foot of the page
click *Home*
if there appears a modification date and a revision, with reference to a commit, you're in an outdated document
if you're in an outdated document, page down to the foot of the page
click *documentation* then (e.g.) *Handbook*.
As far as I can tell: with current editions of documentation such as the Handbook, there's no modification date at the head of each chapter.

I do like documents to be dated but in this case, it's paradoxically helpful to be without a date because it probably signifies that it's up-to-date.


*Ports in the FreeBSD Handbook*

At a glance, the current edition of the Handbook is similarly non-prepared for completion of the transition.

I'll make a separate post with an outline of (or link to) what's probably required.


----------



## befreesd (Apr 3, 2021)

Whoops, sorry about that. I Googled "freebsd handbook" and unfortunately the top result is the outdated one I clicked into. I do see that the current version of 24.5.3 Updating Source has the git instructions I needed for /usr/src. I'll keep an eye out for advice regarding the ports tree. Thanks!


----------



## grahamperrin@ (Apr 3, 2021)

Beginning to use Git for FreeBSD ports: a guide


----------



## grahamperrin@ (Apr 3, 2021)

befreesd said:


> Whoops, sorry about that. I Googled …


No need to apologise. 

Any number of other search engines might be similarly affected; FreeBSD Forums directing readers to oudated documentation; plus there are countless news articles and blog posts with outdated links; and so on.

In IRC, someone mentioned work to improve the situation. I assume that redirects will be involved.


----------



## Deleted member 30996 (Apr 3, 2021)

If I should be worried I'm not and don't plan on changing anything till the ports tree is updated.

`root@bakemono:/ # portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Ports tree hasn't changed since last snapshot.
No updates needed.
Ports tree is already up to date.
root@bakemono:/ # pkg audit -F
vulnxml file up-to-date
0 problem(s) in 0 installed package(s) found.
root@bakemono:/ # freebsd-update fetch
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 12.2-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 12.2-RELEASE-p5.
root@bakemono:/ #`

But I don't always make the best decisions. Please feel free to correct me.


----------



## grahamperrin@ (Apr 3, 2021)

You need not worry. From the opening post: 



> portsnap(8) users should not be impacted by this change.


----------



## Deleted member 30996 (Apr 3, 2021)

I just built www/youtube_dl from ports using ports-mgmt/portmaster with the flag set for LAME, am watching the video I just posted in that thread and got sound with it:


```
===>  Installing for youtube_dl-2021.03.31
===>  Checking if youtube_dl is already installed
===>   Registering installation for youtube_dl-2021.03.31
Installing youtube_dl-2021.03.31...
If you want to use mp3 audio conversion please make sure multimedia/ffmpeg is
built with the "LAME" option enabled.


===>>> The following actions were performed:
    Installation of multimedia/librtmp (librtmp-2.4.20190330)
    Installation of multimedia/rtmpdump (rtmpdump-2.4.20190330)
    Installation of www/youtube_dl (youtube_dl-2021.03.31)

root@bakemono:/ #
```

Which is what I expected to happen.


----------



## lasuit (Apr 3, 2021)

Sir Dice has said that portsnap should not be affected, and I believe that, however, I just want to point out that I am using 12.2 RELEASE-p4 GENERIC amd64 and nothing has changed since March 31, 2021.  Even when ports that I have don't change there is usually some port change or addition on a daily basis, so I am surprised to see zero changes for three days.


----------



## grahamperrin@ (Apr 3, 2021)

lasuit said:


> …  surprised to see zero changes for three days.



No surprise. https://old.reddit.com/r/freebsd/comments/mizghz/-/gt7gv7u/ point 1 refers to the wiki. Write access is disabled.

It's also on the first page here: 



Zirias said:


> Note that we will have a few days without any ports upgrades whatsoever:
> 
> 
> Git - FreeBSD Wiki


----------



## fernandel (Apr 3, 2021)

grahamperrin said:


> No surprise. https://old.reddit.com/r/freebsd/comments/mizghz/-/gt7gv7u/ point 1 refers to the wiki. Write access is disabled.
> 
> It's also on the first page here:


On mailing list I saw recomendations for gitup. Should be /usr/ports and /var/db/ports removed and than run `gitup -v 1 ports` for example?


----------



## zirias@ (Apr 3, 2021)

/var/db/ports is configuration (build options) written _by_ ports, and has nothing to do with how you update your ports tree. Don't remove unless you _want_ to forget everything configured with dialog4ports interface.


----------



## scottro (Apr 3, 2021)

I have a page that doesn't cover ports, but gives the basic command to update source.


			Using git To Get FreeBSD Source Code


----------



## grahamperrin@ (Apr 4, 2021)

Thanks, 



scottro said:


> … doesn't cover ports, … to update source. …



– in general, --depth 1 is no longer recommended; see the red alert at https://docs.freebsd.org/en/articles/committers-guide/#_shallow_clone

Yesterday someone in IRC found a mistake in my (rough) guide. That one was corrected … 

… please, can you see any other mistake?


----------



## zirias@ (Apr 4, 2021)

grahamperrin said:


> – in general, --depth 1 is no longer recommended; see the red alert at https://docs.freebsd.org/en/articles/committers-guide/#_shallow_clone


This is the _committers' guide_. For a committer, it makes little sense to omit history. Furthermore, the warning in this section _could_ be relevant if you follow -STABLE or -CURRENT, depending on your local workflow. It isn't really relevant for releases, as you will know from the name (RELEASE-p1, RELEASE-p2 etc) which security fixes are already present.

If all you want is the most recent source of a branch, there's nothing wrong with `--depth 1`. In that case, you might also want to use net/gitup instead.


----------



## grahamperrin@ (Apr 4, 2021)

Zirias said:


> portsnap is deprecated, … I don't know for how long it will still work. …



Not yet deprecated.

There was:

the plan – *[HEADS UP] Planned deprecation of portsnap* (2020-08-04) then
amongst other things https://lists.freebsd.org/pipermail/freebsd-ports/2020-August/119114.html from the same author made clear that "… 11.x and 12.x will have it of course, …" and mentioned the prospect of full removal.
I should not expect deprecation until some time after the final RELEASE of 12.⋯ reaches end of life.

*Postscripts*

Cross reference: Deprecation of portsnap, and use of portsnap (2021-04-09)

I see portsnap being retired - what's the alternative? (2020-08-04, five pages, locked) but I haven't read it; I'm aware of alternatives.


----------



## zirias@ (Apr 4, 2021)

Nitpick: Deprecated doesn't mean removed or defunctional. It just means an announcement, so everybody knows it is scheduled for removal and has time to adapt.

Still, I'm surprised portsnap is still built by default in 13-RC5. Maybe removal will be much later then...


----------



## grahamperrin@ (Apr 4, 2021)

Thanks,



> … I'm surprised portsnap is still built by default in 13-RC5. Maybe removal will be much later then...



This might help (a little) to paint the big picture:

https://www.freebsd.org/cgi/man.cgi?query=ports&sektion=7&manpath=FreeBSD+12.2-RELEASE does mentions portsnap
https://www.freebsd.org/cgi/man.cgi?query=ports&sektion=7&manpath=FreeBSD+13.0-current does not; https://cgit.freebsd.org/src/commit/?id=8c72577900e317393dc43be3df159e0dd4036ddc



Zirias said:


> … If all you want is the most recent source of a branch, there's nothing wrong with `--depth 1`. …



Will that eventually result in a different count? For example,

git -C /usr/src rev-list --count HEAD


----------



## zirias@ (Apr 4, 2021)

With `--depth 1`, your local clone will start with a branch containing only the one latest commit, so, probably yes.

But that's not much different from extracting a source tarball  These things are only interesting if you intend to do things like work on the code yourself, or cherry-pick some patches, etc.


----------



## grahamperrin@ (Apr 4, 2021)

Zirias said:


> … only interesting if you intend to do things like work on the code yourself, or cherry-pick some patches, etc.


I have a script that writes the count into the name of the boot environment.


----------



## a6h (Apr 4, 2021)

fernandel said:


> On mailing list I saw recomendations for gitup. Should be /usr/ports and /var/db/ports removed and than run `gitup -v 1 ports` for example?



Some notes about gitup(1):

* /var/db/gitup is default list location for gitup
* /usr/local/etc/gitup.conf is where you can change gitup default config.
* `git.freebsd.org` is the default host.
* Since 0.90_1 gitup is considered stable.
* Don't mix gitup and Git

/usr/ports is the target directory:
IF /usr/ports does not exist THEN it clones the repo.
IF /usr/ports exists THEN it'll pull down the latest commit.

443 is the default port. gitup relies on 443 for Smart HTTP over HTTPS

What's Smart HTTP

Git has different transfer protocol types:
1. git:// : unauthenticated access.
2. ssh:// : authenticated access.
3. _Smart HTTP_: Both authenticated and unauthenticated access (as opposite to _dumb HTTP_: prior to Git 1.6.6)

Smart HTTP
1. Smart HTTP relies on git-http-backend CGI script (Git over HTTP).
2. Smart HTTP operates similar to ssh:// and git://, but runs over HTTPS and uses HTTP authentication.
3. git-http-backend negotiates with client to send/receive data over HTTP.
4. git-http-backend allows Git client to access to the Git repo over http:// and https://


----------



## grahamperrin@ (Apr 4, 2021)

Thanks. FreeBSD bugs:

254759 – net/gitup manual page, section and availability
254760 – net/gitup default configuration options for ports
From 254759: 


> <https://www.freebsd.org/cgi/man.cgi....2-RELEASE+and+Ports&arch=default&format=html> finds nothing. …


----------



## Mayhem30 (Apr 4, 2021)

The manuals for net/gitup are available.


```
man gitup
man gitup.conf
```


----------



## ljboiler (Apr 5, 2021)

```
git -C /usr clone https://git.freebsd.org/ports.git ports
Cloning into 'ports'...
fatal: repository 'https://git.freebsd.org/ports.git/' not found
```

Shouldn't this work?


----------



## zirias@ (Apr 5, 2021)

As soon as the transition is complete.



			Git - FreeBSD Wiki
		


(it isn't yet)


----------



## SirDice (Apr 5, 2021)

Not yet. It's not public yet. They're still testing. https://github.com/freebsd/freebsd-ports.git works though. It's probably the fastest mirror to pick anyway.


----------



## zirias@ (Apr 5, 2021)

Although I _do_ wonder what went wrong, so the transition didn't finish according to the "expected" case schedule. After all, having done the same for docs and src, you'd expect experience should help 

At least, we're not at "worst" case schedule yet


----------



## a6h (Apr 5, 2021)

PSA. VM is good. Just run a virtual machine, and test different scenarios. I'm doing the same; right now.
At the moment, I have 11 VMs and all are still on SVN mood! I have an extra VM to test Git for different
scenarios. Plus there's 2 bare-metal FreeBSD Desktop machines, too. I won't migrate unless I get it right.


----------



## ljboiler (Apr 5, 2021)

SirDice said:


> Not yet. It's not public yet. They're still testing. https://github.com/freebsd/freebsd-ports.git works though. It's probably the fastest mirror to pick anyway.


Silly me.  I was going by the past tense of the thread subject, assuming that it was a done deal...


----------



## grahamperrin@ (Apr 5, 2021)

SirDice said:


> … still testing. …



FreeBSD bug 254775 – net/gitup ld-elf.so.1: /usr/local/bin/gitup: Undefined symbol "ucl_object_iterate"

Not a bug with the transition, however if gitup is to be formally popularised when the transition completes, it'll make sense for the port to work as expected in non-RELEASE environments.



vigole said:


> PSA. VM is good. …



VirtualBox? Which version?


----------



## a6h (Apr 5, 2021)

grahamperrin said:


> VirtualBox? Which version?


VM aka Virtual Machine in General. bhyve(8) on FreeBSD and VB 6.1 on Windows.


----------



## zirias@ (Apr 5, 2021)

grahamperrin said:


> FreeBSD bug 254775 – net/gitup ld-elf.so.1: /usr/local/bin/gitup: Undefined symbol "ucl_object_iterate"


1. Can't reproduce on my 14-CURRENT, that's built just one week before yours. net/gitup runs fine here. Might try again after a `git pull` and rebuild, strongly expect exactly the same result.

2. When using -CURRENT, you should really be aware of a few things. Problems with undefined symbols are mostly caused by linking against different library versions than the ones present when trying to run the program. You should be prepared to recompile all your ports every time you upgrade base. Yes, there are binary packages for -CURRENT, they will only work when you keep your -CURRENT up to date all the time – it *changes often*, as expected in a *development branch*.



grahamperrin said:


> Not a bug with the transition, however if gitup is to be formally popularised when the transition completes, it'll make sense for the port to work as expected in non-RELEASE environments.


I specifically _don't_ think so. Running -CURRENT is for testing and development, so it probably makes a lot of sense to actually use devel/git on these machines.

Nevertheless, nothing "broken" here.


----------



## grahamperrin@ (Apr 5, 2021)

https://cgit.freebsd.org/ports/ is up at the time of writing. This is not an invitation to test, just an observation.


----------



## bagas (Apr 5, 2021)

I have not figured out how to update ports are now?


----------



## chrcol (Apr 5, 2021)

So what is the url for the gits quarterly?

I had prepped a scripted with information from here https://github.com/bsdimp/freebsd-git-docs/blob/main/URLs.md

Instead of using those for the transition I see there is no 2021Q2 branch, the head has no updates since end of march and the above url is now 404.

Where has this information been communicated?

It seems odd as there was already a quarterly branch in place and I assumed it would just be a transition to it at end of march.

grahamperrin https://cgit.freebsd.org/ports/ has no updates since end of march and no current quarterly branch it seems its not been used now.


----------



## chrcol (Apr 5, 2021)

Zirias said:


> Note that we will have a few days without any ports upgrades whatsoever:
> 
> 
> Git - FreeBSD Wiki



Ahh missed this post, so its a WIP.  Thanks


----------



## grahamperrin@ (Apr 5, 2021)

chrcol said:


> … no 2021Q2 …



From Proposed ports git transition schedule:



> > … transition in advance of the 2021Q2 …


----------



## grahamperrin@ (Apr 5, 2021)

Zirias said:


> My recommendation would be to use net/gitup …



If there was previous use of portsnap(8), it's necessary to remove the contents of /usr/ports/ before a first run of gitup(1), yes?


----------



## scottro (Apr 5, 2021)

I'm not sure.  Seems like it would be a good idea. chrcol, looking at my /usr/local/etc/gitup.conf, I have this for quarterly, but don't know if it is correct. Mostly I use packages, the only port I build is dwm because I use a custom config file, and so far, I've been using portsnap and probably will till it's gone.  Anyway, from my gitup.conf, and this is quarterly's  default, because I haven't edited anything but the release section of the file

```
"quarterly" : {
                "host"       : "github.com",
                "repository" : "/freebsd/freebsd-ports.git",
                "branch"     : "quarterly",
                "target"     : "/usr/ports",
                "ignores"    : [
                        "distfiles",
                        "packages",
```


----------



## grahamperrin@ (Apr 5, 2021)

Thanks. I ran `gitup ports -v 2` for maximum verbosity, imagined that it was stuck because there was nothing (not a word) for a few minutes. 

Eventually:


```
root@mowa219-gjp4-8570p:~ # gitup ports -v 2
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd-ports.git
# Target: /usr/ports
GET /freebsd/freebsd-ports.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com:443
User-Agent: gitup/0.90
Git-Protocol: version=2



==> bytes sent: 148
==> bytes read: 537     bytes_expected: 565     total_bytes_read: 570
001e# service=git-upload-pack
0000000eversion 2
0023agent=git/github-gacb49e25c7ae
000cls-refs
0019fetch=shallow filter
0012server-option
0017object-format=sha1
0000

POST /freebsd/freebsd-ports.git/git-upload-pack HTTP/1.1
Host: github.com:443
User-Agent: gitup/0.90
Accept-encoding: deflate, gzip
Content-type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Git-Protocol: version=2
Content-length: 178

0014command=ls-refs
0022agent=git/github-gacb49e25c7ae0016object-format=sha100010009peel
000csymrefs
0014ref-prefix HEAD
001bref-prefix refs/heads/
001aref-prefix refs/tags/
0000

==> bytes sent: 461
==> bytes read: 573     bytes_expected: 2365    total_bytes_read: 2370
00504010f7bbc03638d71781ce091bf40a0907fa12fe HEAD symref-target:refs/heads/main
003f5f4d6e1d6b07bb34e5cbf866b4ef81e8ccdcb2da refs/heads/2014Q1
003fa3377806e58e030668a0406a35c434394a9333e1 refs/heads/2014Q2
003fa0ccd6f83108ca7e16e64b6fd690dacefc5d0d58 refs/heads/2014Q3
003fce9f5d32567600a981517b7e35b2542a1114ece0 refs/heads/2014Q4
003f5bd325869bdeccff8037a64e2089b799abe798e0 refs/heads/2015Q1
003f7d7c2271f6c957574221e8746e5a356435cd114f refs/heads/2015Q2
003fa69940fb27f798fd49cfecb02cac3785412b76d0 refs/heads/2015Q3
003f38269ac51865d5d9bbaa23f6afcc2e6a6db5e9b1 refs/heads/2015Q4
003f847fef4d2acd28d745c9fe57194624dafc02e927 refs/heads/2016Q1
003f859c6d655ba8ec51549be3d8f52a302e7023d4ad refs/heads/2016Q2
003fdbe3dbb507e6a2068dee12ee7844edc3797e7dea refs/heads/2016Q3
003fe113cbca02fc10dfb61a77d9c547550718cc7190 refs/heads/2016Q4
003f995d6409b4234b211b27bd14989a9c467f4d4b1a refs/heads/2017Q1
003fee7b50bc8addebc46b27f99488b0f2c3a2ef6b94 refs/heads/2017Q2
003fa16d9e9f97932862f4433c03ee35493ec26fd29d refs/heads/2017Q3
003ff204b02d6dcefb91393ebeac39b3b6741b9ae5b1 refs/heads/2017Q4
003f8e67d346991ca584ffbe6ad23380c597e10d0ddf refs/heads/2018Q1
003f06d6d83e775282797de36344c649a2c33a70861d refs/heads/2018Q2
003f20f7d7640bc5014692574a4e09a7191dc021ecce refs/heads/2018Q3
003f6948b132a446b2a84e021a6dd26b5ac7ea8e575a refs/heads/2018Q4
003fd2f5722e633ae238daba15792d4949b15d22c174 refs/heads/2019Q1
003fab797ee3a93f4bb4a3b22900257b9cbd91349887 refs/heads/2019Q2
003fd5d60820925216fc48c025190e0c30a79f52bf3b refs/heads/2019Q3
003ff24745656dbbc619a6ee0360a5cababea241813c refs/heads/2019Q4
003f28b4da349ddcc9e8493e8c3d013eb2fc4107098a refs/heads/2020Q1
003f22e399d695717c14566f60255c72aae4c8c3d08c refs/heads/2020Q2
003fc0d44897151cceb0c4e63f2f7763b030f3019b66 refs/heads/2020Q3
003fc9104cb760e6d462d2e0a768cbbf0d95968012fa refs/heads/2020Q4
003f1b85cb72f7defaa7d2fba7c39a48841996a4ec5c refs/heads/2021Q1
003d4010f7bbc03638d71781ce091bf40a0907fa12fe refs/heads/main
0000

gitup: get_commit_details: refs/heads/master doesn't exist in /freebsd/freebsd-ports.git: Invalid argument
root@mowa219-gjp4-8570p:~ #
```

Things are not yet ready for the forward-looking change that I suggested at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254760#c0 but I guess that I should (at least) change the default to reflect the non-existence of master.


----------



## zirias@ (Apr 5, 2021)

Check your gitup config. See what the server offers:
`00504010f7bbc03638d71781ce091bf40a0907fa12fe HEAD symref-target:refs/heads/main`
vs what you're requesting:
`gitup: get_commit_details: refs/heads/master doesn't exist […]`

Obviously, there's no branch named "master" and the default branch is named "main". This seems to be a new default on Github, for "reasons"…

Or, just use the official repo on git.freebsd.org instead of the Github mirror.


----------



## grahamperrin@ (Apr 5, 2021)

Zirias said:


> … the default branch is named "main". …


Yep, fixed a couple of days ago: https://lists.freebsd.org/pipermail/freebsd-git/2021-April/000849.html

Here, there was a wait of around ten minutes before the first line of output:


```
root@mowa219-gjp4-8570p:~ # time gitup ports -v 1
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd-ports.git
# Target: /usr/ports
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: main
# Action: clone
 * /usr/ports/CHANGES
…
 * /usr/ports/x11/zenity/Makefile
 - /usr/ports/.portsnap.INDEX
 - /usr/ports/INDEX-14
 - /usr/ports/net/gitup/dialog4ports.core
 - /usr/ports/net/gitup/work
…
 - /usr/ports/net/gitup/work/stage/usr/local/www
10.079u 8.477s 10:50.11 2.8%    49+357k 106059+109255io 0pf+0w
root@mowa219-gjp4-8570p:~ #
```

I imagine things becoming faster after https://git.freebsd.org/ports.git begins to redirect.


----------



## zirias@ (Apr 5, 2021)

grahamperrin said:


> Yep, fixed a couple of days ago: https://lists.freebsd.org/pipermail/freebsd-git/2021-April/000849.html


_Completely_ unrelated and different issue.



grahamperrin said:


> Here, there was a wait of around ten minutes before the first line of output:


Output suggests it did a "clone". This is expected to take considerable time, given a repository that large. It should be _a lot_ faster for consecutive runs, doing "pull" instead.

"Fishy" things in your output: A portsnap Index? a coredump from dialog4ports? a workdir from building gitup? Looks like you tried it on an existing ports tree. Initial "clone" _might_ be faster on a clean directory.



grahamperrin said:


> I imagine things becoming faster after https://git.freebsd.org/ports.git begins to redirect.


Redirect what? From your output, I can see gitup is configured to use the mirror on Github, not the official repo on git.freebsd.org.


----------



## grahamperrin@ (Apr 6, 2021)

Zirias said:


> … Redirect what? From your output, I can see gitup is configured to use the mirror on Github, not the official repo on git.freebsd.org.


Configured to use a GitHub URL because https://git.freebsd.org/ports.git is still *404, not found* (no surprise, during the transition).



Zirias said:


> … use the official repo on git.freebsd.org instead of the Github mirror.


https://git.freebsd.org/docs.git redirects to https://cgit.freebsd.org/doc where https://git.FreeBSD.org/doc.git is the primary non-developer address for cloning. I ignore non-working links such as (public-mirror).

https://git.freebsd.org/src.git redirects to https://cgit.freebsd.org/src where https://git.FreeBSD.org/src.git is the primary non-developer address for cloning. 

As the transition progresses, I expect https://git.freebsd.org/ports.git to begin redirecting to https://cgit.freebsd.org/ports …


----------



## grahamperrin@ (Apr 6, 2021)

Zirias said:


> … on a clean directory. …




```
root@mowa219-gjp4-8570p:~ # time rm -r /usr/ports/*
0.574u 7.715s 2:52.24 4.8%      9+161k 14098+0io 0pf+0w
root@mowa219-gjp4-8570p:~ # rm -r /usr/ports/.git*
root@mowa219-gjp4-8570p:~ # ls -adhl /usr/ports/.*
drwxr-xr-x   2 root  wheel     3B Apr  6 02:51 /usr/ports/.
drwxr-xr-x  17 root  wheel    17B Jan  3 07:57 /usr/ports/..
-rw-r--r--   1 root  wheel   115B Nov 23 04:40 /usr/ports/.arcconfig
root@mowa219-gjp4-8570p:~ # cat /usr/ports/.arcconfig
{
        "repository.callsign" : "P",
        "phabricator.uri" : "https://reviews.freebsd.org/",
        "history.immutable" : true
}
root@mowa219-gjp4-8570p:~ # rm /usr/ports/.arcconfig
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd-ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: main
! /usr/ports/.arcconfig is missing.
! /usr/ports/.gitattributes is missing.
! /usr/ports/.gitauthors is missing.
! /usr/ports/.gitignore is missing.
! /usr/ports/.gitmessage is missing.
! /usr/ports/CHANGES is missing.
…
! /usr/ports/x11/zenity/pkg-plist is missing.
gitup: build_repair_command: There are too many files to repair -- please re-clone the repository: Argument list too long
0.663u 0.195s 0:05.18 16.4%     51+169k 108+0io 0pf+0w
root@mowa219-gjp4-8570p:~ # time gitup ports -c
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd-ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: main
0.388u 0.166s 0:01.05 51.4%     51+168k 1+1io 0pf+0w
root@mowa219-gjp4-8570p:~ # df -h /usr/ports
Filesystem              Size    Used   Avail Capacity  Mounted on
copperbowl/usr/ports    190G    104K    190G     0%    /usr/ports
root@mowa219-gjp4-8570p:~ # time gitup ports -c
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd-ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: main
0.429u 0.127s 0:00.98 55.1%     50+166k 0+1io 0pf+0w
root@mowa219-gjp4-8570p:~ # df -h /usr/ports
Filesystem              Size    Used   Avail Capacity  Mounted on
copperbowl/usr/ports    190G    104K    190G     0%    /usr/ports
root@mowa219-gjp4-8570p:~ # time rm -r /usr/ports/*
rm: No match.
0.000u 0.000s 0:00.00 0.0%      0+0k 0+0io 0pf+0w
root@mowa219-gjp4-8570p:~ # rm -r /usr/ports/.git*
root@mowa219-gjp4-8570p:~ # ls -adhl /usr/ports/.*
drwxr-xr-x   2 root  wheel     2B Apr  6 02:58 /usr/ports/.
drwxr-xr-x  17 root  wheel    17B Jan  3 07:57 /usr/ports/..
root@mowa219-gjp4-8570p:~ # time gitup ports -c
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd-ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: main
0.415u 0.127s 0:01.22 43.4%     50+166k 0+1io 0pf+0w
root@mowa219-gjp4-8570p:~ #
```


----------



## grahamperrin@ (Apr 6, 2021)

… worked around by removing /usr/ports/.gituprevision and the database for gitup(1), after which an initial run of `gitup ports` with an empty /usr/ports/ completed relatively quickly.

The second run took *much* longer. Timed:
`6.873u 11.903s 36:15.25 0.8%    50+165k 141126+1io 0pf+0w`
– I'm not complaining. YMMV, depending on hardware, time of day and so on.


```
root@mowa219-gjp4-8570p:~ # ls -hl /var/db/gitup
total 9725
-rw-r--r--  1 root  wheel    13M Apr  5 20:42 ports
root@mowa219-gjp4-8570p:~ # rm /var/db/gitup/ports
root@mowa219-gjp4-8570p:~ # ls -adhl /usr/ports/.*
drwxr-xr-x   2 root  wheel     3B Apr  6 02:59 /usr/ports/.
drwxr-xr-x  17 root  wheel    17B Jan  3 07:57 /usr/ports/..
-rw-r--r--   1 root  wheel    14B Apr  6 02:59 /usr/ports/.gituprevision
root@mowa219-gjp4-8570p:~ # rm /usr/ports/.gituprevision
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd-ports.git
# Target: /usr/ports
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: main
# Action: clone
 + /usr/ports/.arcconfig
 + /usr/ports/.gitattributes
 + /usr/ports/.gitauthors
 + /usr/ports/.gitignore
 + /usr/ports/.gitmessage
 + /usr/ports/CHANGES
…
 + /usr/ports/x11/zenity/pkg-plist
6.593u 13.397s 2:02.72 16.2%    49+3329k 761+217601io 0pf+0w
root@mowa219-gjp4-8570p:~ # df -h /usr/ports
Filesystem              Size    Used   Avail Capacity  Mounted on
copperbowl/usr/ports    190G    670M    190G     0%    /usr/ports
root@mowa219-gjp4-8570p:~ # time gitup ports
load: 0.32  cmd: gitup 9088 [zio->io_cv] 995.65r 2.95u 4.57s 0% 280632k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8
load: 0.43  cmd: gitup 9088 [zio->io_cv] 1213.15r 3.31u 5.26s 0% 280648k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8
load: 0.30  cmd: gitup 9088 [zio->io_cv] 1979.81r 5.19u 9.06s 0% 325832k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd-ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: main
6.873u 11.903s 36:15.25 0.8%    50+165k 141126+1io 0pf+0w
root@mowa219-gjp4-8570p:~ #
```


----------



## zirias@ (Apr 6, 2021)

Now, the tense of the title should be correct 





__





						ports - FreeBSD ports tree
					






					cgit.freebsd.org


----------



## grahamperrin@ (Apr 6, 2021)

grahamperrin said:


> … second run took *much* longer. Timed:
> `6.873u 11.903s 36:15.25 0.8% 50+165k 141126+1io 0pf+0w`
> – I'm not complaining. YMMV, depending on hardware, time of day and so on. …



Third run of `gitup ports`: quicker. Around thirteen minutes:
`5.910u 8.047s 12:38.38 1.8%     50+166k 143029+1io 0pf+0w`

In due course, when things are complete (I shouldn't treat https://wiki.freebsd.org/git#Ports_Schedule as fully comprehensive) I'll test with https://git.freebsd.org/ports.git instead of GitHub. For my use case:

I don't expect a significant difference in speed from a change of source
I do expect git(1) and portsnap(8) to be faster than gitup(1).


----------



## zirias@ (Apr 6, 2021)

The transision *is* complete, there are already a few commits updating ports. I'm right now cloning the ports tree on my builder machine…

```
builder# git clone https://git.freebsd.org/ports.git .
Cloning into '.'...
remote: Enumerating objects: 991, done.
remote: Counting objects: 100% (991/991), done.
remote: Compressing objects: 100% (175/175), done.
Receiving objects:  55% (2710986/4907881), 468.39 MiB | 6.63 MiB/s
```

What's missing from the schedule is just getting the sync to mirrors (like Gitup) up and running again.

edit: Done, and I love it – finally some structure for my local changes on top of origin/HEAD:

```
706ae52ff248 (HEAD -> local) security/stunnel: downgrade to 5.48
aa07aaa1ce22 emulators/virtualbox-ose: fix build with libressl
17bec063cb3e sysutils/gkrellm2: fix build with libressl
8dd6367e86c1 www/chromium: fix build with MIT krb5
cf44ad3f0ed7 sysutils/pam_mount: fix build with libressl
3858ab6b0b10 security/openvpn: fix build with libressl
a465c8a99ee4 graphics/gtkam: fix dependency
f8229dc6cc57 add micropolis-fbsd
868ae2421595 add spindle
b602ce4d15ea add exomizer2
7162fcf86480 add mkd64
05017885ce98 (origin/main, origin/HEAD, main) devel/awscli: Update 1.19.36 -> 1.19.44
```


----------



## grahamperrin@ (Apr 6, 2021)

From a gitup perspective, "Transition almost done …".


----------



## zirias@ (Apr 6, 2021)

Again: No. Pushing to mirrors is nothing "gitup" would need. Just configure the official repository, it's there, it's actively used, updates coming in pretty fast right now*. If you have an old config, use this one instead: https://github.com/johnmehr/gitup/blob/main/gitup.conf

---
* that's the log since commits are possible (only two hours so far):

```
848355ff18d2 graphics/mesa-devel: update to 21.0.b.4158
1e8c41d07054 emulators/citra: update to s20210403
a67b6931f198 emulators/yuzu: update to s20210404
6ce48c1cfb86 emulators/rpcs3: update to 0.0.15.12048
60a8b261e7f6 x11-servers/xwayland-devel: update to 1.20.0.886
4c47c324b2d7 www/py-flask-restx: update to 0.3.0
ad7765865cfe www/gallery-dl: update to 1.17.2
f2aa5c86adf0 www/youtube_dl: update to 2021.04.01
72f7b59ccdc8 x11/wezterm: update to 20210405.110924.a5.b5.b8
0751da285384 devel/rust-bindgen: update to 0.58.0
5a89dd6e7c23 devel/cargo-c: update to 0.8.0
4cc8ae097fc4 multimedia/mpv: update to 0.33.1
f8732455e4bb multimedia/rav1e: update to 0.4.1
cc87022694a6 graphics/libplacebo: update to 3.120.0
0f984aa02a5b lang/intel-compute-runtime: update to 21.13.19438
6cc47c27a695 multimedia/libva-utils: update to 2.11.1
ea90aab81688 multimedia/intel-media-sdk: update to 21.1.3
09d06e7228aa multimedia/libva-intel-media-driver: update to 21.1.3
8c16874b817f multimedia/gmmlib: update to 21.1.1
d34abcb3e18f devel/git-cinnabar: update to 0.5.7
f59286f56314 x11-wm/sway: unbreak fetch
722e4c5e303e x11-toolkits/wlroots: unbreak fetch
5254fca65db2 Chase GH_ACCOUNT rename for Greg V
1f090b794c9b audio/ardour6: drop bsd.port.pre.mk
021faaac0be6 audio/ardour6: unbreak on aarch64
f990e2b38bff devel/cargo-c: drop libgit2 < 1.1 workaround after e7d94d42f82e
b1a2d52166ab Document gitlab-ce vulnerabilities.
1532b5be3796 devel/rust-analyzer: Update to 2021-04-05
9b79896e8f15 Major upgrade to 13.10 which includes security related upgrade to 13.10.1. Changelog: https://about.gitlab.com/releases/2021/03/22/gitlab-13-10-released/ https://about.gitlab.com/releases/2021/03/31/security-release-gitlab-13-10-1-released/
c0b6ad5b5016 Update to 13.10.1 which is required for gitlab-ce 13.10.
c2acab3b6bbe Update to 1.36.0 which is required for gitlab-ce 13.10.
46c5af56a39e Update to 16.10.7 which is required for gitlab-ce 13.10.
462dbbcdec08 Update to 16.11.7 which is required for gitlab-ce 13.10. Changelog: https://github.com/chef/chef/blob/master/RELEASE_NOTES.md
6bb00d2ad2d0 Fix dependencies to work with chef upgrade.
d0407b2937ab Update to 16.11.7. This update is required for a security related update of gitlab-ce to 13.10.1. PR:              254548
0b4d6a9a3de1 Update to 1.1.0 which is required for gitlab-ce 13.10.
4d8acc04bf20 Update to 1.7.1 which is required for gitlab-ce 13.10.
43dfa5f733d1 Update to 1.0.1 which is required for gitlab-ce 13.10.
8702c15bcb25 Update to 3.9.0. This update is required for a security update of www/gitlab-ce.
5c0415a6535d Update to 0.3.10. This update is required for a security related upgrade of gitlab-ce.
42d771f692bb Update to 0.0.9 which is required for gitlab-ce 13.10.
9f48b4070467 Update to 1.6.0 which is required for gitlab-ce 13.10.
11be3d86f769 Update to 1.11.8 which is required for gitlab-ce 13.10.
68e3cf9815ff Update to 0.5.5 which is required for gitlab-ce 13.10.
6c5e5b3b7c8f Update to 1.3.1 which is required for gitlab-ce 13.10.
8c8badcc0838 Update to 0.16.2 which is required for gitlab-ce 13.10.
3fba248f462e Update to 0.5.1 which is required for gitlab-ce 13.10.
4376e187f101 Update to 13.10.1 which is required for gitlab-ce 13.10.
b6ed29de5bba Added new port which is required for gitlab-ce 13.10.
e7d94d42f82e Update libgit2 to 1.1.0. This update is also required for www/gitlab-ce 13.10 upgrade.
634a88402a52 Update to Version 4.0.0
6bf14cb7b01b devel/awscli: Update 1.19.44 -> 1.19.45
2b488206169f devel/py-botocore: Update 1.20.44 -> 1.20.45
11b77b717cb1 security/openssl-unsafe: Unbreak with FreeBSD 13
05017885ce98 devel/awscli: Update 1.19.36 -> 1.19.44
3ba3a94c3578 devel/py-botocore: Update 1.20.36 -> 1.20.44
dda06a776cea www/node: Update 15.12.0 -> 15.13.0
a26284996496 www/node12: Update 12.21.0 -> 12.22.0
8c7fa30e3e42 www/node10: Mark as deprecated and set expiration date
c8481a0e21c4 net/py-python-socks: Update to 1.2.4
785f39d86108 editors/emacs-devel: Update to 2021-04-05 commit, 0342354
b9ad876d99f0 devel/py-Js2Py: update to 0.71
a663968e406b audio/fasttracker2: Update to 1.46
ed8d3eda309d Mark the repository has been converted to Git
```


----------



## grahamperrin@ (Apr 6, 2021)

Don't shoot the messenger …


----------



## zirias@ (Apr 6, 2021)

I'm not sure what the problem is here. The transition is complete. Pushing changes to mirrors on Github, Gitlab etc is _entirely_ irrelevant for using the official repo. net/gitup will work just fine with it.


----------



## grahamperrin@ (Apr 6, 2021)

The quote "Transition almost done …" came from a gitup area around an hour ago; please don't shoot the messenger.



grahamperrin said:


> I don't expect a significant difference in speed from a change of source



True. Around sixteen minutes:


```
root@mowa219-gjp4-8570p:~ # grep repository /usr/local/etc/gitup.conf
                "repository" : "/ports.git",
                "repository" : "/ports.git",
                "repository" : "/src.git",
                "repository" : "/src.git",
                "repository" : "/src.git",
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 848355ff18d260cf954ca13c37f4d4d266ee6d0e
# Branch: main
# Action: pull
 * /usr/ports/README
…
 - /usr/ports/www/node/files/patch-tools_genv8constants.py
7.677u 9.738s 15:57.54 1.8%     51+398k 143400+78033io 1pf+0w
root@mowa219-gjp4-8570p:~ #
```


----------



## zirias@ (Apr 6, 2021)

grahamperrin said:


> The quote "Transition almost done …" came from a gitup area around an hour ago; please don't shoot the messenger.


This doesn't make sense, gitup won't tell you anything about a "transition". The repository is now available for a while, and since then, gitup will work correctly on it.



grahamperrin said:


> True. Around sixteen minutes:
> 
> ```
> 7.677u 9.738s 15:57.54 1.8%     51+398k 143400+78033io 1pf+0w
> ```


Trying to replicate what you did (so starting from commit `4010f7bbc03638d71781ce091bf40a0907fa12fe` on an empty /usr/ports, then updating…), I can't reproduce any slowness:
(1 minute 15 seconds for initial clone, 11.4 seconds for updating from `4010f7bbc03638d71781ce091bf40a0907fa12fe`)

```
nexus# time gitup -w 4010f7bbc03638d71781ce091bf40a0907fa12fe ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: (detached)
# Action: clone
[...]
 + /usr/ports/x11/zenity/distinfo
 + /usr/ports/x11/zenity/pkg-descr
 + /usr/ports/x11/zenity/pkg-plist
gitup -w 4010f7bbc03638d71781ce091bf40a0907fa12fe ports  10,74s user 13,68s system 32% cpu 1:15,12 total
nexus# time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 7444131cd19a49d73952e741999e08e7dc5905c6
# Branch: main
# Action: pull
[...]
 - /usr/ports/www/chromium/files/patch-third__party_zlib_BUILD.gn
 - /usr/ports/www/chromium/files/patch-ui_base_data__transfer__policy_data__transfer__endpoint.h
 - /usr/ports/www/chromium/files/patch-weblayer_browser_content__browser__client__impl.h
 - /usr/ports/www/node/files/patch-tools_genv8constants.py
gitup ports  4,44s user 4,30s system 76% cpu 11,444 total
```
It seems you have to troubleshoot this locally. Bad network connectivity?

*edit:* Are you running this on `-CURRENT`? Do you use a `GENERIC` kernel? What about `MALLOC_PRODUCTION`? If you have all kinds of debugging stuff enabled, catastrophic performance is expected.


----------



## grahamperrin@ (Apr 6, 2021)

grahamperrin said:


> … I'm not complaining. YMMV, depending on hardware, time of day and so on.





Zirias said:


> … catastrophic performance …



Really, I'm not complaining. 

Slowness in my case is not network-related. Hardware is a factor; and the slowest run loosely coincided with a time of day when slowness was to be expected. I'm still testing.


----------



## grahamperrin@ (Apr 6, 2021)

grahamperrin said:


> For my use case:
> 
> … I do expect git(1) and portsnap(8) to be faster than gitup(1).



True, at least for git(1). Abridged:


```
root@mowa219-gjp4-8570p:~ # time git -C /usr/ports pull --ff-only
remote: Enumerating objects: 292, done.
remote: Counting objects: 100% (292/292), done.
remote: Compressing objects: 100% (120/120), done.
remote: Total 212 (delta 149), reused 113 (delta 92), pack-reused 0
Receiving objects: 100% (212/212), 45.30 KiB | 6.47 MiB/s, done.
Resolving deltas: 100% (149/149), completed with 49 local objects.
From https://git.freebsd.org/ports
   7444131cd19a..4b13a4c45a26  main       -> freebsd/main
Updating 7444131cd19a..4b13a4c45a26
Fast-forward
 .gitignore                                                                                                                       |   2 +-
 audio/schismtracker/Makefile                                                                                                     |   2 +-
…
 create mode 100644 databases/mongodb49/pkg-plist
0.835u 1.863s 0:30.81 8.7%      2965+762k 8073+403io 364pf+0w
root@mowa219-gjp4-8570p:~ #
```

– around thirty seconds.


----------



## zirias@ (Apr 6, 2021)

So, _do_ you use -CURRENT with all the debugging stuff enabled? If so, that's most likely the "problem" with gitup. Even 30 seconds is a lot for a pull…


----------



## grahamperrin@ (Apr 6, 2021)

grahamperrin said:


> … I'm still testing.





Zirias said:


> So, _do_ you use -CURRENT with all the debugging stuff enabled? …



No. 

Please be patient. I'm still testing. 



Zirias said:


> This doesn't make sense, gitup won't tell



I quoted a person in a gitup area. It wasn't a quote from gitup.


----------



## grahamperrin@ (Apr 6, 2021)

From IRC:


> First run of `gitup ports`: around forty-six seconds. Second run: around twelve minutes. Those were with file system compression relaxed, to zstd. I don't perceive a problem with -CURRENT.



More generally, emphatically:

*I do not perceive a problem*
– I simply do not expect gitup(1) to be as fast as git(1).

https://github.com/johnmehr/gitup/blob/2b055c47f4e5f703c7ae81ec5da5e6c41ed0ef6c/gitup.1.in#L129 explains that gitup(1) "… relies on the known remote files lists stored in /var/db/gitup and the current state of the local repository to reconstruct data …".

I know very little about git(1) but I assume that it takes a more expeditious approach to determining the state of the local repository; no need to _reconstruct data_.


```
root@mowa219-gjp4-8570p:~ # date ; uptime
Tue Apr  6 11:25:41 BST 2021
11:25AM  up 15:50, 5 users, load averages: 1.58, 1.83, 2.18
root@mowa219-gjp4-8570p:~ # df -h /usr/ports
Filesystem              Size    Used   Avail Capacity  Mounted on
copperbowl/usr/ports    190G    1.5G    189G     1%    /usr/ports
root@mowa219-gjp4-8570p:~ # zfs set compression=zstd copperbowl/usr/ports
root@mowa219-gjp4-8570p:~ # time rm -r /usr/ports/*
0.669u 7.811s 6:31.69 2.1%      10+169k 14121+0io 0pf+0w
root@mowa219-gjp4-8570p:~ # df -h /usr/ports
Filesystem              Size    Used   Avail Capacity  Mounted on
copperbowl/usr/ports    190G    847M    190G     0%    /usr/ports
root@mowa219-gjp4-8570p:~ # time du -hs /usr/ports
847M    /usr/ports
0.002u 0.000s 0:00.20 0.0%      0+0k 7+0io 1pf+0w
root@mowa219-gjp4-8570p:~ # rm -r /usr/ports/.git*
root@mowa219-gjp4-8570p:~ # ls -adhl /usr/ports/.*
drwxr-xr-x   2 root  wheel     3B Apr  6 11:41 /usr/ports/.
drwxr-xr-x  17 root  wheel    17B Jan  3 07:57 /usr/ports/..
-rw-r--r--   1 root  wheel   115B Apr  6 10:53 /usr/ports/.arcconfig
root@mowa219-gjp4-8570p:~ # rm /usr/ports/.arcconfig
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 848355ff18d260cf954ca13c37f4d4d266ee6d0e
# Want: 570a7bea9906e581648cea3e52c7f491da93d5ee
# Branch: main
 ! /usr/ports/.arcconfig is missing.
 ! /usr/ports/.gitattributes is missing.
 ! /usr/ports/.gitauthors is missing.
 ! /usr/ports/.gitignore is missing.
 ! /usr/ports/.gitmessage is missing.
 ! /usr/ports/CHANGES is missing.
…
 ! /usr/ports/x11/zenity/pkg-plist is missing.
gitup: build_repair_command: There are too many files to repair -- please re-clone the repository: Argument list too long
0.786u 0.224s 0:06.30 15.8%     53+175k 113+0io 6pf+0w
root@mowa219-gjp4-8570p:~ # ls -adhl /usr/ports/.*
drwxr-xr-x   2 root  wheel     2B Apr  6 11:41 /usr/ports/.
drwxr-xr-x  17 root  wheel    17B Jan  3 07:57 /usr/ports/..
root@mowa219-gjp4-8570p:~ # rm /var/db/gitup
rm: /var/db/gitup: is a directory
root@mowa219-gjp4-8570p:~ # rm /var/db/gitup/ports
root@mowa219-gjp4-8570p:~ # ls -hl /var/db/gitup
total 0
root@mowa219-gjp4-8570p:~ # time gitup ports
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: 570a7bea9906e581648cea3e52c7f491da93d5ee
# Branch: main
# Action: clone
 + /usr/ports/.arcconfig
 + /usr/ports/.gitattributes
 + /usr/ports/.gitauthors
 + /usr/ports/.gitignore
 + /usr/ports/.gitmessage
 + /usr/ports/CHANGES
…
 + /usr/ports/x11/zenity/pkg-plist
7.054u 12.562s 0:45.50 43.0%    50+3166k 142+217743io 0pf+0w
root@mowa219-gjp4-8570p:~ # time gitup ports
load: 1.72  cmd: gitup 28070 [zio->io_cv] 283.53r 2.56u 3.55s 2% 281096k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
load: 1.89  cmd: gitup 28070 [zio->io_cv] 693.76r 5.34u 8.63s 3% 326260k
mi_switch+0xc1 sleepq_timedwait+0x2f _cv_timedwait_sbt+0x107 zio_wait+0x2f9 dmu_buf_hold_array_by_dnode+0x29d dmu_read_uio_dnode+0x3a dmu_read_uio_dbuf+0x3b zfs_read+0x1da zfs_freebsd_read+0x44 VOP_READ_APV+0x1f vn_read+0x1ed vn_io_fault_doio+0x43 vn_io_fault1+0x15c vn_io_fault+0x1a4 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8 
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 570a7bea9906e581648cea3e52c7f491da93d5ee
# Want: 64d94165652f40041bf71bdf7a775f867e7b80a9
# Branch: main
# Action: pull
 * /usr/ports/security/openssl/Makefile
7.403u 10.415s 12:30.76 2.3%    50+165k 144157+77802io 0pf+0w
root@mowa219-gjp4-8570p:~ #
```

If you like, take a holistic view of https://bsd-hardware.info/?probe=d9ec051372 … if not holistic, then please at least be aware of NODEBUG.


----------



## zirias@ (Apr 6, 2021)

You can assume a lot, but there's a reason gitup keeps a dabatase.

A much more likely scenario is that extensive use of malloc(3) makes gitup slow on a -CURRENT userland. Debugging options don't exist _only_ in the kernel, see for example





__





						src.opts.mk « mk « share - src - FreeBSD source tree
					






					cgit.freebsd.org
				



vs




__





						src.opts.mk « mk « share - src - FreeBSD source tree
					






					cgit.freebsd.org
				




There might be other reasons as well. Try on a -RELEASE first.


----------



## a6h (Apr 6, 2021)

grahamperrin :
I've asked it on IRC, but for the record, I'll ask it here too: Are you on main branch, or on tag 0.90? gitup.conf has changed since tag 0.90.


----------



## grahamperrin@ (Apr 6, 2021)

Thanks, I'm aware of the changes; please see 254760 under https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504243

In IRC when I mentioned using packaged gitup i.e. <https://www.freshports.org/net/gitup/#packages>, I forgot my recent build (sorry):


```
% pkg query '%o %v %R' gitup

net/gitup 0.90_1 unknown-repository
%
```


----------



## grahamperrin@ (Apr 6, 2021)

With an unnaturally clean installation of FreeBSD 12.2-RELEASE-p4, and the local repository up-to-date: 

_git_ pull took less than one second
pulls with _gitup_ took between eighty-five and one hundred and twenty-five seconds.


----------



## Jose (Apr 6, 2021)

scottro said:


> I'm not sure.  Seems like it would be a good idea. chrcol, looking at my /usr/local/etc/gitup.conf, I have this for quarterly, but don't know if it is correct...
> 
> ```
> "quarterly" : {
> ...


That is not correct. There is no generic "quarterly" branch. Each individual quarterly branch gets a unique name in the form _yyyy_Q_[1-4]_. The latest quarterly branch is 2021Q1. There is no Q2 branch yet. It's part of the migration:


			Git - FreeBSD Wiki
		




grahamperrin said:


> https://git.freebsd.org/docs.git redirects to https://cgit.freebsd.org/doc where https://git.FreeBSD.org/doc.git is the primary non-developer address for cloning. I ignore non-working links such as (public-mirror).
> 
> https://git.freebsd.org/src.git redirects to https://cgit.freebsd.org/src where https://git.FreeBSD.org/src.git is the primary non-developer address for cloning.
> 
> As the transition progresses, I expect https://git.freebsd.org/ports.git to begin redirecting to https://cgit.freebsd.org/ports …


Not quite. The git.freebsd.org URLs are the canonical URLs used for cloning repositories. The cgit.freenbsd.org URLs are for the *Web interface* to the Git repositories. They are used to browse the source online and are not to be used for cloning (won't work anyway). See https://github.com/lwhsu/freebsd-git-docs/blob/main/URLs.md



grahamperrin said:


> With an unnaturally clean installation of FreeBSD 12.2-RELEASE-p4, and the local repository up-to-date:
> 
> _git_ pull took less than one second
> pulls with _gitup_ took between eighty-five and one hundred and twenty-five seconds.


More information needed. A `git pull` does very little if your local clone is up-to-date with the remote. What are the exact commands you are running? But I suggest you take this subject to its own topic. Jmehr has been very responsive on these forums in the recent past.


----------



## suntzu00 (Apr 6, 2021)

if you're using poudriere and you want to follow 'main'(replace it with the branch name if you're interested in following quarterly) :

`poudriere ports -c -B main -m git -U 'https://git.FreeBSD.org/ports.git' -p MAIN`

maybe someone will need this


----------



## grahamperrin@ (Apr 6, 2021)

Jose said:


> … What are the exact commands you are running?



For gitup: essentially, `gitup ports` – nothing complicated.

For git: as outlined in the guide that I linked from https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504066 on page 2. Again, nothing complicated.



> But I suggest you take this subject to its own topic. …



+1 

– if anyone else would like to do so. I don't have a problem with the test results, other people are welcome to use the results as points of reference in a new topic.


----------



## zirias@ (Apr 6, 2021)

grahamperrin even if you're fine with it, they _do_ show a problem though, gitup shouldn't be slower than git (at least not substantially). But as long as nobody else finds similar problems, there will probably be no thread then


----------



## grahamperrin@ (Apr 6, 2021)

Zirias with respect, we seem to be going round in circles here.


Zirias said:


> … a reason gitup keeps a dabatase. …


– the preceding post did link to a reason. 

Really, if you perceive a problem, please start a new topic. Thanks.


----------



## zirias@ (Apr 6, 2021)

If there's one thing I don't see, it's respect. No, there wasn't any reason, just assumptions that are pretty unlikely. A lot of people are trying gitup, you're the only one so far reporting it to be a LOT slower than git. There is a problem for sure. If you don't have any interest in having it fixed, that's fine. But don't claim it "works that way", that's nonsense.


----------



## Mayhem30 (Apr 6, 2021)

On my slowest machine that has a Intel i3 CPU, 8GB ddr2 ram and 1TB HDD it takes 23 seconds for `gitup ports` to complete.

I'm not sure how that compares to git .. but I'm happy with those numbers, so I'm going to stick with it.


----------



## grahamperrin@ (Apr 6, 2021)

suntzu00 said:


> `poudriere ports -c -B main -m git -U 'https://git.FreeBSD.org/ports.git' -p MAIN`


Thanks, I needed that.

With poudriere-ports(8) as a starting point, how would a person discover the use of option `-U` for the git method?

I chose to delete then recreate my _default_ ports tree, encountered a bug that prevented me from destroying the filesystem for the deleted tree. Worked around by killing gvfsd-trash, which is not necessarily recommended but it suits me.

For reference only (not seeking help with this):


```
root@mowa219-gjp4-8570p:~ # poudriere ports -c -B main -m git -U 'https://git.freebsd.org/ports.git'
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default...cannot create 'copperbowl/poudriere/ports/default': dataset already exists
fail
[00:00:00] Error: Failed to create FS copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # zfs destroy copperbowl/poudriere/ports/default
cannot unmount '/copperbowl/poudriere/ports/default': pool or dataset is busy
root@mowa219-gjp4-8570p:~ # poudriere ports -l
PORTSTREE   METHOD TIMESTAMP PATH
portoverlay -                /usr/local/poudriere/ports/portoverlay
root@mowa219-gjp4-8570p:~ # killall gvfsd-trash
root@mowa219-gjp4-8570p:~ # zfs destroy copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # poudriere ports -c -B main -m git
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default... done
[00:00:00] Cloning the ports tree... done
root@mowa219-gjp4-8570p:~ # cat /usr/local/poudriere/ports/default/.git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = git://github.com/freebsd/freebsd-ports.git
        fetch = +refs/heads/main:refs/remotes/origin/main
[branch "main"]
        remote = origin
        merge = refs/heads/main
root@mowa219-gjp4-8570p:~ # poudriere ports -d default
[00:00:00] Are you sure you want to delete the ports tree default at /usr/local/poudriere/ports/default? [y/N] y
[00:00:02] Deleting portstree "default"... done
root@mowa219-gjp4-8570p:~ # poudriere ports -c -B main -m git -U 'https://git.freebsd.org/ports.git'
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default...cannot create 'copperbowl/poudriere/ports/default': dataset already exists
fail
[00:00:00] Error: Failed to create FS copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # killall gvfsd-trash
No matching processes were found
root@mowa219-gjp4-8570p:~ # zfs destroy copperbowl/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # rm -r /usr/local/poudriere/ports/default
root@mowa219-gjp4-8570p:~ # time poudriere ports -c -B main -m git -U 'https://git.freebsd.org/ports.git'
[00:00:00] Creating default fs at /usr/local/poudriere/ports/default... done
[00:00:00] Cloning the ports tree... done
11.174u 7.791s 1:05.23 29.0%    2879+737k 3024+170812io 1095pf+0w
root@mowa219-gjp4-8570p:~ # cat /usr/local/poudriere/ports/default/.git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = https://git.freebsd.org/ports.git
        fetch = +refs/heads/main:refs/remotes/origin/main
[branch "main"]
        remote = origin
        merge = refs/heads/main
root@mowa219-gjp4-8570p:~ # pkg query '%o %v %R' poudriere-devel
ports-mgmt/poudriere-devel 3.3.99.20210303_1 FreeBSD
root@mowa219-gjp4-8570p:~ #
```

(I was curious about how /usr/local/poudriere/ports/default/.git/config would differ without option `-U`.)


----------



## grahamperrin@ (Apr 6, 2021)

Zirias said:


> If you don't have any interest in having it fixed,



If you must go round in circles, please don't put words in my mouth.



Zirias said:


> there wasn't any reason, just assumptions





grahamperrin said:


> https://github.com/johnmehr/gitup/blob/2b055c47f4e5f703c7ae81ec5da5e6c41ed0ef6c/gitup.1.in#L129 explains that gitup(1) "… relies on the known remote files lists stored in /var/db/gitup and the current state of the local repository to reconstruct data …".



That was a reference to the manual page. Not an assumption.


----------



## zirias@ (Apr 6, 2021)

So, stop playing silly games now. The manual page has nothing to say about this taking time. That's *your* assumption, although it contradicts the experience of many who see gitup working pretty fast.


----------



## mickey (Apr 6, 2021)

Can someone please explain what exactly the _--ff-only_ flag on `git pull` does? Everything I've read so far was a bit shallow, to say the least. And if `git pull --ff-only`  is the recommended way of updating the tree, why isn't it used in /usr/ports/Makefile:

```
.  else
        @echo "--------------------------------------------------------------"
        @echo ">>> Updating ${.CURDIR} from git repository"
        @echo "--------------------------------------------------------------"
        cd ${.CURDIR}; ${GIT} pull
.  endif
```
Also I saw instructions mentioning the use of _-o freebsd_ in the clone command, while others omitted this flag. What difference does it make?


----------



## zirias@ (Apr 6, 2021)

`--ff-only` will outright _refuse_ to create a merge commit, which would be necessary if you locally added commits to the same branch. Adding commits to a branch you will never push is certainly a bad idea (if you need local commits, use a separate local branch for them, you can rebase it) – but `--ff-only` is just a safety measure here: Fail fast instead of writing a local history that diverges from the upstream branch more and more.

IOW, ignore it if you know you handle your local repository correctly


----------



## a6h (Apr 6, 2021)

mickey said:


> Can someone please explain what exactly the _--ff-only_ flag on `git pull` does?


Thread help-with-migration-to-git.79365/#post-500223


----------



## mickey (Apr 6, 2021)

Zirias said:


> IOW, ignore it if you know you handle your local repository correctly


I only have a handful of additional patch files (read: untracked files) in my ports tree, that git is not aware of. For the time being I'd like to keep it that way and I guess that shouldn't be a problem then?


----------



## richardtoohey2 (Apr 6, 2021)

Thanks everyone for this thread - the to-ing and fro-ing has helped to explain why some things are they way they are.

My old ports flow (as a ports user rather than developer):
	
	



```
portsnap fetch update
pkg version -vIL=
```
Before starting with the new flow, I built net/gitup, replaced the /usr/local/etc/gitup.conf with the one referenced in this thread, mv'd /usr/ports to /usr/old-ports and made a new /usr/ports directory and started in there with the new flow:
	
	



```
gitup ports
make fetchindex
pkg version -vIL=
```
(And then I use _whisper_ portm**ter)

I didn't time the original gitup ports - was very _roughly_ a minute. All this on FreeBSD 12.2p6.

EDIT: but might have to change the make fetchindex part - I'll see how I go: https://forums.freebsd.org/threads/ports-git-portsnap-warning-about-index.79689/


----------



## zirias@ (Apr 6, 2021)

mickey said:


> I only have a handful of additional patch files (read: untracked files) in my ports tree, that git is not aware of. For the time being I'd like to keep it that way and I guess that shouldn't be a problem then?


It's as much a problem as it was with SVN. but as long as you don't _commit_ these to your local branch tracking main from freebsd, `--ff-only` won't have any effect.

The thing is, with GIT, you have a much better option for local changes: Manage them in your own local branch. Roughly like this:


```
git pull
git checkout -b local
git add mycat/addedport
git commit
[edit commit message: "add port mycat/addedport"]
git add devel/frobnicate/Makefile
git commit
[edit commit message: "devel/frobnicate: fix build with libfoobar"]
[…]
```

Then, to update and rebase, you can do

```
git fetch origin main:main
git rebase main
```


----------



## mickey (Apr 6, 2021)

Zirias said:


> It's as much a problem as it was with SVN. but as long as you don't _commit_ these to your local branch tracking main from freebsd, `--ff-only` won't have any effect.


I have no intention of committing those files.


Zirias said:


> The thing is, with GIT, you have a much better option for local changes: Manage them in your own local branch.


I'd rather call those additions than changes, as these are all additional files that live in the corresponding port's files directory. So no file under version control is actually changed. Most of these are additional patches that get applied during the patch phase when building the port. The only exception is an .asterisk.makeopts file which is pretty much expendable, as it can easily be recreated. And from your explanation it seems that I would have to perform additional steps each and every time I update the ports tree, so I don't think it's worth it for the time being.


----------



## grahamperrin@ (Apr 7, 2021)

Zirias said:


> The manual page has nothing to say about this taking time. That's *your* assumption,


Again: *stop putting words in my mouth*. If you must go round in circles and read between the lines, please be correct about what's between.


----------



## grahamperrin@ (Apr 7, 2021)

mickey said:


> … I saw instructions mentioning the use of _-o freebsd_ in the clone command, while others omitted this flag. What difference does it make?



Good question. `-o <name>, --origin <name>` appears in git-clone(1) however I know so little about git, I can't tell the reason(s) for inclusion or omission in cases such as this.

I took my cue (to include) from things such as Warner Losh's *FreeBSD mini-Git Primer* (FreeBSD Journal, November/December 2020). Under _Deep Clone_ (page 2): 

`% git clone -o freebsd $URL -b branch [dir]`

– no use of `-o` in the earlier review copy at https://hackmd.io/hJgnfzd5TMK-VHgUzshA2g#Normal-Clone 
bsdimp (or anyone with the knowhow) please, can you answer the question? Thanks.


----------



## Jose (Apr 7, 2021)

grahamperrin said:


> – no use of `-o` in the earlier review copy at https://hackmd.io/hJgnfzd5TMK-VHgUzshA2g#Normal-Clone
> bsdimp (or anyone with the knowhow) please, can you answer the question? Thanks.


There are two ways to make a Git repository appear on your filesystem. You can go into a directory and type git-init(1). You'll now have a brand-spankin' new Git repository with no commits in it. It will also not have any connections to any remote repositories. It will bask in its unique snowflake nature until you connect it to the mean and nasty Internet.

The other way is to git-clone(1) an existing Git repository from somewhere. That somewhere can be out in the Internets, on some other machine in the same network, or even somewhere else in the same filesystem. This kind of Git repo is not as pure and unsullied. It contains a reference to where it came from in the form of a remote called "origin". However, this is purely convention. You may decide that you want to call the original remote "upstream" or "booyah". That's what the `-o` flag is for, to specify a different name for the remote whence the repo came.

There's absolutely nothing special about the name "origin" other that it's the default. You might consider using something else for this original remote a violation of the principle of least surprise. I don't feel strongly about it one way or the other.


----------



## jmehr (Apr 7, 2021)

Jose said:


> That is not correct. There is no generic "quarterly" branch. Each individual quarterly branch gets a unique name in the form _yyyy_Q_[1-4]_. The latest quarterly branch is 2021Q1. There is no Q2 branch yet. It's part of the migration:
> 
> 
> Git - FreeBSD Wiki
> ...



The "quarterly" section is designed to be an alias for the current quarterly branch or the previous quarterly branch if the current quarterly branch doesn't exist.  The goal is to make tracking quarterly ports branches seamless and to avoid the pain of having to manually update gitup.conf every three months.  Unfortunately, there's a bug in 0.90 where this is broken  -- it's fixed for 0.91 which I'll be pushing out once proxy support is fully implemented (hopefully soon).

Unfortunately, net/gitup will always be slower than the official git client and will be more noticeable with slower hardware (and especially with slower drives).

Because net/gitup minimizes the disk space required, it only deals with shallow copies and does not save the raw packfiles sent from the Git servers.  When requesting pulls, the official Git client depends on the packfile data that has been previously fetched.  If these packfiles contain all of the information needed to create a local repository, then a pristine local repository can be used to reconstruct the information that existed in the original packfiles and net/gitup must walk the local tree and calculate the checksum for every file to do this, sacrificing execution speed to save disk space.


----------



## Argentum (Apr 7, 2021)

Hello!

Here is my 5 cents to this thread - did it on my test desktop machine. Here is my action log:


```
zfs snapshot flash/usr/ports@portsnap

zfs create flash/usr/ports_git

zfs set mountpoint=/usr/ports_portsnap flash/usr/ports

zfs set mountpoint=/usr/ports flash/usr/ports_git

vi /usr/local/etc/gitup.conf

        "ports" : {
                "repository" : "/ports.git",
                "branch"     : "main",
                "target"     : "/usr/ports",
                "ignores"    : [
                        "distfiles",
                        "packages",
                ],
        },

gitup ports -v 1

# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: d73721414d8ded2d68448effc25c895057225a0e
# Branch: main
# Action: clone

portversion -vl "<"

[Reading data from pkg(8) ... - 1474 packages found - done]
Fetching the ports index ... /usr/bin/env  fetch -am -o /usr/ports/INDEX-13.bz2 https://www.FreeBSD.org/ports/INDEX-13.bz2
/usr/ports/INDEX-13.bz2                               2271 kB 2448 kBps    01s
done
[Updating the portsdb <format:bdb_btree> in /usr/ports ... - 31029 port entries found .........1000.........2000.........3000.........4000.........5000.........6000.........7000.........8000.........9000.........10000.........11000.........12000.........13000.........14000.........15000.........16000.........17000.........18000.........19000.........20000.........21000.........22000.........23000.........24000.........25000.........26000.........27000.........28000.........29000.........30000.........31000 ..... done]
```

*Few minutes all the procedure.*


----------



## zirias@ (Apr 7, 2021)

mickey said:


> I have no intention of committing those files.


That's exactly the scenario 



mickey said:


> I'd rather call those additions than changes, as these are all additional files that live in the corresponding port's files directory. So no file under version control is actually changed. Most of these are additional patches that get applied during the patch phase when building the port. The only exception is an .asterisk.makeopts file which is pretty much expendable, as it can easily be recreated.


The benefit is to have your own history with meaningful commits, the ability to track your own changes and easily correct mistakes, and a much easier way to resolve any conflicts should they ever happen. It's nothing you _have_ to do, just a nice tool. To take full advantage, you'd have to learn how to work with "rebase", e.g. from here: https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History

A simpler alternative, if you just continue to keep all your changes in the working copy only, is `git stash`. This will be obligatory _if_ you ever run into a conflict, then you'll have to `git stash` before you can successfully `git pull`. To get your changes back afterwards, write `git stash pop`. In that case, the "stash" is kind of a single commit that will be rebased onto your working copy with "pop". It's your decision whether this is good enough for you. I just recommend the local branch for easy management of several unrelated changes 



mickey said:


> And from your explanation it seems that I would have to perform additional steps each and every time I update the ports tree, so I don't think it's worth it for the time being.


Uhm, I don't think having to issue two commands (a merge and a rebase) for updating instead of just one pull command is really a drawback


----------



## zirias@ (Apr 7, 2021)

jmehr said:


> Because net/gitup minimizes the disk space required, it only deals with shallow copies and does not save the raw packfiles sent from the Git servers. When requesting pulls, the official Git client depends on the packfile data that has been previously fetched. If these packfiles contain all of the information needed to create a local repository, then a pristine local repository can be used to reconstruct the information that existed in the original packfiles and net/gitup must walk the local tree and calculate the checksum for every file to do this, sacrificing execution speed to save disk space.


Sure, walking a large tree takes time, but attributing a difference of _magnitudes_ to it still looks like ignoring a problem…


----------



## mickey (Apr 7, 2021)

I can only (again) wonder about the space savings potential of git compared to svn. In my local ports tree there are no work directories and no distfiles, both are kept on separate volumes each. Until yesterday, this has been the usage of the ports tree updated by svn:

```
sys/usr/ports  used                  3,89G                                 -
sys/usr/ports  referenced            3,89G                                 -
sys/usr/ports  compressratio         2.62x                                 -
sys/usr/ports  compression           lz4                                   local
sys/usr/ports  copies                1                                     default
sys/usr/ports  dedup                 off                                   default
sys/usr/ports  logicalused           5,85G                                 -
sys/usr/ports  logicalreferenced     5,85G                                 -
```
Now after I deleted, recreated and populated the dataset using `git clone -o freebsd --single-branch --depth 1 https://git.freebsd.org/ports.git .` the usage is quite different:

```
sys/usr/ports  used                  789M                                  -
sys/usr/ports  referenced            789M                                  -
sys/usr/ports  compressratio         1.74x                                 -
sys/usr/ports  compression           lz4                                   local
sys/usr/ports  copies                1                                     default
sys/usr/ports  dedup                 off                                   default
sys/usr/ports  logicalused           502M                                  -
sys/usr/ports  logicalreferenced     502M                                  -
```
I know svn has the annoying habit of keeping a _pristine_ copy of everything, effectively doubling the space required, but still...


----------



## mickey (Apr 7, 2021)

Zirias said:


> The benefit is to have your own history with meaningful commits, the ability to track your own changes and easily correct mistakes, and a much easier way to resolve any conflicts should they ever happen.


I understand that it would make my life much easier if it were actually changes (that have a history) we're talking about, But the files in question are not present at all in the repo and are unified diffs against the original sources of a specific port. Nothing you would ever want to manually edit. Either those patches apply cleanly, or (if the original sources have diverged too much) you will have to manually merge the changes and then create a new diff. And given the fact we're talking about a total of 8 files relating to 4 ports, I don't think the additional overhead is justified. If at some point it gets more (and more complex) I would probably think about such a more organized solution though.


----------



## jmos (Apr 7, 2021)

Zirias said:


> The thing is, with GIT, you have a much better option for local changes:


The "always bad thing" about switching from A to B is that your usual workflow no longer works, but you neither know nor are able to use the advantages of the new tool; you only get disadvantages at first.

One big difference I've noticed: "gitup ports" removes any own file in the ports tree directory (a clean "reset" for everything), while "portsnap fetch update" (or a "svn up") handles only files explicit versioned. This "much better option for local changes" may be there, but that will take some more time for me…

However, I'm sure I will make my peace with GIT, and as a friend of simple setups I've also switched now my own projects to GIT (sadly GIT cannot handle a directory structure without containing at least one file - that's so far the ugliest disadvantage).


----------



## zirias@ (Apr 7, 2021)

jmos said:


> One big difference I've noticed: "gitup ports" removes any own file in the ports tree directory (a clean "reset" for everything), while "portsnap fetch update" (or a "svn up") handles only files explicit versioned. This "much better option for local changes" may be there, but that will take some more time for me…


With local changes, you should use git, not gitup.


----------



## ShelLuser (Apr 7, 2021)

jmos said:


> Git is just the next cool must have to play with. It has zero benefit to me, but is a additional burden to deal with.)


Sorry, but that's totally wrong. You _do_ realize that it took FreeBSD 2 or so years before they came to this decision and carried out the change? See 3 years ago I wrote this guide on Git. Back then it was the next new cool thing but after 3 more years?

As for people being more inclined to participate on development because of Git...  I can't tell for sure of course considering that I'm not involved with this myself, but to be honest I wouldn't be surprised if this were true. For example..  if I want to work on a project managed by Subversion then I have to check out the source and after that I'm fully tied to the repository I got the source code from.

With Git on the other hand you have way more freedom. I can work on a project and keep track of my own developments. I can provide my changes to the "powers that be" but if they refuse any of my work then it won't be gone "just like that", I still keep my own _local_ changes. Heck, I can even work on my own stuff while still keeping up to date with the official developments.. You can't easily do that with Subversion. It gets even better: I can also provide these local changes to others.

Sure.. I can understand that it may not seem like much for an end user. But FreeBSD never was about the "user experience" in the first place, things need to work and work as efficiently as possible. That's what happened here.


----------



## olli@ (Apr 7, 2021)

Zirias said:


> Sure, walking a large tree takes time, but attributing a difference of _magnitudes_ to it still looks like ignoring a problem…


Just traversing the ports tree (`find . >/dev/null`) takes a little less than 2 minutes on one of my older machines that still has a HDD (i.e. rotating rust). And that’s without actually opening the files, let alone calculating checksums. This is UFS mounted with the noatime flag; otherwise it would probably take even longer.
 
Just for the record, on my newer machine with SSD (NVMe) the traversal takes only 2.5 seconds. Not too bad for a tree of about 40k directories and 140k files.


----------



## zirias@ (Apr 7, 2021)

olli@ said:


> Just traversing the ports tree (`find . >/dev/null`) takes a little less than 2 minutes on one of my older machines that still has a HDD (i.e. rotating rust).


Hm. I don't own any SSDs, my testing is on a raidz pool with 4 HDDs, and I can't reproduce any substantial(!) slowdown with gitup. But maybe ZFS' ARC helps a lot here?


----------



## olli@ (Apr 7, 2021)

Zirias said:


> Hm. I don't own any SSDs, my testing is on a raidz pool with 4 HDDs, and I can't reproduce any substantial(!) slowdown with gitup. But maybe ZFS' ARC helps a lot here?


Good question. I did my test with empty cache, because that’s the typical case when updating the ports tree. When I repeat the test immediately, the traversal is much faster, of course, because the whole ports tree is still in the cache. Also, I don’t have any RAID (I guess most users have only one disk for FreeBSD).
 
I haven’t used gitup for ports so far (I intend to continue using portsnap for the time being, as I don’t really have a good reason to change right now), so I can’t compare. But I use gitup for updating my src tree, and that is roughly as fast as svnup that I used before. So I’m happy with it.


----------



## suntzu00 (Apr 7, 2021)

> With poudriere-ports(8) as a starting point, how would a person discover the use of option -U for the git method?











						poudriere/ports.sh at master · freebsd/poudriere
					

Port/Package build and test system. Contribute to freebsd/poudriere development by creating an account on GitHub.




					github.com


----------



## Argentum (Apr 7, 2021)

olli@ said:


> Good question. I did my test with empty cache, because that’s the typical case when updating the ports tree. When I repeat the test immediately, the traversal is much faster, of course, because the whole ports tree is still in the cache. Also, I don’t have any RAID (I guess most users have only one disk for FreeBSD).
> 
> I haven’t used gitup for ports so far (I intend to continue using portsnap for the time being, as I don’t really have a good reason to change right now), so I can’t compare. But I use gitup for updating my src tree, and that is roughly as fast as svnup that I used before. So I’m happy with it.


First time running on my desktop gave:


```
time gitup ports -v 1

# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: 9aae35b48f216c96c71d782f0f14e3cfd6a70f90
# Branch: main
# Action: clone

Executed in   55.59 secs    fish           external
   usr time    7.04 secs  512.00 micros    7.04 secs
   sys time   14.61 secs  256.00 micros   14.61 secs
```

Next time:


```
root@Tuna2 ~# time gitup ports -v 1
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: cf118ccf875508b9a1c570044c93cfcc82bd455c
# Want: 0997d61ae7a84a95b0e81febfac0e4da9a247ac2
# Branch: main
# Action: pull

________________________________________________________
Executed in   15.16 secs    fish           external
   usr time    3.38 secs  514.00 micros    3.38 secs
   sys time    7.19 secs  265.00 micros    7.19 secs
```


And just few minutes later without updates:


```
time gitup ports -v 1
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Have: 0997d61ae7a84a95b0e81febfac0e4da9a247ac2
# Want: 0997d61ae7a84a95b0e81febfac0e4da9a247ac2
# Branch: main

________________________________________________________
Executed in    8.37 secs    fish           external
   usr time    2.33 secs    0.00 millis    2.33 secs
   sys time    5.13 secs    5.16 millis    5.13 secs
```


----------



## Jose (Apr 7, 2021)

grahamperrin said:


> Thanks, I needed that.
> 
> With poudriere-ports(8) as a starting point, how would a person discover the use of option `-U` for the git method?


It's unfortunately not documented, AFAIK. That command was posted to `freebsd-ports` a couple of days ago:


			poudriere and gitup
		


There are recommendations in that mail thread for alternatives to the `-U` switch:


			poudriere and gitup
		



			poudriere and gitup
		


I haven't tested any because as I said, I use `-m null` and `-M /path/to/git/ports` to point Poudriere at a manually-maintained Git checkout of the ports tree.


----------



## Jose (Apr 7, 2021)

mickey said:


> I understand that it would make my life much easier if it were actually changes (that have a history) we're talking about, But the files in question are not present at all in the repo and are unified diffs against the original sources of a specific port. Nothing you would ever want to manually edit. Either those patches apply cleanly, or (if the original sources have diverged too much) you will have to manually merge the changes and then create a new diff. And given the fact we're talking about a total of 8 files relating to 4 ports, I don't think the additional overhead is justified. If at some point it gets more (and more complex) I would probably think about such a more organized solution though.


Disclaimer: I'm trying to help you. Let me know if I'm annoying you instead and I'll shut up. No need to read the following if that's the case.

Well git-diff(1) creates unidiff patches by default. Imagine this scenario:

You keep a local tracking branch of the quarterly branch called `2021Q1`
You have a private local branch called `mychanges` with your local port changes
Workflow 1: Update the quarterly tracking branch

```
git switch 2021Q1
git pull --ff-only
git switch mychanges
git rebase 2021Q1
```
The rebase will warn you if there are conflicts, and will leave modified files in your working tree with conflict markers. You would then resolve your conflicts and commit your changes. Your local branch now has history for the changes you've had to make to keep up with upstream.

Workflow 2: New quarterly ports branch

```
git switch -c 2021Q2 -t origin/2021Q2
git switch mychanges
git rebase 2021Q2
```
And resolve the conflicts as before.

Workflow 3: Create unified diffs of your changes against the quarterly tracking branch

```
git switch 2021Q2
git pull --ff-only
git diff mychanges..
```

I haven't actually tested these git command sequences, but they should be pretty close to accurate.


----------



## bagas (Apr 7, 2021)

Can't find a guide on how to set up gitup to get ports.
Created a config file, but I don't quite understand if it is correct.


> /usr/local/etc/gitup.conf
> {
> "defaults" : {
> "host"           : "git.freebsd.org",
> ...


----------



## Jose (Apr 7, 2021)

bagas said:


> Can't find a guide on how to set up gitup to get ports.











						Solved - starting port updates
					

Hello. The last time the ports were updated a week ago. Now I start updating ports and receive messages that the ports have not changed, there are no updates.  # portsnap fetch update Looking up portsnap.FreeBSD.org mirrors... none found. Fetching snapshot tag from portsnap.FreeBSD.org... done...




					forums.FreeBSD.org


----------



## Argentum (Apr 7, 2021)

bagas said:


> Can't find a guide on how to set up gitup to get ports.


Here it is:

1. Install net/gitup
2. Edit /usr/local/etc/gitup.conf

```
"ports" : {
                "repository" : "/ports.git",
                "branch"     : "main",
                "target"     : "/usr/ports",
                "ignores"    : [
                        "distfiles",
                        "packages",
                ],
        },
```

See my post how I did it


----------



## bagas (Apr 7, 2021)

Jose said:


> Solved - starting port updates
> 
> 
> Hello. The last time the ports were updated a week ago. Now I start updating ports and receive messages that the ports have not changed, there are no updates.  # portsnap fetch update Looking up portsnap.FreeBSD.org mirrors... none found. Fetching snapshot tag from portsnap.FreeBSD.org... done...
> ...


Thanks.
fixed gitup.conf.


----------



## Jose (Apr 7, 2021)

Ports transition to Git complete:




__





						New 2021Q2 branch
					





					lists.freebsd.org
				





			Git - FreeBSD Wiki


----------



## grahamperrin@ (Apr 7, 2021)

olli@ said:


> … a tree of about 40k directories and 140k files. …



For posterity/fun:



			https://github.com/johnmehr/gitup/files/6269723/2021-04-06.09.20.ports.317.7.MB.jdr.tar.gz
		


a report on my local /usr/ports as it appeared whilst the repository was frozen (shortly before the first commit to Git)
readable with sysutils/jdiskreport




– and so on.



Jose said:


> unfortunately not documented, AFAIK.



suntzu00 got it ☑ at https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504759 – it's documented in the main manual page for poudriere. Embarrassingly I hadn't thought to look there, because I (lazily) guessed that a URL would apply only in the context of creation of a ports tree. ¶ Thanks for replying, and for the other stuff.


----------



## Jose (Apr 7, 2021)

grahamperrin said:


> suntzu00 got it ☑ at https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504759 – it's documented in the main manual page for poudriere. Embarrassingly I hadn't thought to look there, because I (lazily) guessed that a URL would apply only in the context of creation of a ports tree. ¶ Thanks for replying, and for the other stuff.


Where are you seeing it? I can't find `-U` in either poudriere(8) or poudriere-ports(8).

You're welcome!


----------



## Argentum (Apr 7, 2021)

grahamperrin said:


> readable with sysutils/jdiskreport


or deskutils/baobab


----------



## grahamperrin@ (Apr 7, 2021)

Jose said:


> Where are you seeing it?


Sorry! I wasn't paying attention. My mistake, it's not in the main manual page, it's in a script. Here's the point at which it was added, a few years ago: 









						Level the ports git integration to the same level as jail · freebsd/poudriere@d547c13
					

Add a new argument -U to allow to to directly speficy the URL where to fetch the ports from Deprecate GIT_URL (but still use it in case already defined in poudriere.conf. Probably here we should no...




					github.com


----------



## mickey (Apr 7, 2021)

Jose said:


> Disclaimer: I'm trying to help you. Let me know if I'm annoying you instead and I'll shut up.


I am not annoyed and I really appreciate your help getting a better understanding of how git works, which is quite different than what I am used to, coming from CVS and later SVN. I just don't feel that keeping my few additional patches in a local git branch would really make my life easier, at least not given my usual workflow that basically so far has been:

make -C /usr/ports update fetchindex
portmaster -a -d
And then repeat step 2 on every other machine that has the same ports tree mounted read-only via NFS.

Now after I made the switch to git not much has changed so far:

git -C /usr/ports pull --ff-only
portmaster -a -d
and again repeat step 2 on other machines. As I wrote in another thread, I decided to ditch the index and go without it, so I will not run `make fetchindex` anymore as part of my daily update routine. I could continue using `make -C /usr/ports update` which effectively does a `git pull` but I decided against that and defined an alias that includes the _--ff-only_ flag and is more convenient to use, as I dont have to specify the directory manually.

You see the sole purpose of my local copy of the ports tree is to keep my machines updated. I don't use quarterly branches (in fact I never have), HEAD is the only thing I need, which is also why I decided to go with a shallow, single branch clone for the time being. The few additional files I keep in there were mostly born out of necessity and should that necessity cease to exist, I will more than happily remove those files as I generally prefer to have a clean tree. In some cases I almost forgot I put those files there long ago and running `git status` gives a quick overview of what files are there and where, that's fine by me.

git comes with substantial differences compared to CVS/SVN that are not easy to wrap your head around at first. But I guess it will get better once I start using git for my own pet projects or maybe even convert some of my repositories. So far I can say that I love the speed of git and the savings in disk space. I guess I will keep it


----------



## zirias@ (Apr 7, 2021)

mickey said:


> In some cases I almost forgot I put those files there long ago and running `git status` gives a quick overview of what files are there and where, that's fine by me.


Sometimes, it helps more to have a commit message attached to them explaining _why_ these files are there  At least, that's why I've been waiting to have the official ports on git for years, so I'm pretty happy right now, having a local log like this:

```
334a65f5f620 (local) security/stunnel: downgrade to 5.48
4c6e5b95cd26 emulators/virtualbox-ose: fix build with libressl
8342e38e2f02 sysutils/gkrellm2: fix build with libressl
4e8215bada15 www/chromium: fix build with MIT krb5
[...]
25feb0d686a1 (origin/main, origin/HEAD, main) science/dlib-cpp and science/py-dlib: Update to 19.22
34a84984bcb2 net-p2p/xmrig: Update to 6.11.1
```

But yes, maybe it's easier after having played with git in some small repository created by yourself


----------



## chrcol (Apr 7, 2021)

Everything working here now, thanks to all those who posted information on this.

I may migrate to using gitup though as that tool is for people like me who just want to check out the changes for maintaining the system, as on normal git I am not sure the best flags to use for this `--ff-only` etc.


----------



## jb_fvwm2 (Apr 8, 2021)

```
gitup -v 2 ports | grep descr | tee -a /tmp/472021-gitup.dat
```
Then, if that works as it does for me, the file at the top will have
+ pkg-descr which were added [ new ports, or revised ? ] which is
a functionality I was used to in svn.
It needs the latest working /usr/local/etc/gitup.conf though.


----------



## chrcol (Apr 8, 2021)

Feedback on performance, using baseline git (not yet tried gitup), its very quick, the slow part is making the index, would love `make fetchindex` to work on the quarterly branch.  Or does it work now? as I see it mentioned a few posts up.


----------



## zirias@ (Apr 8, 2021)

chrcol probably the best recommendation as of now is not to use an INDEX at all:








						[ports][GIT][portsnap] Warning about INDEX
					

In the context of ports moving to GIT, as well as (expected) deprecation of portsnap, I've seen some questions regarding INDEX, which was automatically fetched by portsnap.  Right now, I learned the following in an IRC channel full of devs: < XXXXXX> INDEX is currently dangerously misleading...




					forums.freebsd.org
				




What do you need it for?


----------



## chrcol (Apr 8, 2021)

I use portupgrade which needs it? (just tested now after your post to be sure, if it doesnt exist it does a make fetchindex which is bad as thats the HEAD version)

I have a script that runs each night after ports tree is updated and a command it runs to inform of which ports need updating seems to require the index as well.

I have now just altered my script though which is now able to detect if any updates were pulled over git, if none pulled it doesnt make a new index.  So its not a huge problem.


----------



## zirias@ (Apr 8, 2021)

I remember having read about several issues with portupgrade, don't remember the details. Yes, one of them is relying on INDEX. You should probably move away from that tool.

One possibility is portmaster. IMHO a lot better is a tool like either poudriere or synth.


----------



## chrcol (Apr 8, 2021)

I am happy with portupgrade, thanks.


----------



## zirias@ (Apr 8, 2021)

Right now, it's already broken by the fact that it _uses_ INDEX, which can't be generated correctly in presence of flavors (and, a lot of ports now use them). So, inevitably, it will occassionaly do the wrong thing. Don't complain afterwards.

I often see people refusing to change their workflow. Typically, something has to go badly wrong to change your mind. Well, you have been warned


----------



## richardtoohey2 (Apr 8, 2021)

I use the index in 
	
	



```
pkg version -vIL=
```



Zirias said:


> Don't complain afterwards.


I won't!


----------



## mickey (Apr 8, 2021)

Zirias said:


> Sometimes, it helps more to have a commit message attached to them explaining _why_ these files are there


I know exactly why these files are there. One removes a call to syscons from x11/nvidia-driver which causes it to fail if your kernel is built with vt(4) only, two files fix IPv6 prefix lifetimes in net/dhcp6, 4 files are additional patches to a specific port and the last is a custom build configuration for net/asterisk18 as generated by `make menuselect`.  


Zirias said:


> But yes, maybe it's easier after having played with git in some small repository created by yourself


Certainly. Using a VCS to update src/ports is an entirely different thing from using it in active development.


Zirias said:


> I remember having read about several issues with portupgrade, don't remember the details. Yes, one of them is relying on INDEX.


If that hasn't changed in all those years, it depended on ruby (*yuk*) and used bdb to perform it's own _housekeeping_ and also had a tendency to fail often. I'm really surprised anyone still uses it nowadays. Many years ago when I switched to ports-mgmt/portmaster it felt like I was one of the last to leave the sinking ship already.


richardtoohey2 said:


> I use the index in
> 
> 
> 
> ...


Use `pkg version -vPL=` and it will use the ports tree instead of the index file. You can also omit the option that specifies the method and it will automatically switch to using the ports tree in case an index file is not present: `pkg version -vL=`


----------



## olli@ (Apr 8, 2021)

Zirias said:


> I remember having read about several issues with portupgrade, don't remember the details. Yes, one of them is relying on INDEX. You should probably move away from that tool.
> 
> One possibility is portmaster. IMHO a lot better is a tool like either poudriere or synth.


In fact, I think that portmaster works better than portupgrade in general, not only because of the INDEX issue.
But YMMV, of course.


----------



## SirDice (Apr 8, 2021)

mickey said:


> If that hasn't changed in all those years, it depended on ruby (*yuk*) and used bdb to perform it's own _housekeeping_ and also had a tendency to fail often.


I didn't have a problem with Ruby but the constant battling with a corrupt database made me seriously question my sanity. I was so happy when portmaster was released. Besides having no dependencies it worked much better than portupgrade ever did.


----------



## scottro (Apr 8, 2021)

I just tried with git, well,  actually gitup. It was very quick, so quick that I thought it hadn't worked. It doesn't make an INDEX, which messes up psearch, a package that I've used for years. I think a person who I was friends with on the old #freebsd irc, (before freenode, though I can't even remember which server--this was years ago), created it.

But aside from the lack of the INDEX file, it works quite well  Usually, the only package I build from ports is dwm, because of some customization, so, I wouldn't notice too many other effects.


----------



## dR3b (Apr 8, 2021)

Why hasn't the branch "2021Q2" been created on the Github mirror yet?

[1] https://github.com/freebsd/freebsd-ports


----------



## olli@ (Apr 8, 2021)

scottro said:


> I just tried with git, well,  actually gitup. It was very quick, so quick that I thought it hadn't worked. It doesn't make an INDEX, which messes up psearch, a package that I've used for years.


I don’t know exactly what features psearch provides, but maybe Porgle is useful for you. It’s a web-based search engine for the FreeBSD ports collection. It has low browser requirements, it also works fine with text-based browsers (lynx, w3m, links).


----------



## zirias@ (Apr 8, 2021)

dR3b said:


> Why hasn't the branch "2021Q2" been created on the Github mirror yet?
> 
> [1] https://github.com/freebsd/freebsd-ports


Good question, possibly a bug? I tried to ping lwhsu@ about it…


----------



## dR3b (Apr 8, 2021)

Zirias said:


> Good question, possibly a bug? I tried to ping lwhsu@ about it…


That would be nice. This would also solve my problems with "poudriere".

```
poudriere ports -c -B 2021Q1 -m git+https -p 2021Q1
```


----------



## zirias@ (Apr 8, 2021)

Any reason not to use the official repository? Is this "hardwired" to github in poudriere?


----------



## dR3b (Apr 8, 2021)

Zirias said:


> Any reason not to use the official repository? Is this "hardwired" to github in poudriere?


Yes it is: /usr/local/share/poudriere/common.sh

```
7581 : ${SVN_HOST="svn.freebsd.org"}
7582 : ${GIT_BASEURL="github.com/freebsd/freebsd.git"}
7583 : ${GIT_PORTSURL="github.com/freebsd/freebsd-ports.git"}
7584 : ${FREEBSD_HOST="https://download.FreeBSD.org"}
```


----------



## zirias@ (Apr 8, 2021)

```
16:28:35 <@lwhsu> Zirias: it will be addressed soon
```


----------



## Deleted member 30996 (Apr 8, 2021)

ShelLuser said:


> Sorry, but that's totally wrong. You _do_ realize that it took FreeBSD 2 or so years before they came to this decision and carried out the change? See 3 years ago I wrote this guide on Git. Back then it was the next new cool thing but after 3 more years?


I was going to recommend someone participating in the thread make a Tutorial out of their efforts to save answering the same question over and over.

And since you have, saved me the work of figuring it out myself. 


Ports good. Change bad. Like ports. Love FreeBSD. Make change. Dung flung. 

Adapting to change and making it the new normal survival of the fittest. 

Natures Way... I'm such a fan of it.


----------



## zirias@ (Apr 8, 2021)

90% of complaints around transition to git is category "change is hard". Duh.


----------



## grahamperrin@ (Apr 8, 2021)

*Discussions of gitup*


Jose said:


> … I suggest you take this subject to its own topic. …





For those of you who are unaware, there's a *gitup*-specific topic (begun by the developer):









						gitup
					

Hello everyone,  With the upcoming transition from Subversion to Git, I've been plugging away at building "gitup", a replacement for net/svnup and I'm happy to report that I've got a working "git clone" prototype.  Looking back at when I was writing net/svnup, one of the things I wished I had...




					forums.freebsd.org


----------



## grahamperrin@ (Apr 8, 2021)

dR3b said:


> Why hasn't the branch "2021Q2" been created on the Github mirror yet?


If it helps, whilst you wait: 









						Files · 2021Q2 · FreeBSD / FreeBSD ports · GitLab
					

GitLab.com




					gitlab.com


----------



## dR3b (Apr 8, 2021)

Currently this is my "workaround":

```
poudriere ports -c -B 2021Q2 -m git+https -p 2021Q1 -U https://git.freebsd.org/ports.git
```


----------



## zirias@ (Apr 8, 2021)

Hm, for ports, I just always use the `null` method with poudriere. I don't see any drawback updating directly with `git pull`


----------



## mtu (Apr 8, 2021)

This is very topical, it popped up on IRC, allegedly came from some Discord:









						FreeBSD Ports has switched to git
					

Confusion abounds as the FreeBSD Ports Tree repository has been switched from svn to git with none of the supporting architecture able to cope.




					www.captiongenerator.com


----------



## chrcol (Apr 8, 2021)

SirDice said:


> I didn't have a problem with Ruby but the constant battling with a corrupt database made me seriously question my sanity. I was so happy when portmaster was released. Besides having no dependencies it worked much better than portupgrade ever did.


I prefer this doesnt become another what ports tool to use thread as lately on here there seems to be a campaign to get people to move over to the newer tools, but I will post a summary of why its my preference, although I dont use it on every server.

Essentially it boils down to I prefer how it deals with specific situations, the nice summary when its done, the automatic restore of old version of port if install fails.  I do get however it is getting more and more dated and some do not like how it works.

I found a nasty looping problem with portmaster, where it can be made to go in a never ending loop with dependencies, there is a problem report on the freebsd website and a developer has taken the problem on, he is refactoring portmaster, but decided to start again so no ETA on that.

I personally dont want to have a duplicated port build system on my server which is why I am not using poudriere or synth, its not so much about workflow, I can change it to adjust to new tools, but rather just my preference in how the tools work.

My opinion on the index problem is likely an outspoken one, the way I see it, if someone changes the way ports work aka flavor's, its their responsibility to make sure all parts of the ports tree function correctly including the index feature.  But I have already found a workaround for portupgrade on git lite, although over time as more and more ports switch to flavor's I may decide the work applying these workarounds to more and more ports isnt worth it, I am assuming at this point index wont be fixed as I expect there is a popular opinion of wanting to retire tools like portupgrade and keeping index broken is a way of doing that.


----------



## omarsidd (Apr 9, 2021)

Would like to see more importance given to documenting (or even just announcing) major changes like this. 
Not as an afterthought once a few dozen comments have piled up here. But as a common sense part of such transitions.

Eg:
* The FreeBSD handbook should have been updated immediately as the reference of record. 
* The ports page (default handbook version) had no mentions of git when I checked earlier. There are still versions coming up in search (yes, on the docs.freebsd.org site) that don't have the current info.
* Check those top search results for likely strings like "updating freebsd ports" and make sure those resources are accurate?
* The UPDATING or README files in the subversion repo for ports don't have anything saying "this is now static / unsupported". Shouldn't that have been in the last commit?
* The front page news on the freebsd website has no mention. Ports affects a lot more people than, say, the newest RC being out. Spread awareness.
* The project twitter feed has no recent mentions.
* Perhaps a feedback link on the git/ports wiki to let users say "hey you missed these spots"?
* Based on other comments here, test common tools, not just developer favorites?

Yes, yay for progress. But it's 2021, updating matching resources should be a given, not an afterthought.
A newcomer to the community this week would certainly have been confused, and ports are not optional for most use cases.


----------



## grahamperrin@ (Apr 9, 2021)

Food for thought:

the primary *FreeBSD News* page – <https://www.freebsd.org/news/>
the subdirectory for news flashes – <https://www.freebsd.org/news/newsflash/>
How many of you were aware of the primary page?

The *About* sidebar at the FreeBSD site includes a link to News that misleads to <https://www.freebsd.org/news/newsflash/> with no way back to the News page.

Much of what's _flashed_ is not truly news flash-worthy. Critically:

if *every* item of news is treated (or mistreated) as flashy, it can become difficult for an average reader to *discern, at a glance, things that are of special or critical importance*.



omarsidd said:


> Would like to see more importance given to documenting (or even just announcing) major changes like this.



Warner Losh's FreeBSD mini-Git Primer (FreeBSD Journal, November/December 2020) began (emphasis: mine):

❝The FreeBSD project has begun its transition from Subversion to Git. This move has been *over a year in the planning* and represents the next step in FreeBSD’s continuing efforts to improve its workflow. …❞

Looking back a year or so, we'll probably find relevant announcements, however some of them might be nested in quarterly reports or away in mailing lists. Not treated as directly newsworthy (or news flash-worthy) at the time.
I'll probably edit this post to include some of what was relevant.



> Not as an afterthought once a few dozen comments have piled up here. But as a common sense part of such transitions.
> 
> Eg:
> * The FreeBSD handbook should have been updated immediately as the reference of record.
> * The ports page (default handbook version) had no mentions of git when I checked earlier.



Please, how much earlier? <https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504060> around six days ago: "… At a glance, the current edition of the Handbook is similarly non-prepared for completion of the transition. …".

<https://docs.freebsd.org/en/books/handbook/ports/#ports-using-git-method> does mention Git.

<https://cgit.freebsd.org/doc/log/?qt=grep&q=Git> three days ago, _Handbook: adjust for the various git migrations_.

Your most recent view might have been an outdated edition of the Handbook, or prior to the adjustment.









						FreeBSD Forums header area (above the Forums button) directing readers to oudated content (outdated Handbook, and so on)
					

At a glance, at least six of the links under Documentation are outdated.   https://docs.freebsd.org/en_US.ISO8859-1/books/handbook/ is an outdated edition of the FreeBSD Handbook; and so on.   Please aim for consistency with the main FreeBSD site. Thanks.




					forums.freebsd.org
				






> There are still versions coming up in search (yes, on the docs.freebsd.org site) that don't have the current info.
> * Check those top search results for likely strings like "updating freebsd ports" and make sure those resources are accurate?



<https://forums.FreeBSD.org/threads/ports-transitioned-to-git.79598/post-504067> "… In IRC, someone mentioned work to improve the situation. I assume that redirects will be involved."



> * The UPDATING or README files in the subversion repo for ports don't have anything saying "this is now static / unsupported". Shouldn't that have been in the last commit?



Both <https://cgit.freebsd.org/ports/commit/?id=4010f7bbc03638d71781ce091bf40a0907fa12fe> (2021-03-31) and <https://cgit.freebsd.org/ports/commit/?id=ed8d3eda309dd863fb66e04bccaa513eee255cbf> (2021-04-06) made reference to <https://wiki.freebsd.org/git> where it's clear that write access to Subversion was disabled. At the foot of the page:

❝Documents at imp's github repo were drafted as material for the FreeBSD Handbook, which was converted to AsciiDoc/Hugo around the same time. There remain some items of interest.❞

– if I recall correctly, one of the most interesting items was _FreeBSD moving to Git: Why_. Subscribe to <https://github.com/bsdimp/freebsd-git-docs/issues/91>, if you like.



> * The front page news on the freebsd website has no mention. Ports affects a lot more people than, say, the newest RC being out. Spread awareness.



See above, the food for thought.



> * The project twitter feed has no recent mentions.
> * Perhaps a feedback link on the git/ports wiki to let users say "hey you missed these spots"?
> * Based on other comments here, test common tools, not just developer favorites?
> 
> ...



I suspect that in the noise, some people lost sight of the opening post (emphasis: mine):



SirDice said:


> Last SVN commit to the ports tree:
> 
> 
> 
> ...



*Postscript*

FreeBSD and Git a.k.a. _Evaluating GIT for FreeBSD_ (Ed Maste) ◀ October 2018 FreeBSD Developer Summit | FreeBSD Foundation

Warner's Random Hacking Blog: FreeBSD Subversion to Git Migration: Pt 1 Why? (2020-09-19) ◀ https://old.reddit.com/r/freebsd/comments/iw2b69/-/


----------



## Deleted member 30996 (Apr 9, 2021)

I never thought about what it will do to my Tutorial and the use of ports... Nothing till I change from using them.

Get to typing and don't let me hear those keys quiet till you rewrite it or Shakespeare.

Alas, poor portmaster! I knew him, Beasite, a fellow of infinite text, of most excellent fancy. He hath borne me on his back a thousand times, and now...

Monkey see? Monkey do.


----------



## mtu (Apr 9, 2021)

I'm also bugged by most of the annoyances mentioned here. I have a mental model that I think explains what's going on.

The basic hypothesis is: _*FreeBSD is geared towards the needs of those working on it.*_ That's volunteer devs, employed devs and (by extension) vendors using FreeBSD for their businesses and contributing back (e.g. Netflix, but not Apple). FreeBSD specifically does *not* cater to the needs of whoever could be considered an "average user", which includes most people on these Forums.

Some examples that I think illustrate this point:

For development and server usage, consumer hardware support and grapical desktops are not a priority. Hence, the sorry state of the installer, graphical desktop integration and WiFi.
Poudriere is the favorite and best-suited tool for developers and professional deployment. Hence, other ways of using ports are poorly supported and documented: manual ports compilation, binary packages and related tools like portmaster. In the same vein, developers are likely to maintain and work on their own ports tree(s) with SVN/git, and so alternatives for users' convenience such as portsnap, INDEX or gitup are not a priority.
There's almost no overlap between the users gathered on these Forums or on Reddit and the people working on FreeBSD. Naturally, the communication channels and tools relevant to the development of the project are those used in development. Mailing lists are key, as is a private Slack used by core devs, and IRC to some degree. Phabricator and the Bugzilla play important roles, issues and pull requests on Github do not. Pointing out flaws or suggesting improvements on these Forums is akin to screaming into the void.
Documentation is recognized to be important, but not a priority when implementing changes – because those working on FreeBSD already know how to do things. Documentation therefore tends to lag behind current events and the latest-and-greatest ways of deployment/development.
This is emphatically *not an attack on "the devs"* in the name of "the users". Some people are devoting much of their lives to FreeBSD (or successfully using it to do business), and *I am grateful for getting a great piece of software out of it* without even paying any money.

I am, however, irritated by inconsistencies that arise from these realities in contrast with the stated goals of the project. For example:

poudriere is the de-facto standard for dealing with ports—but it's not part of the base system, and not documented in accordance with its importance. What _is_ in the base system, like portsnap and ports maintenance by raw Makefiles, gets little attention in comparison.
The base system is meant to provide the tools needed for development, which is why it contains a compiler for example. The project also went to great pains to banish GPL'ed code from the base system. And yet it's practically impossible to do serious work on FreeBSD without git, which is GPLv2, and therefore cannot be part of the base system.
Now, there's always a voice that says "stop complaining and help out". Fair enough – I for one am trying to contribute by maintaining a small port, diligently reporting bugs and fighting my way through a Phabricator review every once in a while. But the reality is that not everyone that likes to use FreeBSD is in a position to dedicate as much time and energy as is needed to make a difference.

*What to make of all this?* I don't see a way to change this state of affairs, much as I would like to see some things done differently – it's an uphill battle I can't win with the time and energy on my hands. I'm trying to *adapt my expectations*, and tell myself: In any aspect of using FreeBSD, you can either do it the way most devs do, or be largely on your own.


----------



## PMc (Apr 9, 2021)

mtu said:


> The basic hypothesis is: _*FreeBSD is geared towards the needs of those working on it.*_


I don't think You're much wrong - only, I would rather pronounce it as "working _with_ it".  It is the difference between business customers (B2B) and consumers (B2C). 

It is the same when you buy a car, everything is explained in a very foolproof fashion. When you buy heavy-industry vehicle like forestry-machines, this is very different: you are expected to know how such equipment works, how it is to be handled and what are the important things (or have personnel who knows).
And certainly you can use your caterpillar for everyday driving, too - but then, maybe the stereo will not have all the perfect sound. (With real hardware, the segregation usually happes per price tag.)

Back in the days when FreeBSD got published, there was still a clear distinction between personal computer users (windows) and business applications (workstation/server=>unix). Now with Berkeley on the apples and linux on the gadgets the distinction is no longer clear.


----------



## olli@ (Apr 9, 2021)

PMc said:


> Back in the days when FreeBSD got published, there was still a clear distinction between personal computer users (windows) and business applications (workstation/server=>unix). Now with Berkeley on the apples and linux on the gadgets the distinction is no longer clear.


In this context, it is also interesting to see how the FreeBSD motto changed over the years.
In the 90s its motto was “Turning PCs into Workstations” – I still have a bunch of FreeBSD stickers and case plates made by Walnut Creek that have this motto on them.
For the past few years the motto has been “The Power to Serve”. This pretty much indicates the target audience and main intended use of FreeBSD.


----------



## ShelLuser (Apr 9, 2021)

At the risk of going slightly offtopic (not my intention) but...



mtu said:


> Some examples that I think illustrate this point:
> 
> Poudriere is the favorite and best-suited tool for developers and professional deployment.


"Best suited tool"? I _strongly_ beg to differ.

I maintain 3 VPS servers, one is leading and the others provide backups. I prefer full control and thus I'm using the Ports collection to install my software. I dislike overhead and therefor my main server provides a package repository for my other servers to use. This means I build once and install many.

All I'm using is ports-mgmt/portmaster (and a custom shell script to keep things in check). I simply install ("update") a new port, Portmaster places the package of the new port in /usr/ports/distfiles/All and a daily crontab makes sure that the repository databases get updated ("`pkg repo /root/repo/pub.key`"), all distributed by Apache.

In other words...  I install a port upgrade, and the next day it'll be automatically available for pkg to install.

Just because something is popular doesn't automagically imply it's also the best.



mtu said:


> Hence, other ways of using ports are poorly supported and documented: manual ports compilation, binary packages and related tools like portmaster.


 _Yeah right._

Lessee now...  ports(7) which lists all build targets (you _do_ realize that manual pages are actual manual pages on FreeBSD, right?), chapter 4.5 of the FreeBSD handbook (which even mentions Portmaster for crying out loud!) which obviously brings me to: portmaster(8) (remember my comment about manual pages?).

And about those binary packages: chapter 4.4 of said handbook.

This also brings us to pkg(8) which can refer you to pkg-install(8) and/or pkg-repo(8), if not from the command summary then surely from the SEE ALSO section.

(edit):

_I probably shouldn't... oh well._



mtu said:


> Documentation is recognized to be important, but not a priority when implementing changes


So that's why Git got mentioned in the handbook _long_ before the whole transition had been finished.

I can't help think you don't realize the massive effort that goes into some proper documentation. And I say that because of this:









						[Guide] Using Git to manage ports, source and documentation.
					

Hi gang!  Disclaimer: I am honestly a little excited about recent developments so expect to find some (small) opinionated parts in this guide. Nothing excessive mind you, but I can sometimes get a little carried away and despite some believes I never really plan guides like this.  Editorial  In...




					forums.freebsd.org
				





See, if they really didn't give this any priority then they'd postpone the whole thing until after the transition was finished (like I did). It's _a lot_ easier to add a referral to the project overview than to add actual documentation. But that's not what they did now, is it?



mtu said:


> – because those working on FreeBSD already know how to do things. Documentation therefore tends to lag behind current events and the latest-and-greatest ways of deployment/development.


See my point above.

It would really help your post if you would back up your unfounded claims by providing actual examples.

But... _auch_, that works both ways  => Did you notice there's a "Git method" section in chapter 4.5 of the handbook?

Now, it's a real shame that they no longer provide a "last updated" section in the handbook but... I has source code:


```
commit a3ad91f11fea5cc457eb053c46c26e4dd43df599
Author: Rene Ladan <rene@FreeBSD.org>
Date:   Tue Apr 6 20:59:06 2021 +0200

    Handbook: adjust for the various git migrations.

    - prefer to install git as a package over building it oneself.
    - update dates in in the introduction
    - let some Subversion URLs move to the src repository, as it is the only
      repository with an Subversion exporter active.
    - let the user install git instead of subversion for doing ports work.

    Reviewed by:    emaste, mat
    Differential Revision:  https://reviews.freebsd.org/D29450
```
Your post: today (9th of April), these documentation changes: 6th of April.

Oh, it gets much better:


```
commit abff932fe818c144380fd5ebfdcdf44f93de46de
Author: Warner Losh <imp@FreeBSD.org>
>>> Date:   Tue Mar 16 21:55:44 2021 -0600

    Merge in the migration from subversion / github

    Add in the migrating from the old subversion / git docs. This section
    should be removed in a few months maybe.

    Contributions by: lwshu@, Alexander Richardson, kib@, Ceri Davies

[...]

commit cf77cf8be5142f3069e313f6eef3a36597b3dab3
Author: Li-Wen Hsu <lwhsu@FreeBSD.org>
>>> Date:   Tue Jan 12 00:46:00 2021 +0800

    Update information of source code repositories

    Doc and src repositories are in git now and ports tree is in progress.

    Reviewed by:    uqs, ygy
    Approved by:    ygy
    Differential Revision: https://reviews.freebsd.org/D28080

[...]

commit f3bf61d5614d54621681c2613edfa77c3eb659db
Author: Li-Wen Hsu <lwhsu@FreeBSD.org>
>>> Date:   Thu Jan 7 17:13:44 2021 +0800

    Add link to git pepository

    Approved by:    ygy
    Differential Revision:  https://reviews.freebsd.org/D28015

[...]

commit e98cc8974139ae77d1550f60dba75e3d7435915d
Author: John Baldwin <jhb@FreeBSD.org>
>>>> Date:   Mon Dec 21 11:07:24 2020 -0800

    Generate links to cgit instead of svn in document title blocks.

    PR:             251940
    Reviewed by:    emaste, gjb, ygy
    Differential Revision:  https://reviews.freebsd.org/D27704
```
Concluding: the documentation got updated _long_ before the transition was even _close_ to completion.



mtu said:


> poudriere is the de-facto standard for dealing with ports—but it's not part of the base system, and not documented in accordance with its importance. What _is_ in the base system, like portsnap and ports maintenance by raw Makefiles, gets little attention in comparison.


I'd argue that Poudriere is a widely used tool when people want to maintain their own package repository. I don't see many people installing Poudriere just for the sake of installing a single port.

Got any arguments to back up your claim?



mtu said:


> The base system is meant to provide the tools needed for development, which is why it contains a compiler for example.


I'm actually not too sure about this to be honest.  From my point of view the only reason we have a compiler onboard is because src is a part of the whole distribution. When installing FreeBSD you can opt to have the Ports collection and source tree installed as well. But what good would those be without a compiler?

And installing ports or building your own setup does not imply development.



mtu said:


> The project also went to great pains to banish GPL'ed code from the base system. And yet it's practically impossible to do serious work on FreeBSD without git, which is GPLv2, and therefore cannot be part of the base system.


Huh? Ever heard of freebsd-update(8)? It's "only" the de-facto way to upgrade your system, no biggie. It even allows you to customize its settings where you can opt-out from specific parts such as the ports collection and/or the source tree (as well as world/games  )



mtu said:


> Now, there's always a voice that says "stop complaining and help out".


Here's a voice that says: stop throwing wild statements around without _any_ arguments to back up your claims.



mtu said:


> But the reality is that not everyone that likes to use FreeBSD is in a position to dedicate as much time and energy as is needed to make a difference.


It's not about making a difference in the first place, it's about trying to help out. "Making a difference" makes it sound like a popularity contest, _yech!_



mtu said:


> In any aspect of using FreeBSD, you can either do it the way most devs do, or be largely on your own.


What a load of nonsense.

What accounts for "being on your own" anyway?  I mean...  all the documentation is there, you only need to check it out, read and learn and then do things *your* way.

And I say this because, yah, I _seriously_ dislike both Poudriere and Synth alike, both can take a hike for all I care. _Don't read between the lines_: I have the utmost respect for both projects, however that doesn't change the fact that "me, myself, and *I*" dislikes both programs with a passion (heck.. this even triggered my edit ). But my dislike doesn't automagically make both environment useless. What about all those other people who do enjoy them?

So all I did was check up with the Portmaster documentation, I studied pkg and learned about pkg-repo(8) and that resulted in my own "Poudrier & Synth"-free setup.

I even wrote a guide about that:









						[Guide] Building a package repository with Portmaster
					

Hi gang!  Editorial  When it comes to maintaining a ports tree and setting up a package repository most people rely on software such as ports-mgmt/poudriere or ports-mgmt/synth. Interesting and definitely impressive projects for sure, but to be perfectly honest I never liked them myself. For the...




					forums.freebsd.org
				




FreeBSD is *NOT* about "either the devs way or the highway". Sorry, I personally find that comment borderline to insulting. But once again: that's me, myself and I again.

All the documentation you might need is already out there. And there are a lot of people working really hard to make sure all of it stays up to date.


Instead of throwing accusations around you'd actually make a difference if you'd provide examples. Tell us what documentation section is bad according to you, give us links and then we (= the _community_; those are all which matters) actually have something to work on. Then we can actually look at those links and improve them.

Wanted to make a difference, now's your chance...

(edit: sorry for temporarily b0rking the thread).


----------



## dal36 (Apr 9, 2021)

bagas said:


> Thanks. fixed gitup.conf.



Has this change propagated through yet? I'm still not seeing any changes to the ports tree through portsnap. Or perhaps I misunderand - is there an additional step that needs making to continue using portsnap?


----------



## bagas (Apr 9, 2021)

dal36 said:


> Has this change propagated through yet? I'm still not seeing any changes to the ports tree through portsnap.


What are you about?
Ports are now received through git.
Portsnap is no longer up to date.


----------



## dal36 (Apr 9, 2021)

I think I misunderstood your message - sorry about that.


bagas said:


> Thanks.
> fixed gitup.conf.


The first post on the thread indicated that portsnap should not be impacted. I think I'm seeing the same thing as in your separate thread, i.e. portsnap not updating the ports tree. I take it you didn't find a solution to continue using Portsnap?


SirDice said:


> Last SVN commit to the ports tree:
> 
> 
> 
> ...


----------



## bagas (Apr 9, 2021)

dal36 said:


> The first post on the thread indicated that portsnap should not be impacted. I think I'm seeing the same thing as in your separate thread, i.e. portsnap not updating the ports tree. I take it you didn't find a solution to continue using Portsnap?


Create an alias for the Portsnap command using the git in your environment.
But it's easier to use ports updates, git pull, or gitup ports.


----------



## olli@ (Apr 9, 2021)

bagas said:


> What are you about?
> Ports are now received through git.
> Portsnap is no longer up to date.


Portsnap will continue to be supported. It is deprecated, but it will continue to work for several years, probably. Note that FreeBSD 13 ships with portsnap, too (and it’s even still in 14-current).
Right now the portsnap data is still “frozen”. After the recent migration from Subversion to Git, the portsnap servers need to be adapted, too.  Just have a little patience.
 
*Edit:* Alternatively, if you want to make the switch right now, net/gitup is a lightweight, easy-to-use replacement that covers all the typical use-cases of portsnap. Unlike git, it has zero dependencies, has a BSD license, and requires no knowledge about version control systems in general or git in particular.


----------



## grahamperrin@ (Apr 9, 2021)

*Portsnap and new git ports tree*



dal36 said:


> … not seeing any changes to the ports tree through portsnap. Or perhaps …





grahamperrin said:


> … (I shouldn't treat https://wiki.freebsd.org/git#Ports_Schedule as fully comprehensive) …



<https://lists.freebsd.org/pipermail/freebsd-questions/2021-April/293732.html> (2021-04-07):

❝Work is in progress to convert the portsnap build infrastructure to git. The changes are just about ready for testing, but it will probably be a couple of days yet before it is deployed.❞

*FreeBSD mottos*



olli@ said:


> … For the past few years the motto has been “The Power to Serve”. …



I recognise the phrase, but I can't recall when I last saw it.

There's a mixture in the Everyday FreeBSD Brochure (2018-02-20). The Get Involved Brochure (2016-04-06) appears fine without a motto. Both items referred from <https://freebsdfoundation.org/our-work/education-advocacy/>.

*Overlaps*



mtu said:


> … almost no overlap between the users gathered on these Forums or on Reddit and the people working on FreeBSD. …





grahamperrin said:


> ‥ I spend far more time in Reddit than here because sadly, discussion of -CURRENT is forbidden.











						Please create a -current forum
					

Looks like folks would like to have a place to discuss experiences with the -current branch. Better here than on Reddit.  kpedersen convinced me that this is a bad idea here: https://forums.FreeBSD.org/threads/please-create-a-current-forum.79595/post-503475




					forums.freebsd.org
				




*Documentation*



mtu said:


> … Documentation is recognized to be important, but not a priority when implementing changes …



I get the impression that things are suitably prioritised but sometimes, it's simply not possible (or appropriate) to have an updated document until after the dust has settled around something.

Posted yesterday: 









						[Guide] Using Git to manage ports, source and documentation.
					

Hi gang!  Disclaimer: I am honestly a little excited about recent developments so expect to find some (small) opinionated parts in this guide. Nothing excessive mind you, but I can sometimes get a little carried away and despite some believes I never really plan guides like this.  Editorial  In...




					forums.freebsd.org


----------



## olli@ (Apr 9, 2021)

grahamperrin said:


> *FreeBSD mottos*
> 
> I recognise the phrase, but I can't recall when I last saw it.


Oh, that’s easy. It’s in the title banner at the top of www.freebsd.org.


----------



## grahamperrin@ (Apr 9, 2021)

Rocking. With. Laughter!


----------



## mtu (Apr 9, 2021)

ShelLuser said:


> Here's a voice that says: stop throwing wild statements around without any arguments to back up your claims.


Fair enough. My personal observations seem to be incomplete, or ill-informed, in a number of ways.



ShelLuser said:


> Lessee now...  ports(7) which lists all build targets (you _do_ realize that manual pages are actual manual pages on FreeBSD, right?), chapter 4.5 of the FreeBSD handbook (which even mentions Portmaster for crying out loud!) which obviously brings me to: portmaster(8) (remember my comment about manual pages?).
> 
> And about those binary packages: chapter 4.4 of said handbook.
> 
> This also brings us to pkg(8) which can refer you to pkg-install(8) and/or pkg-repo(8), if not from the command summary then surely from the SEE ALSO section.


I'd argue that one can run into sticky situations (by experimentation and/or stupidity) that cannot be resolved by just reading the docs. But yes, it can help greatly to know them, and you're right: there _is_ a lot of helpful documentation.

I guess when I was learning to use ports and occasionally fumbled it, I let "don't know, just use poudriere" statements on IRC get to me.



ShelLuser said:


> So that's why Git got mentioned in the handbook _long_ before the whole transition had been finished.


Yes, there _was_ a lot of documentation work. I was drawing incorrect conclusions from discussions here on the Forums.



ShelLuser said:


> mtu said:
> 
> 
> > [*]The base system is meant to provide the tools needed for development, which is why it contains a compiler for example. The project also went to great pains to banish GPL'ed code from the base system. And yet it's practically impossible to do serious work on FreeBSD without git, which is GPLv2, and therefore cannot be part of the base system.
> ...


That's true. I should have said "impossible to enter into development without git", because "serious work" can be anything. And yes, (re-)compiling the base system is a legitimate use for a compiler in base apart from development.



ShelLuser said:


> What accounts for "being on your own" anyway?  I mean...  all the documentation is there, you only need to check it out, read and learn and then do things *your* way.


In a world with unlimited time+energy for getting such things done, that's absolutely correct. Real-world constraints can, however, can make certain software solutions impractical, even if technically possible. By "being on your own", I meant: "not having anyone to consult with, or records of that thing having been done before".

Which, apparently, is exactly what happened to you when you …


ShelLuser said:


> So all I did was check up with the Portmaster documentation, I studied pkg and learned about pkg-repo(8) and that resulted in my own "Poudrier & Synth"-free setup.
> 
> I even wrote a guide about that:
> 
> ...


Seriously, great work. I'll be sure to study that and see if it can replace poudriere for my setup.



ShelLuser said:


> FreeBSD is *NOT* about "either the devs way or the highway". Sorry, I personally find that comment borderline to insulting. But once again: that's me, myself and I again.



See above. I didn't mean to say: "another way of using the system is impossible/forbidden/discouraged".

Just that it's probably harder, more time consuming, and lonesome – which can make it impractical for many people.



ShelLuser said:


> Instead of throwing accusations around you'd actually make a difference if you'd provide examples. Tell us what documentation section is bad according to you, give us links and then we (= the _community_; those are all which matters) actually have something to work on. Then we can actually look at those links and improve them.


Fair point. One little paragraph in the 13.0 Releaes Notes is my work, more to come.


----------



## ShelLuser (Apr 9, 2021)

A like alone is not enough in my opinion, so...



mtu said:


> I'd argue that one can run into sticky situations (by experimentation and/or stupidity) that cannot be resolved by just reading the docs. But yes, it can help greatly to know them, and you're right: there _is_ a lot of helpful documentation.


I can agree with your argument here as well, and I also want to applaud you for raising it.



mtu said:


> Seriously, great work. I'll be sure to study that and see if it can replace poudriere for my setup.


Feel free to drop me a PM if you want some help with that. I cannot guarantee a quick response (I'm on and off so to speak) but I _can_ promise that I won't brush it off as I normally do with PM's asking for help. Fair disclaimer (no offense): I do need to limit this to the whole "package repository" setup, not because I don't trust you but because I don't know you.  (= ask me about this topic in private and I'll help in private, ask me about how to build FreeBSD from source and I'll politely direct you to the appropriate documentation or suggest you start a public topic).



mtu said:


> See above. I didn't mean to say: "another way of using the system is impossible/forbidden/discouraged".


For what's it worth I gained more respect for your stance. I still don't agree with your opinion (obviously) but I also think you raised some good points.

(edit)



grahamperrin said:


> Posted yesterday:


Bear in mind that I am not part of any official team or project or such. What you see there is me, as a person, trying to make a better place of this forum. It has nothing to do with the FreeBSD documentation project.

I _really_ appreciate the acknowledgement, but it does not counter mtu 's statement within their context.


----------



## grahamperrin@ (Apr 10, 2021)

Thanks,


ShelLuser said:


> … nothing to do with the FreeBSD documentation project. …


– understood. It was a generic *Documentation* subheading (not to imply FreeBSD Documentation Project) but I see that it could have been taken out of context.

OT, the current list of projects needs some love … and the 1–2–3 for submitting changes could be made more broadly attractive by *not* using vi(1) as the example editor.


----------



## grahamperrin@ (Apr 10, 2021)

omarsidd said:


> … announcing) major changes like this. …


If you have not already done so, please subscribe to *freebsd-announce* – _messages deemed important to *ALL* FreeBSD users_. Thanks.

Four months ago: [FreeBSD-Announce] FreeBSD-doc Subversion to Git migration – "… the first transition from Subversion to Git, …"; ports is the third.


----------



## zirias@ (Apr 10, 2021)

olli@ said:


> Portsnap will continue to be supported. It is deprecated, but it will continue to work for several years, probably. Note that FreeBSD 13 ships with portsnap, too (and it’s even still in 14-current).


I'd put it that way: It's scheduled for deprecation, at least, that's what I take from the "heads up" posted on freebsd-ports@ by portmgr@. To make the deprecation official, I agree with you, at least an official announcement would be necessary. All we can take for now is that deprecation (and later removal) probably will happen eventually.

But indeed, for now, portsnap is the documented method for "end users" to keep their ports tree up to date, so, nothing wrong with using it.


----------



## grahamperrin@ (Apr 10, 2021)

`portsnap update` does not yet provide the most recent updates (post-transition updates, I guess): 


```
root@mowa219-gjp4-8570p:~ # date && grep coturn /usr/ports/MOVED && portsnap update && grep coturn /usr/ports/MOVED && freebsd-version -kru && gitup ports && grep coturn /usr/ports/MOVED
Sat Apr 10 15:31:02 BST 2021
root@mowa219-gjp4-8570p:~ # date && grep coturn /usr/ports/MOVED ; portsnap update && grep coturn /usr/ports/MOVED ; freebsd-version -kru && gitup ports && grep coturn /usr/ports/MOVED
Sat Apr 10 15:31:26 BST 2021
Ports tree is already up to date.
14.0-CURRENT
14.0-CURRENT
14.0-CURRENT
# Host: git.freebsd.org
# Port: 443
# Repository: /ports.git
# Target: /usr/ports
# Want: 67227bea67da9c164fefcf30fa8b540e2b080f46
# Branch: main
# Action: clone
 * /usr/ports/.gitignore
 * /usr/ports/CHANGES
…
 * /usr/ports/MOVED
…
 - /usr/ports/x11/xmon/pkg-plist
net/coturn|net/turnserver|2021-04-09|Remove duplicate port: coturn is another name for turnserver
root@mowa219-gjp4-8570p:~ # date 
Sat Apr 10 15:49:37 BST 2021
root@mowa219-gjp4-8570p:~ #
```

If the local copy was truly up-to-date, then grep(1) should have found _coturn_ (see <https://cgit-dev.freebsd.org/ports/commit/MOVED?id=b91f6147e4171425696f74c1f26de83a6ea3593a>).

I'm in no rush. Recalling: 



grahamperrin said:


> *Portsnap and new git ports tree*
> 
> …
> 
> ...



– <https://lists.freebsd.org/pipermail/freebsd-questions/2021-April/thread.html#293732> there's not yet a follow up to Ed Maste's e-mail.


----------



## dR3b (Apr 12, 2021)

Zirias said:


> ```
> 16:28:35 <@lwhsu> Zirias: it will be addressed soon
> ```


May I please ask: What is his definition of: "soon"? The Branch still does not exist.

[1] https://github.com/freebsd/freebsd-ports


----------



## YuryG (Apr 12, 2021)

Will the source tree /usr/src migrate to Git from SVN? (It hasn't yet, has it?)


----------



## drhowarddrfine (Apr 12, 2021)

YuryG It has and, iirc, was one of the first.


----------



## grahamperrin@ (Apr 12, 2021)

YuryG said:


> Will the source tree /usr/src migrate to Git from SVN? …


Above:


grahamperrin said:


> … [FreeBSD-Announce] FreeBSD-doc Subversion to Git migration – "… the first transition from Subversion to Git, …"; ports is the third.


– src was the second.

From page 1:


Zirias said:


> … git - FreeBSD Wiki


----------



## SirDice (Apr 12, 2021)

Note that the source trees for 11 and 12 will remain maintained in SVN until their demise. They are also available on git though. From 13.0 onward the source is only available through git.


----------



## Itsacon (Apr 13, 2021)

I'm reading the remarks about portsnap being deprecated in a future version with great fear and amazement.

Requiring something from the ports tree in order to _fetch_ the ports tree is, frankly, a good way to scare away new users. (and is generally called a catch-22)

For multiple reasons, I never use packages, building everything from source. The only exception to this is pkg itself, which I often immediately rebuild after bootstrapping it, just to make sure. We all know that mixing self-build ports and pre-built packages is generally a recipe for disaster.

For this reason, any new install I do generally gets these commands first:
`freebsd-update fetch install
portsnap fetch extract`

After which I install whatever I want for that specific machine. 

When I want to use a box for dev purposes, git is something that'll be on there, but for the umpteen webservers I maintain, having git installed, along with all its dependencies, is a security risk and nothing more. Something like gitup would be a decent alternative, the speed penalty described in this thread doesn't bother me, if it gets too bad, it goes in a nightly cron job. 

_*But it needs to be in base*_. If that means writing opengit, freegit or libregit so we can have a proper license, then that needs to be written. 

And such a tool should preferably have an interface that removes the repository-related terminology from it. That's the power of portsnap: a normal end-user doesn't have to know or care where his sources are coming from: binaries, git, svn or Microsoft Sourcetree. He just want the latest version. 
`portsnap fetch update` is much clearer to a non-developer than `git clone`.

If there is noone who is capable or willing to take that project up (I have neither the time nor the skills to do so, unfortunately), then quite simply, portsnap can't really be deprecated, as there is no viable alternative.

From Wikipedia:


> In several fields, deprecation is the discouragement of use of some terminology, feature, design, or practice, typically *because it has been superseded*



Until we have a git-based tool to fetch and update the ports-tree in base, portsnap hasn't been superseded by anything, so it can't really be regarded as deprecated.

And before someone says 'it's free software' or 'do it yourself': When I say: 'it needs to be done', I don't mean 'someone has to do it or I'll be mad'. I'm a dev, I'll manage fine without those tools, even if I won't like it. (not to mention that even if I were to write something, I don't have the authority to put it in base)


What I mean is : 'it needs to be done or we'll scare off new users'.

Thank you for reading.


----------



## tingo (Apr 13, 2021)

Don't worry; if / when portsnap disappears, it will have been replaced with some other tool that allows you to do the same thing.
People who has been using FreeBSD long enough have been through a few of these transitions, so we sort of know how it is going to play out.


----------



## Argentum (Apr 13, 2021)

Itsacon said:


> When I want to use a box for dev purposes, git is something that'll be on there, but for the umpteen webservers I maintain, having git installed, along with all its dependencies, is a security risk and nothing more. Something like gitup would be a decent alternative, the speed penalty described in this thread doesn't bother me, if it gets too bad, it goes in a nightly cron job.
> 
> _*But it needs to be in base*_. If that means writing opengit, freegit or libregit so we can have a proper license, then that needs to be written.


Personally, I have moved my machines to net/gitup and must confess that it is working. `gitup ports` does the same thing as `portsnap fetch update` did.


----------



## SirDice (Apr 13, 2021)

Itsacon said:


> Requiring something from the ports tree in order to _fetch_ the ports tree is, frankly, a good way to scare away new users. (and is generally called a catch-22)


When I started with FreeBSD many, many years ago, the only way to get the ports tree was by using CVS. In fact, the only way to update the OS (even -RELEASE versions) was by checking out the source and building it. There was no freebsd-update(8), there was no portsnap(8). The old package format was horrible to use, you needed additional tools in order to get dependencies correct and yes, you had to install those separately from the OS. Yet, I still managed. We all did. New and old users alike. 



Itsacon said:


> which I often immediately rebuild after bootstrapping it, just to make sure.


Make sure of what exactly?



Itsacon said:


> When I want to use a box for dev purposes, git is something that'll be on there, but for the umpteen webservers I maintain, having git installed, along with all its dependencies, is a security risk and nothing more.


Managing a bunch of servers through ports is not very efficient and rather error prone. It'll also easily lead to configuration differences and you get a whole bunch of, otherwise useless, build dependencies. How's that for a security risk? Set up your own repository, then you can build from ports and have the benefits of that (options, default versions, etc) while keeping the ease of management of packages. It might take you a bit of time to set it all up but once you do you will wonder why you haven't done it sooner.


----------



## Itsacon (Apr 13, 2021)

Argentum said:


> Personally, I have moved my machines to net/gitup and must confess that it is working. `gitup ports` does the same thing as `portsnap fetch update` did.



But how do you get gitup on a fresh system? If gitup was to become part of base, I'd be fine with that, but right now, it's not.

(Yes, you can get it through pkg, but if you're using pkg in the first place, why do you need the ports tree?)



SirDice said:


> When I started with FreeBSD many, many years ago, the only way to get the ports tree was by using CVS. In fact, the only way to update the OS (even -RELEASE versions) was by checking out the source and building it. There was no freebsd-update(8), there was no portsnap(8). The old package format was horrible to use, you needed additional tools in order to get dependencies correct and yes, you had to install those separately from the OS. Yet, I still managed. We all did. New and old users alike.


I started back in FreeBSD 3, I know what you mean. If I remember correctpy, csup or cvsup was part of base at some point, since you needed it to upgrade world, for example. We've come a long way, that's for sure. That's why I hope this doesn't become a step backwards from a usability perspective. I have fond memories of rebuilding world, but freebsd-update is infinitely more user friendly...



SirDice said:


> Managing a bunch of servers through ports is not very efficient and rather error prone. It'll also easily lead to configuration differences and you get a whole bunch of, otherwise useless, build dependencies. How's that for a security risk? Set up your own repository, then you can build from ports and have the benefits of that (options, default versions, etc) while keeping the ease of management of packages. It might take you a bit of time to set it all up but once you do you will wonder why you haven't done it sooner.


I know, I know. Problem is that the servers are spread over many locations and datacenters, so it would also require a (secure yet efficient) way to distribute the packages. And one of the reasons I don't use packages is that many of the servers have specific (differing) version requirements.  I'll get around to it solving that problem someday. Hopefully.  

As for build dependencies; that's what pkg autoremove is for...


----------



## olli@ (Apr 13, 2021)

Itsacon said:


> I'm reading the remarks about portsnap being deprecated in a future version with great fear and amazement.


 Beastie says:

Many people are afraid of change. But change is the only way to progress.
Many people are afraid of the unknown. But there is a simple remedy for this: learning.
 


> Requiring something from the ports tree in order to _fetch_ the ports tree is, frankly, a good way to scare away new users. (and is generally called a catch-22)


That is _not_ correct.
 
First, daily snapshots of the ports tree are made available as compressed tarballs (independent of portsnap) via FTP and HTTP(S), so you can use ftp(1) or fetch(1) to download an initial copy of the ports tree if you need to. Both of these tools are in FreeBSD’s base system.
 
Second, portsnap – even though deprecated – will probably stay for several years, and when it is finally removed, It’s safe to assume that there will be a replacement ready for use. That might be gitup (net/gitup) or something else yet to be written. Personally I think gitup would be perfect to go into base. It is BSD-licensed, dependency-free, and as easy to use as portsnap, while it supports more things than portsnap (i.e. it can download quarterly branches, and you can also use it to download a src or doc tree). Replacing portsnap with gitup is almost a no-brainer.


----------



## Itsacon (Apr 13, 2021)

olli@ said:


> Beastie says:
> 
> Many people are afraid of change. But change is the only way to progress.
> Many people are afraid of the unknown. But there is a simple remedy for this: learning.


As I tried to point out: I'm not afraid of change. I'm afraid of all this talk of portsnap already being deprecated, despite there being no valid replacement. 

I'm fully onboard with replacing portsnap with something new, if needed. But that replacement should have been in base for at least a full version, in my opinion, before removing portsnap.



olli@ said:


> That is _not_ correct.
> 
> First, daily snapshots of the ports tree are made available as compressed tarballs (independent of portsnap) via FTP and HTTP(S), so you can use ftp(1) or fetch(1) to download an initial copy of the ports tree if you need to. Both of these tools are in FreeBSD’s base system.


But that snapshot can then not be reliably used as a base for either portsnap or gitup. 
If you install `/usr/ports` from a tarball (or installation media), and then run `portsnap fetch update`, it will tell you that the ports tree wasn't created using portsnap, and to use `portsnap extract` instead. 

I haven't used gitup yet, but knowing git, cloning a repo into a directory that already contains a different but similar set of data (without the git info) is probably not a good way to get a stable repository.

So basically to use gitup you need to download the whole ports tree twice. Which can't be the intention.



olli@ said:


> Second, portsnap – even though deprecated – will probably stay for several years, and when it is finally removed, It’s safe to assume that there will be a replacement ready for use. That might be gitup (net/gitup) or something else yet to be written. Personally I think gitup would be perfect to go into base. It is BSD-licensed, dependency-free, and as easy to use as portsnap, while it supports more things than portsnap (i.e. it can download quarterly branches, and you can also use it to download a src or doc tree). Replacing portsnap with gitup is almost a no-brainer.



That's all I'm asking for


----------



## SirDice (Apr 13, 2021)

Itsacon said:


> Problem is that the servers are spread over many locations and datacenters, so it would also require a (secure yet efficient) way to distribute the packages.


Put them on a VPS or some other server that's accessible over the internet. The server where you have to build ports will need to fetch their distfiles too so I assume it's not that problematic to have them fetch packages from a internet accessible host. If you sign your packages with a key you can be fairly sure it's your verified packages they're fetching. 


Itsacon said:


> And one of the reasons I don't use packages is that many of the servers have specific (differing) version requirements.


A lot can already be archived through flavors. And when that's not good enough you can create multiple repositories quite easily (at least you can with poudriere). For a client I've set this up ages ago. We're currently migrating from the old Ruby 2.5 to Ruby 2.6 or 2.7 for example. So I have three repositories, one built for Ruby 2.5, one for 2.6 and one for 2.7. If I need to "switch" one of the machines to one version or another I simply change the repository and let pkg-upgrade(8) automatically figure it out. Only takes me a few minutes for each machine. Best of all, I'm in control of what, when and how.


----------



## olli@ (Apr 13, 2021)

Itsacon said:


> As I tried to point out: I'm not afraid of change. I'm afraid of all this talk of portsnap already being deprecated, despite there being no valid replacement.
> 
> I'm fully onboard with replacing portsnap with something new, if needed. But that replacement should have been in base for at least a full version, in my opinion, before removing portsnap.


That would be five years (FreeBSD supports a stable branches for a period of five years). That’s too much to ask for, in my opinion.
Currently it seems that portsnap will continue to be supported for the lifetime of the stable/13 branch that has just been created not very long ago, so that will be five years from now. That’s really plenty of time to put, say, gitup or something else in base.
 


> But that snapshot can then not be reliably used as a base for either portsnap or gitup.
> […]
> So basically to use gitup you need to download the whole ports tree twice. Which can't be the intention.


So the situation at least doesn’t get worse, since it’s not different with portsnap either, right? 
 
Apart from that, a compressed tarball of the ports collection isn’t _that_ large, so it’s really not a big deal. The current ports tree is just 60 MB.


----------



## tuaris (Apr 14, 2021)

It looks like something broke on (at least with my Poudriere) with the transition to Git.  I am now getting a weird failure when trying to do a bulk build (with the -a option).


```
[00:00:01] Creating the reference jail... done
[00:00:03] Mounting system devices for 12amd64-weekly-desktop
[00:00:03] Mounting ports/packages/distfiles
[00:00:03] Stashing existing package repository
[00:00:04] Mounting ccache from: /usr/local/poudriere/data/ccache/12amd64
[00:00:04] Mounting packages from: /usr/local/poudriere/data/packages/12amd64-weekly-desktop
[00:00:04] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:00:04] Appending to make.conf: /usr/local/etc/poudriere.d/desktop-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/12amd64-weekly-desktop/ref/etc/resolv.conf
[00:00:04] Starting jail 12amd64-weekly-desktop
[00:00:13] Logs: /usr/local/poudriere/data/logs/bulk/12amd64-weekly-desktop/2021-04-14_02h23m35s
[00:00:13] WWW: http://pkg.morante.net/poudriere/build.html?mastername=12amd64-weekly-desktop&build=2021-04-14_02h23m35s
[00:00:13] Loading MOVED for /usr/local/poudriere/data/.m/12amd64-weekly-desktop/ref/usr/ports
[00:00:15] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:15] Gathering ports metadata
[00:00:22] Warning: (databases/R-cran-DBI): Error: deps_fetch_vars: databases/R-cran-DBI already known as R-cran-DBI-1.1.1
[00:00:22] Warning: (databases/R-cran-RMySQL): Error: deps_fetch_vars: databases/R-cran-RMySQL already known as R-cran-RMySQL-0.10.21
[00:00:22] Warning: (databases/R-cran-RPostgreSQL): Error: deps_fetch_vars: databases/R-cran-RPostgreSQL already known as R-cran-RPostgreSQL-0.6.2_3
[00:00:27] Warning: (deskutils/affiche): Error: deps_fetch_vars: deskutils/affiche already known as affiche-0.6.0_11
[00:00:27] Warning: (deskutils/akonadi-calendar-tools): Error: deps_fetch_vars: deskutils/akonadi-calendar-tools already known as akonadi-calendar-tools-20.12.3
[00:00:27] Warning: (deskutils/akonadi-import-wizard): Error: deps_fetch_vars: deskutils/akonadi-import-wizard already known as akonadi-import-wizard-20.12.3
[00:00:29] Warning: (devel/9base): Error: gather_port_vars_port: Already had devel/9base (rdep=listed)
[00:00:29] Warning: (devel/ChipmunkPhysics): Error: deps_fetch_vars: devel/ChipmunkPhysics already known as ChipmunkPhysics-7.0.1_2
[00:00:29] Warning: (devel/ElectricFence): Error: deps_fetch_vars: devel/ElectricFence already known as electricfence-2.2.2_2
[00:01:08] Warning: (emulators/advancemame): Error: deps_fetch_vars: emulators/advancemame already known as advancemame-1.4_3
[00:01:08] Warning: (emulators/adamem): Error: deps_fetch_vars: emulators/adamem already known as adamem-1.0_4
[00:01:08] Warning: (emulators/advancemenu): Error: deps_fetch_vars: emulators/advancemenu already known as advancemenu-2.8_3
[00:01:25] Warning: (irc/bip): Error: deps_fetch_vars: irc/bip already known as bip-0.9.0.r4
[00:01:25] Warning: (irc/atheme-services): Error: deps_fetch_vars: irc/atheme-services already known as atheme-services-7.2.9
[00:01:25] Warning: (irc/anope): Error: deps_fetch_vars: irc/anope already known as anope-2.0.9_1
[00:02:02] Warning: (ports-mgmt/bsdadminscripts2): Error: gather_port_vars_port: Already had ports-mgmt/bsdadminscripts2 (rdep=listed)
[00:02:02] Warning: (ports-mgmt/chucky): Error: deps_fetch_vars: ports-mgmt/chucky already known as chucky-1.0_1
[00:02:02] Warning: (ports-mgmt/caronade): Error: gather_port_vars_port: Already had ports-mgmt/caronade (rdep=listed)
[00:02:14] Warning: (sysutils/3mux): Error: gather_port_vars_port: Already had sysutils/3mux (rdep=listed)
[00:02:14] Warning: (sysutils/3dm): Error: deps_fetch_vars: sysutils/3dm already known as 3dm-2.11.00.021_4,1
[00:02:14] Warning: (sysutils/44bsd-more): Error: deps_fetch_vars: sysutils/44bsd-more already known as 44bsd-more-20000521_1
[00:02:38] Warning: (www/R-cran-RgoogleMaps): Error: deps_fetch_vars: www/R-cran-RgoogleMaps already known as R-cran-RgoogleMaps-1.4.5.3_1
[00:02:38] Warning: (www/R-cran-bslib): Error: deps_fetch_vars: www/R-cran-bslib already known as R-cran-bslib-0.2.4
[00:02:38] Warning: (www/R-cran-Rook): Error: deps_fetch_vars: www/R-cran-Rook already known as R-cran-Rook-1.1.1_4
[00:02:53] Warning: (x11/3ddesktop): Error: deps_fetch_vars: x11/3ddesktop already known as 3ddesktop-0.2.9_14
[00:02:53] Warning: (x11/9menu): Error: gather_port_vars_port: Already had x11/9menu (rdep=listed)
[00:02:53] Warning: (x11/9box): Error: deps_fetch_vars: x11/9box already known as 9box-0.2.1_3
[00:03:00] Warning: (x11-themes/Kvantum): Error: deps_fetch_vars: x11-themes/Kvantum already known as Kvantum-qt5-0.19.0
[00:03:00] Warning: (x11-themes/adapta-gtk-theme): Error: deps_fetch_vars: x11-themes/adapta-gtk-theme already known as adapta-gtk-theme-3.95.0.11_1
[00:03:00] Warning: (x11-themes/adapta-backgrounds): Error: deps_fetch_vars: x11-themes/adapta-backgrounds already known as adapta-backgrounds-0.5.2.3
[00:03:03] Error: Fatal errors encountered gathering initial ports metadata
[00:03:03] Cleaning up
[00:03:04] Unmounting file systems
```

My port trees:


```
# poudriere ports -l
PORTSTREE  METHOD    TIMESTAMP           PATH
default    portsnap  2019-01-08 23:54:32 /usr/local/poudriere/ports/default
monthly    portsnap  2020-07-07 20:11:39 /usr/local/poudriere/ports/monthly
weekly     svn+http  2021-02-19 04:04:42 /usr/local/poudriere/ports/weekly
```

I deleted "weekly" and recreated it as git:


```
# poudriere ports -l
PORTSTREE  METHOD    TIMESTAMP           PATH
default    portsnap  2019-01-08 23:54:32 /usr/local/poudriere/ports/default
monthly    portsnap  2020-07-07 20:11:39 /usr/local/poudriere/ports/monthly
weekly     git+https 2021-04-14 01:35:19 /usr/local/poudriere/ports/weekly
```

I also updated the other port trees for the first time since the move to Git.  They are now failing with the same problem.  For example I try to use the tree named "monthly":


```
[00:00:00] Creating the reference jail... done
[00:00:02] Mounting system devices for 12amd64-monthly-desktop
[00:00:02] Mounting ports/packages/distfiles
[00:00:03] Stashing existing package repository
[00:00:19] Mounting ccache from: /usr/local/poudriere/data/ccache/12amd64
[00:00:19] Mounting packages from: /usr/local/poudriere/data/packages/12amd64-monthly-desktop
[00:00:19] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:00:19] Appending to make.conf: /usr/local/etc/poudriere.d/desktop-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/12amd64-monthly-desktop/ref/etc/resolv.conf
[00:00:19] Starting jail 12amd64-monthly-desktop
[00:00:19] Logs: /usr/local/poudriere/data/logs/bulk/12amd64-monthly-desktop/2021-04-14_02h50m52s
[00:00:19] WWW: http://pkg.morante.net/poudriere/build.html?mastername=12amd64-monthly-desktop&build=2021-04-14_02h50m52s
[00:00:19] Loading MOVED for /usr/local/poudriere/data/.m/12amd64-monthly-desktop/ref/usr/ports
[00:00:21] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:21] Gathering ports metadata
[00:00:27] Warning: (databases/R-cran-DBI): Error: deps_fetch_vars: databases/R-cran-DBI already known as R-cran-DBI-1.1.1
[00:00:27] Warning: (databases/R-cran-RMySQL): Error: deps_fetch_vars: databases/R-cran-RMySQL already known as R-cran-RMySQL-0.10.21
[00:00:27] Warning: (databases/R-cran-RPostgreSQL): Error: deps_fetch_vars: databases/R-cran-RPostgreSQL already known as R-cran-RPostgreSQL-0.6.2_3
[00:00:30] Warning: (deskutils/affiche): Error: deps_fetch_vars: deskutils/affiche already known as affiche-0.6.0_11
[00:00:30] Warning: (deskutils/akonadi-import-wizard): Error: deps_fetch_vars: deskutils/akonadi-import-wizard already known as akonadi-import-wizard-20.12.3
[00:00:30] Warning: (deskutils/akonadi-calendar-tools): Error: deps_fetch_vars: deskutils/akonadi-calendar-tools already known as akonadi-calendar-tools-20.12.3
[00:00:30] Warning: (devel/9base): Error: deps_fetch_vars: devel/9base already known as 9base-20170701
[00:00:30] Warning: (devel/ElectricFence): Error: deps_fetch_vars: devel/ElectricFence already known as electricfence-2.2.2_2
[00:00:30] Warning: (devel/ChipmunkPhysics): Error: deps_fetch_vars: devel/ChipmunkPhysics already known as ChipmunkPhysics-7.0.1_2
[00:00:48] Warning: (emulators/adamem): Error: deps_fetch_vars: emulators/adamem already known as adamem-1.0_4
[00:00:48] Warning: (emulators/advancemenu): Error: deps_fetch_vars: emulators/advancemenu already known as advancemenu-2.8_3
[00:00:48] Warning: (emulators/advancemame): Error: deps_fetch_vars: emulators/advancemame already known as advancemame-1.4_3
[00:00:56] Warning: (irc/bip): Error: deps_fetch_vars: irc/bip already known as bip-0.9.0.r4
[00:00:56] Warning: (irc/atheme-services): Error: deps_fetch_vars: irc/atheme-services already known as atheme-services-7.2.9
[00:00:56] Warning: (irc/anope): Error: deps_fetch_vars: irc/anope already known as anope-2.0.9_1
[00:01:13] Warning: (ports-mgmt/bsdadminscripts2): Error: deps_fetch_vars: ports-mgmt/bsdadminscripts2 already known as bsdadminscripts2-0.3.0
[00:01:13] Warning: (ports-mgmt/chucky): Error: deps_fetch_vars: ports-mgmt/chucky already known as chucky-1.0_1
[00:01:13] Warning: (ports-mgmt/caronade): Error: deps_fetch_vars: ports-mgmt/caronade already known as caronade-0.4.0
[00:01:19] Warning: (sysutils/3dm): Error: deps_fetch_vars: sysutils/3dm already known as 3dm-2.11.00.021_4,1
[00:01:19] Warning: (sysutils/44bsd-more): Error: deps_fetch_vars: sysutils/44bsd-more already known as 44bsd-more-20000521_1
[00:01:19] Warning: (sysutils/3mux): Error: deps_fetch_vars: sysutils/3mux already known as 3mux-1.1.0
[00:01:29] Warning: (www/R-cran-RgoogleMaps): Error: deps_fetch_vars: www/R-cran-RgoogleMaps already known as R-cran-RgoogleMaps-1.4.5.3_1
[00:01:29] Warning: (www/R-cran-Rook): Error: deps_fetch_vars: www/R-cran-Rook already known as R-cran-Rook-1.1.1_4
[00:01:29] Warning: (www/R-cran-bslib): Error: deps_fetch_vars: www/R-cran-bslib already known as R-cran-bslib-0.2.4
[00:01:37] Warning: (x11/9menu): Error: deps_fetch_vars: x11/9menu already known as 9menu-1.10
[00:01:37] Warning: (x11/9box): Error: deps_fetch_vars: x11/9box already known as 9box-0.2.1_3
[00:01:37] Warning: (x11/3ddesktop): Error: deps_fetch_vars: x11/3ddesktop already known as 3ddesktop-0.2.9_14
[00:01:40] Warning: (x11-themes/adapta-backgrounds): Error: deps_fetch_vars: x11-themes/adapta-backgrounds already known as adapta-backgrounds-0.5.2.3
[00:01:40] Warning: (x11-themes/Kvantum): Error: deps_fetch_vars: x11-themes/Kvantum already known as Kvantum-qt5-0.19.0
[00:01:40] Warning: (x11-themes/adapta-gtk-theme): Error: deps_fetch_vars: x11-themes/adapta-gtk-theme already known as adapta-gtk-theme-3.95.0.11_1
[00:01:41] Error: Fatal errors encountered gathering initial ports metadata
[00:01:41] Cleaning up
[00:01:42] Unmounting file systems
```


----------



## pantos (Apr 14, 2021)

I have the same metadata problem using portsnap:


```
portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Wed Apr 14 02:09:41 CEST 2021:
0b344cd7869783c8d72504314f54f27b9ac6225ce21fae          88 MB  161 kBps 09m23s
Extracting snapshot... done.
Verifying snapshot integrity... done.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Apr 14 02:09:41 CEST 2021 to Wed Apr 14 09:06:50 CEST 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... /usr/sbin/portsnap: cannot open a26ff8dcc066fedb59483514c8c5b17b8de665b8508bf0e74c1a171bf6c53c55.gz: No such file or directory
metadata is corrupt.
```

Any ideas how to solve this?


----------



## tuaris (Apr 14, 2021)

pantos said:


> I have the same metadata problem using portsnap:
> 
> 
> ```
> ...


I've run into that every now and then.  Just delete the portsnap cache and try again.
Something like 

`mv /var/db/portsnap /var/db/portsnap.broken
portsnap fetch extract`


----------



## pantos (Apr 14, 2021)

tuaris said:


> I've run into that every now and then.  Just delete the portsnap cache and try again.
> Something like
> 
> `mv /var/db/portsnap /var/db/portsnap.broken
> portsnap fetch extract`


I tried this, but the error persists.

*EDIT*: After numerous attempts, it workes now...


----------



## SirDice (Apr 14, 2021)

portsnap fetch error
					

Hi I tried doing a portsnap fetch and I get the following error: root@bsdcompile:~ # portsnap fetch Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found. Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done. Fetching snapshot metadata... done. Updating from Wed Mar 31 14:10:16...




					forums.FreeBSD.org


----------



## Deleted member 30996 (Apr 14, 2021)

pantos said:


> I have the same metadata problem using portsnap:
> Any ideas how to solve this?


I nagged it to death and kept running `portsnap fetch update` and and `portsnap fetch extract` till that command downloaded a snapshot update of close to 30,000 patches. Then it was back to normal operation.

I updated this one last night with all the port breakage in the process detailed now fixed.


----------



## grahamperrin@ (Apr 17, 2021)

Re: <https://lists.freebsd.org/pipermail/freebsd-questions/2021-April/293732.html>



grahamperrin said:


> `portsnap update` does not yet provide the most recent updates …



Two days ago, that was still true (with a test machine): 






This evening, things seem to be better.


----------

