# gitup



## jmehr (Nov 27, 2020)

Hello everyone,

With the upcoming transition from Subversion to Git, I've been plugging away at building "gitup", a replacement for net/svnup and I'm happy to report that I've got a working "git clone" prototype.

Looking back at when I was writing net/svnup, one of the things I wished I had done differently was I didn't really engage the community in the process.  So, as I start working on adding "git pull" functionality, I'd like to reach out to all of you for your thoughts and feedback about how I can make it best meet your needs.

For example, are there things about the git clone/pull process that you wish worked differently?  If you've used net/svnup, are there things about it that you wish worked differently?  If you tried using net/svnup and decided it wasn't for you, what could I do to fix that?  Is your environment resource constrained and, if you don't have a lot of memory or disk space, how can gitup best accomodate your device's specifics?

Constructive criticism is welcome and appreciated.  Thanks!


----------



## SirDice (Nov 27, 2020)

This is too on-topic to be hiding in off-topic  
Was a little split between "Porting new software" and "Userland programming and scripting". But as this isn't about the port or porting process but the actual code itself "Userland.." seems more appropriate.


----------



## bsdimp (Nov 30, 2020)

Sounds super cool.


----------



## jmehr (Dec 7, 2020)

I've implemented "git pull" functionality and the gitup prototype is now ready for wider testing.  If you're interested, you can find the source code at:

https://github.com/johnmehr/gitup

Much like its predecessor net/svnup, gitup is intended for non-developers
who just need to synchronize a local repository without the additional
overhead required by the official git client and is not intended as a
feature-rich, drop-in replacement for it.

The code is beta, so the usual caveats/warnings (don't use it in a
production environment, make sure you've got backups, don't run as
root) apply.

I think I've got most of the common use cases taken care of but there's
always room for improvement so please don't hesitate to offer suggestions,
comments and/or constructive criticism.  Thanks!


----------



## gnath (Feb 7, 2021)

Just for hands on I have tried to use "gitup ports" (forum codes are not working) without any problem. But there is error for making any ports. The old ports tree was fine with 'portsnap'. What else required to use this new ports tree ?
Portmaster also give the error "make: "/usr/ports/Mk/Uses/compiler.mk" line 78: warning: " --version" returned non-zero status"


----------



## unitrunker (Feb 7, 2021)

I think portsnap always grabbed the latest ports tree - which was a problem for folks using quarterly pkg-s. It would be nice if this new tool used the same repo as pkg (quarterly or latest).


----------



## jmehr (Feb 7, 2021)

When you have a moment, could you please run `cat /usr/ports/.gituprevision` and let me know what commit you pulled?  You should see something like master:e5260f3eb.

You can track the quarterly ports branches by running `gitup quarterly` which should seamlessly follow the most recently available quarterly branch (I've only had one quarterly branch change to test and that one went well so it _should_ be working as intended).


----------



## gnath (Feb 7, 2021)

I am following latest package repo for daily use. Old ports tree was used only for 'drm-kmod'. Output from /usr/ports/.gituprevision is 'master:e5260f3eb' . I shall try for quarterly and fresh tree for latest also.


----------



## jmehr (Feb 7, 2021)

Using e5260f3eb, I did a quick test and I was able to compile /usr/ports/graphics/drm-fbsd11.2-kmod.

Looking at /usr/ports/Mk/Uses/compiler.mk, it looks like its trying to determine the version of the compiler you are using and can't for some reason.  I'm not sure where specific compilers are specified (maybe /etc/make.conf?) so hopefully a ports tree expert will be able to lend a hand.


----------



## RypPn (Feb 13, 2021)

Thanks for creating the tool, certainly makes the transition from svn to git relatively painless, one small observation, /usr/ports/.gituprevision wasn't created in my /usr/ports tree


----------



## sidetone (Feb 22, 2021)

Hi, when I ran `psearch` (ports-mgmt/psearch), I got this error:

```
Error reading index file "/usr/ports/INDEX-12": No such file or directory
```
From `cat /usr/ports/.gituprevision`, I got:

```
master:cc83265a6
```
psearch(1)

```
-f FILE, --file=FILE
              Path to INDEX file. Defaults to the standard location of the
              INDEX file on the FreeBSD system that psearch runs on. Non-
              standard locations given in /etc/make.conf are ignored.
```
I removed the /usr/ports/ directory, ran `gitup ports`, and got some output plus:

```
gitup: build_repair_command: There are too many files to repair -- please re-clone the repository: Argument list too long
```
I tried, `gitup -c ports`, the command for cloning according to the manual, which put a few files in my ports directory, but got the same error message when trying to fill the ports directory.

Perhaps `portsnap extract` should be used first on an empty directory?


----------



## jmehr (Feb 23, 2021)

I did some digging and it doesn't look like the INDEX-[os version] files are present in the ports tree. https://github.com/freebsd/freebsd-ports

I'm not a ports tree expert (and hopefully someone who is will chime in and correct me if I'm wrong) but I believe you can manually create these index files by running:

`cd /usr/ports`
`make index`

It looks like `make index` can take a long time to build so adding INDEX-12 to the "ignores" field of your ports section in gitup.conf should prevent it from being deleted during subsequent pulls.

`gitup -c ports` should not have gone into repair mode and I'll see if I can figure out what went wrong there.  In the meantime, when you remove /usr/ports, you'll also want to remove /var/db/gitup/ports.  Because net/gitup doesn't keep the packfiles it downloads from the repository, it uses the contents of this file to reconstruct what it needs from the local copy of the repository and will go into repair mode if it encounters any discrepancies.  Removing both and then running `gitup ports` should get you back up and running.


----------



## Mjölnir (Mar 6, 2021)

Since some time now, gitup(1) fails to update my _current_ sources.
`gitup current`

```
# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd
# Target: /src/14-CUR
# Have: de1aa3dab23c06fec962a14da3e7b4755c5880cf
# Want: de1aa3dab23c06fec962a14da3e7b4755c5880cf
# Branch: refs/heads/master
```
gitup.conf(5): 



Spoiler: gitup.conf



# $FreeBSD$
#
# Default configuration options for gitup.conf.
# $Id: gitup.conf,v 1.1 2021/03/06 00:54:50 root Exp root $
{
        "defaults" : {
                "host"           : "github.com",
                "port"           : "443",
                "verbosity"      : "1",
                "work_directory" : "/var/db/gitup",
        },

# "latest ports branch is done with portsnap(8) instead; see portsnap.conf(5)
        "ports" : {
                "repository" : "/freebsd/freebsd-ports",
                "branch"     : "quarterly",
                "target"     : "/ports/quarterly",
                "ignore"     : [
                        "./distfiles",
                        "./packages",
                ],
        },

# release sources are updated with freebsd-update(8) instead; see freebsd-update.conf(5)
#       "12.1-release" : {
#               "repository" : "/freebsd/freebsd",
#               "branch"     : "releng/12.1",
#               "target"     : "/src/12.1-REL",
#       },

        "12-stable" : {
                "repository" : "/freebsd/freebsd",
                "branch"     : "stable/12",
                "target"     : "/src/12-STABLE",
        },

        "13-stable" : {
                "repository" : "/freebsd/freebsd",
                "branch"     : "stable/13",
                "target"     : "/src/13-STABLE",
        },

        "current" : {
                "repository" : "/freebsd/freebsd",
                "branch"     : "master",
                "target"     : "/src/14-CUR",
        }
}


Thx in advance for any hints.


----------



## jmehr (Mar 6, 2021)

It looks like the "master" branch has been deprecated https://github.com/freebsd/freebsd-src/commit/de1aa3dab23c06fec962a14da3e7b4755c5880cf and switching to "main" should get you back up and running.

Also, we ran into some issues with repository paths that do not end in ".git".  When you're pulling from github.com, if you change your repository to:


```
/freebsd/freebsd-src.git
```

you should be able to avoid those potential problems.


----------



## Mjölnir (Mar 6, 2021)

So I changed in gitup.conf(5)

```
"current" : {
                "repository" : "/freebsd/freebsd-src.git",
                "branch"     : "main",
                "target"     : "/src/14-CUR",
        }
```
Correct?  Do I also have to change the `"repository": ...` for my other source trees, although they seem to work fine?  Of course, not for the _quarterly_ ports(7) tree, right?
EDIT Just click _Like_ if that's what you meant, else comment.


----------



## sidetone (Mar 6, 2021)

I had an error from compiling my kernel after using gitup for sources, but I'm unsure if it came from gitup or from another mistake. When I selected the custom compiled kernel, it wanted to boot from the cd drive. As if the sources were intended for CD instead of being on the harddisk. Going to a backup kernel, then a generic kernel update from `freebsd-update` made my system work.

It could also have been an error from when my custom KERNCONF files got erased by gitup, which I now know how to prevent, and my backup KERNCONF files may not have been as current.

This was weeks ago. I also haven't tried again to see.


----------



## jmehr (Mar 6, 2021)

Mjölnir said:


> So I changed in gitup.conf(5)
> 
> ```
> "current" : {
> ...



Your config looks good to me.  The repository path problems occurred when using git.freebsd.org and if you continue using github.com, your other source trees should be good.

If you decide to switch over to using git.freebsd.org you'll want to change all of them to 


```
/src.git
```


----------



## jmehr (Mar 6, 2021)

sidetone said:


> I had an error from compiling my kernel after using gitup for sources, but I'm unsure if it came from gitup or from another mistake. When I selected the custom compiled kernel, it wanted to boot from the cd drive. As if the sources were intended for CD instead of being on the harddisk. Going to a backup kernel, then a generic kernel update from `freebsd-update` made my system work.
> 
> It could also have been an error from when my custom KERNCONF files got erased by gitup, which I now know how to prevent, and my backup KERNCONF files may not have been as current.
> 
> This was weeks ago. I also haven't tried again to see.


net/gitup should not have deleted your custom kernel config files (it is supposed to ignore the contents of the various sys/architecture/conf directories in your target directory when deleting files that no longer exist in the repository).

Were any of the directories in your repository symbolic links to other directories in your filesystem?


----------



## Mjölnir (Mar 6, 2021)

That was a heavy update...


jmehr said:


> Your config looks good to me.  The repository path problems occurred when using git.freebsd.org and if you continue using github.com, your other source trees should be good.


So: yes, because it doesn't hurt when using _github_, but _git.freebsd.org_ needs it.  Yes, I'll stick to _github_ until they start to act evilish, what I don't expect, frankly.  Simply because it's nearer; IIUC the seem to have a mirror at the DE-CIX in Frankfurt/Main, Germany (derived from observing mtr(8)).  I don't think _git.freebsd.org_ has a geo-DNS'ed mirror in Europe; do they/we?


jmehr said:


> If you decide to switch over to using git.freebsd.org you'll want to change all of them to
> 
> 
> 
> ...


Ok, I took a note in my gitup.conf(5).  Thx 4 chiming in so quickly.  BTW a _status_ or _check_ command would be nice, as well as a `-n / --dry-run` do-nothing switch.


----------



## sidetone (Mar 6, 2021)

I might have deleted that line under ignore the time it got deleted, in gitup.conf, because I didn't understand it then. Then later it was explained, and I put it back, and updated the sources.


jmehr said:


> Were any of the directories in your repository symbolic links to other directories in your filesystem?


I didn't understand. There weren't symbolic files in /usr/src/sys/amd64/conf/. There were two custom files, that one had an include for a trimmed down CUSTOM copy of GENERIC. The error could have been on my end, from possibly using an outdated KERNCONF (that was previously a backup) than the one needed for this kernel.

I'll try again at a later day. I'll start over then.


----------



## jmehr (Mar 6, 2021)

Mjölnir said:


> That was a heavy update...
> 
> So: yes, because it doesn't hurt when using _github_, but _git.freebsd.org_ needs it.  Yes, I'll stick to _github_ until they start to act evilish, what I don't expect, frankly.  Simply because it's nearer; IIUC the seem to have a mirror at the DE-CIX in Frankfurt/Main, Germany (derived from observing mtr(8)).  I don't think _git.freebsd.org_ has a geo-DNS'ed mirror in Europe; do they/we?
> 
> Ok, I took a note in my gitup.conf(5).  Thx 4 chiming in so quickly.  BTW a _status_ or _check_ command would be nice, as well as a `-n / --dry-run` do-nothing switch.



I'm not sure if there is or will be a mirror in Europe (I can't find any information about this) but an email to the freebsd-git mailing list should connect you with someone who can let you know about the plans for any additional mirrors.  If I run across any, I'll be sure to post them here.

A dry-run command should be very easy to add (I'll put it on the to-do list).  For the status command, what would be most helpful to see?


----------



## diizzy (Mar 6, 2021)

There's one in Amsterdam at least, "This is gitmir.pkt.FreeBSD.org located at Amsterdam, The Netherlands."
git.freebsd.org is geo aware ;-)


----------



## Mjölnir (Mar 6, 2021)

jmehr said:


> For the status command, what would be most helpful to see?


Files behind/deleted/new on the repository (indicators: */-/+); files up-to-date, but broken (checksum doesn't match; indicator: # or ?).  Up-to-date files doesn't need to be reported IMHO.


diizzy said:


> There's one in Amsterdam at least, "This is gitmir.pkt.FreeBSD.org located at Amsterdam, The Netherlands."
> git.freebsd.org is geo aware ;-)


Great - but then I found another reason to stick using github: let M$ pay for traffic & not FreeBSD


----------



## Yze (Mar 15, 2021)

jmehr any plans to add IPv6 support? We moved our build cluster long time ago to ipv6-only jails and so we can't evaluate it to provide you feedback


----------



## jmehr (Mar 17, 2021)

Yze said:


> jmehr any plans to add IPv6 support? We moved our build cluster long time ago to ipv6-only jails and so we can't evaluate it to provide you feedback



I just committed an update for this yesterday.  I'll be tagging and pushing out 0.91 as soon as I can get proxy support fully implemented (hopefully within a week).


----------



## grahamperrin@ (Apr 7, 2021)

Jose said:


> … suggest you take this subject to its own topic. …







richardtoohey2 said:


> … I didn't time the original gitup ports - was very _roughly_ a minute. All this on FreeBSD 12.2p6. …



Thanks. Not significantly different here, where `gitup ports` clones into an empty (or nearly empty) directory.

With slightly outdated 12.2-RELEASE-p4, around a minute and a half (1:31.66). Marginally quicker with things warmed up a little (the second screenshot):










It was easy to get below one minute for _clone_ situations. More memory to the machine, then after deletions and a warm-up: fifty-six seconds for the second run.









Where the effect of `gitup ports` is to _pull_, I see diverse timings from diverse use cases (diverse hardware, and so on). Some runs are long, some not. 

The diversity surprises some people; I find no problem with it. Keyword for any reader: YMMV


----------



## jmehr (Apr 7, 2021)

When cloning, the performance of net/gitup should be pretty close to the performance of the official Git client when requesting a shallow clone -- the size of the pack file downloaded will be the same as a --depth 1 git clone but net/gitup runs in a single thread so there will be a performance hit when processing the pack file.

When pulling, however, net/gitup will be much slower than the official Git client.  In order to minimize the amount of disk space needed, net/gitup does not retain the pack files that the official Git client needs.  Since these pack files contain all the information needed to update a local copy of the repository, a pristine local copy of a repository can be used to reconstruct the missing information that once existed in the pack files and net/gitup must walk the local tree and calculate the checksum for every file to do this.


----------



## trev (Apr 7, 2021)

The only issue I have is with a FreeBSD 12.2R VPS with 512MB memory and 1.5G swap. gitup runs out of swap and the OOM kills processes including gitup. What Is the minimum  amount of memory required?

As a workaround, I'm currently rsync'ing a gitup'ped /usr/pports from another system, so not a big deal.


----------



## jmehr (Apr 7, 2021)

I just did a couple of tests to check memory usage and a clone of the ports repository topped out around 545M and a pull (updating from a few days ago) was less than that.  Cloning/pulling the source can take up to 2GB of memory.

I just added an issue to gitup's to-do list to implement an option to minimize its memory footprint.


----------



## grahamperrin@ (Apr 8, 2021)

jmehr thanks for all your work in these areas. 

I have an idea for an enhancement request (maybe two overlapping ERs). Would you like to discuss such things here, before I raise an issue in GitHub?


----------



## scottro (Apr 8, 2021)

I notice that it doesn't create an INDEX file. (or said file isn't in the git repository).  I use a package, psearch, to search for ports, that relies upon the INDEX file. I know I've seen something on the forums about said file, but don't remember if it was that the file is broken, removed,or something completely different. An INDEX-x file is also needed for make search in /usr/ports

I am able to run make fetchindex after cloning the ports, (and as I've only used gitup to get ports, not git, I don't know if this is something that would also happen if I used git rather than gitup), so it's not a major issue.  Aside from that, the tool is working quite well for me. I


----------



## grahamperrin@ (Apr 8, 2021)

scottro said:


> … something on the forums …


 
Maybe [ports][GIT][portsnap] Warning about INDEX


----------



## zirias@ (Apr 8, 2021)

scottro see








						[ports][GIT][portsnap] Warning about INDEX
					

In the context of ports moving to GIT, as well as (expected) deprecation of portsnap, I've seen some questions regarding INDEX, which was automatically fetched by portsnap.  Right now, I learned the following in an IRC channel full of devs: < XXXXXX> INDEX is currently dangerously misleading...




					forums.freebsd.org
				




In a nutshell:

An index is never part of the repository
A downloaded index can never be fully correct as soon as some options are changed locally
Since introduction of port flavors, index creation is broken, so even creating one locally won't guarantee a correct result
Therefore, best avoid anything relying on an index for now.


----------



## olli@ (Apr 8, 2021)

Well, the INDEX file is good enough for using “make search” (and probably for psearch, too, although I don’t know this tool). So it should be safe to fetch the INDEX file or generate it yourself, if it’s just for those purposes.

Alternatively, there are online search engines for ports, e.g. Porgle. Also, FreshPorts has a search feature (although it doesn’t support regular expressions or complex search expressions).


----------



## scottro (Apr 8, 2021)

olli@ thanks, someone (perhaps you?), had already mentioned porgle. But yes, the only thing I use INDEX for is for psearch. (Since I have psearch,I don' use make search, but mentioned it as more people are familiar with it). 
Zirias, thanks for showing me the link, I *knew* I had seen something but couldn't remember what it was.


----------



## jmehr (Apr 9, 2021)

grahamperrin said:


> jmehr thanks for all your work in these areas.
> 
> I have an idea for an enhancement request (maybe two overlapping ERs). Would you like to discuss such things here, before I raise an issue in GitHub?



I'm ok with either option.  Discussing feature requests here does make it easier for more people to join in and contribute ideas, though.


----------



## grahamperrin@ (Apr 10, 2021)

One of the ideas appeared already: 

Less verbose option? · Issue #59 · johnmehr/gitup
– I intended to request a `--quiet` option.


The overlapping idea is timely attention to relevant parts of UPDATING files. 

It's easier to *draw a user's attention* when things are quiet.

An initial `gitup ports` _clone_ can conclude with a hint for the user to see: 

/usr/ports/UPDATING
`gitup ports` _pulls_ can conclude with a hint for the user to review the same file, only if the pull included an update to the file. 

An initial `gitup current` _clone_ can conclude with a hint for the user to read: 

/usr/src/UPDATING
– with special attention to common items. 

And so on.


----------



## jmehr (Apr 13, 2021)

The less verbose option is almost ready (just have to get the display of the '+' and '*' right to show when ports are added or modified).

I like the special attention feature as I must confess that I don't always read /usr/ports/UPDATING as much as I should. 

One idea I had when thinking about this was that, since the timestamps of the files in /var/db/gitup let us know when the last time gitup was run, we could also add an option that prints out the portions of /usr/ports/UPDATING since the last run (either as the full text or just the first two lines such as "20210411: AFFECTS: users or devel/py-RPyC").

Thoughts?


----------



## grahamperrin@ (Apr 13, 2021)

jmehr said:


> The less verbose option is almost ready (just have to get the display of the '+' and '*' right to show when ports are added or modified).



FWIW I imagine _minimally_ verbose (when no UPDATING file is updated) comprising just two lines:


```
working …
done.
```

– with `working …` apparent immediately after the file system scan commences (context: <https://github.com/johnmehr/gitup/issues/43#issuecomment-813783543>).



> I like the special attention feature …



Side note: COMMON ITEMS is partly within the scope of <https://reviews.freebsd.org/D28062>. Given the duplication of effort, and so on, I'll be not surprised if this section eventually disappears … then it'll be unnecessary to draw special attention to the section.


----------



## grahamperrin@ (Apr 18, 2021)

jmehr said:


> When pulling, … walk the local tree and calculate the checksum for every file to do this.



I wondered whether a simple ZFS cache device would help.

(Detail moved to https://github.com/johnmehr/gitup/issues/58>.)


----------



## grahamperrin@ (Apr 24, 2021)

jmehr FYI Problems with gitup - doesn't delete files / old ports.


----------



## edinilsonjs (Apr 25, 2021)

Something is wrong with IPv6 implementation because if IPv6 isn´t compiled in kernel or is disabled, the following error is raised in FreeBSD 13.0 RELEASE:
gitup: connect_server: socket failure: Address family not supported by protocol family


----------



## jmehr (Apr 26, 2021)

grahamperrin said:


> jmehr FYI Problems with gitup - doesn't delete files / old ports.


Thank you for letting me know about this one!  I'm not currently subscribed to the freebsd-ports list and I just sent him an email asking for more information.


----------



## bagas (May 6, 2021)

Error!


> # gitup ports
> gitup: connect_server: socket failure: Address family not supported by protocol family



ipv6 is completely removed from my system, I don't want to turn it on, I don't need it !!!
How to update ports is now not clear!

Previously, there were no problems with updating ports using gitup.
Why it was implemented via ipv6 is unclear.
We'll have to install a full-fledged git!
Gitup delete to hell!


----------



## covacat (May 6, 2021)

bagas said:


> Error!
> 
> 
> ipv6 is completely removed from my system, I don't want to turn it on, I don't need it !!!
> ...


try to set the ipv4 address in the config file (instead of git.freebsd.org)


----------



## bagas (May 6, 2021)

covacat said:


> try to set the ipv4 address in the config file (instead of git.freebsd.org)


Created a ticket.





						255649 – gitup: connect_server: socket failure: Address family not supported by protocol family
					






					bugs.freebsd.org
				




# host git.freebsd.org
git.freebsd.org is an alias for gitmir.geo.freebsd.org.
gitmir.geo.freebsd.org has address 139.178.72.204
gitmir.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::e6a:1
gitmir.geo.freebsd.org mail is handled by 0 .

# telnet 139.178.72.204 443
Trying 139.178.72.204...
Connected to gitmir.pkt.freebsd.org.
Escape character is '^]'.

q
HTTP/1.1 400 Bad Request
Server: nginx/1.18.0
Date: Thu, 06 May 2021 07:14:28 GMT
Content-Type: text/html
Content-Length: 157
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
Connection closed by foreign host.


I assume that the problem is in the IPv4 address (139.178.72.204).
IPv6 in my system is completely removed, I do not need IPv6.
I will not build the system with ipv6!


----------



## bagas (May 6, 2021)

covacat said:


> try to set the ipv4 address in the config file (instead of git.freebsd.org)


Specified the ipv4 address in place of the domain, the ports were updated.
Strange, then the address selection priority (gitmir.geo) stands firmly ipv6?

gitup.conf
        "ports" : {
#               "host"       : "git.freebsd.org",
                "host"       : "139.178.72.204",


----------



## covacat (May 6, 2021)

bagas said:


> Specified the ipv4 address in place of the domain, the ports were updated.
> Strange, then the address selection priority (gitmir.geo) stands firmly ipv6?
> 
> gitup.conf
> ...


looking at the source i saw it uses getaddrinfo which returns all addresses of the host regardles of the family
and if for some reason the inet6 address comes first and you don't support ipv6 you have a problem


----------



## bagas (May 6, 2021)

gitup-0.92 no more problem.

        "ports" : {
                "host"       : "git.freebsd.org",
                "repository" : "/ports.git",
                "branch"     : "main",


----------



## zirias@ (May 6, 2021)

bagas said:


> IPv6 in my system is completely removed, I do not need IPv6.
> I will not build the system with ipv6!


This is pretty short-sighted. We're already at the point where IPv4 cannot support the "whole internet" any more, ISPs provide IPv4 via CGNAT or DSLITE to customers (sharing a single address for many of them) and there are quite some hosts out there reachable only via IPv6 (most notable here the "beefy" machines building official FreeBSD packages). This doesn't come as a surprise, we all knew 32bit addressing won't hold …

But then:


covacat said:


> looking at the source i saw it uses getaddrinfo which returns all addresses of the host regardles of the family
> and if for some reason the inet6 address comes first and you don't support ipv6 you have a problem


This is a bug anyways. Using getaddrinfo(3) and looping over the results is the correct approach of course, but any error trying to create a socket(2) or connecting on it should be ignored, trying the next result instead. An error should only be thrown after _all_ results failed.


----------



## bagas (May 6, 2021)

Zirias said:


> This is pretty short-sighted. We're already at the point where IPv4 cannot support the "whole internet" any more, ISPs provide IPv4 via CGNAT or DSLITE to customers (sharing a single address for many of them) and there are quite some hosts out there reachable only via IPv6 (most notable here the "beefy" machines building official FreeBSD packages). This doesn't come as a surprise, we all knew 32bit addressing won't hold …


All this is chatter and manipulation of general opinion.
For about 8-15 years, marketers say that ipv4 is about to end, but ipv4 is not over yet!
And most of the internet segment is on ipv4.
We will use ipv4 for at least another 10 years!


----------



## zirias@ (May 6, 2021)

LOL. Yes, and still you're not talking about the same thing. I didn't say IPv4 will go away any time soon. It won't, cause resistence from people just never wanting to change anything is extremely powerful.

The thing is: The fraction of the internet reachable via IPv4 will decrease further. It's already well below 100%. This is something you can't change, because there is just no other way.


----------



## Jose (May 6, 2021)

All possible ipv4 addresses have been used. There is absolutely no slack left in the address space





						IPv4 Exhaustion: What About Class E Addresses? - PacketLife.net
					






					packetlife.net


----------



## edinilsonjs (May 6, 2021)

bagas said:


> Created a ticket.
> 
> 
> 
> ...











						add IPv6 support (AF_INET6) · Issue #54 · johnmehr/gitup
					

since git.freebsd.org and giblab.com also run on ipv6 - it would be great to add support for it.




					github.com


----------



## covacat (May 6, 2021)

Zirias said:


> LOL. Yes, and still you're not talking about the same thing. I didn't say IPv4 will go away any time soon. It won't, cause resistence from people just never wanting to change anything is extremely powerful.
> 
> The thing is: The fraction of the internet reachable via IPv4 will decrease further. It's already well below 100%. This is something you can't change, because there is just no other way.


well, the internet is for pr0n
so when pr0n moves to ipv6, v4 will die


----------



## zirias@ (May 6, 2021)

covacat said:


> well, the internet is for pr0n
> so when pr0n moves to ipv6, v4 will die


LOL again, cause, this might actually happen earlier, don't you think? 
Especially the "big players" can't affort dropping IPv4 of course, so companies buy previously owned addresses (as good as new) for fantastic prices, hehe  I don't think this would apply so much to, well, _this_ specific area


----------



## trev (May 6, 2021)

FWIW, installing devel/git on my VPS with just 512MB of memory worked perfectly to clone ports and keep them updated.

The latest net/gitup failed with OOM (given I have 1.5G of swapfile, I still find that surprising).


----------



## grahamperrin@ (May 7, 2021)

trev said:


> … OOM











						Add an option to minimize memory usage · Issue #58 · johnmehr/gitup
					






					github.com
				




Off-topic from gitup, 



> … I have 1.5G of swapfile, …



Too little, in my experience. <https://forums.FreeBSD.org/threads/swap-battle-stories.80159/post-510437> – join me there …


----------



## covacat (May 7, 2021)

this patch will reduce memory usage to about a third when loading cached remote files from /var/db/gitup/ports by avoiding lots reallocs in append_string

at the end of the function load_remote_files ram usage drops from  290MB to 80MB


```
--- gitup.c    2021-05-05 03:33:34.000000000 +0300
+++ /tmp/gitup.c    2021-05-08 00:33:44.685833000 +0300
@@ -707,8 +707,14 @@
     struct file_node *file = NULL;
     char             *line = NULL, *hash = NULL, *path = NULL, *remote_files = NULL;
     char              temp[BUFFER_UNIT_SMALL], base_path[BUFFER_UNIT_SMALL], *buffer = NULL, *temp_hash = NULL;
-    uint32_t          count = 0, remote_file_size = 0, buffer_size = 0, buffer_length = 0;
-
+    uint32_t          count = 0, remote_file_size = 0, buffer_size = 0, buffer_length = 0,sz_need;
+    char *tuffer, *yab;
+    buffer_size = 32768;
+    tuffer = malloc(buffer_size);
+    if(!tuffer) {
+     err(EXIT_FAILURE, "load_remote_file_list: malloc tuffer 1");
+     }
+    yab = tuffer;
     load_file(connection->remote_files, &remote_files, &remote_file_size);
 
     while ((line = strsep(&remote_files, "\n"))) {
@@ -723,12 +729,16 @@
            obj_tree for what has been read. */
 
         if (strlen(line) == 0) {
-            if (buffer != NULL) {
+            if (buffer_length) {
+                    buffer = malloc(buffer_length);
+                    if(!buffer)
+                     err(EXIT_FAILURE, "load_remote_file_list: malloc buffer");
+                    memcpy(buffer,tuffer,buffer_length);
                 if (connection->clone == false)
                     store_object(connection, 2, buffer, buffer_length, 0, 0, NULL);
 
                 buffer = NULL;
-                buffer_size = buffer_length = 0;
+                buffer_length = 0;
             }
 
             continue;
@@ -770,14 +780,29 @@
 
             /* Add the line to the buffer that will become the obj_tree for this directory. */
 
+            sz_need = strlen(line) + strlen(path) + 2 + 20;
+            if(buffer_length + sz_need > buffer_size) {
+            /*
+            printf("HIT %d %d\n",buffer_length + sz_need,buffer_size);
+            */
+             tuffer = realloc(tuffer,buffer_size + 32768);
+             if(!tuffer)
+               err(EXIT_FAILURE, "load_remote_file_list: tuffer malloc 2");
+             buffer_size += 32768;
+             }
+            yab = tuffer + buffer_length;
             temp_hash = illegible_hash(hash);
-
+            sprintf(yab,"%s %s",line,path);
+            memcpy(yab+strlen(yab) + 1,temp_hash,20);
+            yab += sz_need;
+            buffer_length += sz_need;
+            /*
             append_string(&buffer, &buffer_size, &buffer_length, line, strlen(line));
             append_string(&buffer, &buffer_size, &buffer_length, " ", 1);
             append_string(&buffer, &buffer_size, &buffer_length, path, strlen(path));
             append_string(&buffer, &buffer_size, &buffer_length, "\0", 1);
             append_string(&buffer, &buffer_size, &buffer_length, temp_hash, 20);
-
+            */
             free(temp_hash);
         }
 
@@ -785,7 +810,7 @@
 
         RB_INSERT(Tree_Remote_Path, &Remote_Path, file);
     }
-
+    free(tuffer);
     free(remote_files);
 }
```


----------



## jmehr (May 10, 2021)

Thank you very much for finding this and taking the time to write a patch.  I ended up rewriting append_string() to remove the excessive buffer allocation as the issue affected (though not nearly as bad) everything that used that function.  Thank you!


----------



## covacat (May 10, 2021)

can you make prune_tree errors non fatal (warnings) ?


----------



## covacat (May 10, 2021)

the below patch makes a clone command for ports to use 300mb less ram 
creates a backing file during unpack_objects() and only keeps the offset in the file in the stored object. (object->buffer is kept null)
when the object buffer is needed it is loaded from the backing file (/var/db/gitup/$repo.tmp)

```
--- /usr/ports/net/gitup/work/gitup-0.93/gitup.c    2021-05-09 07:57:39.000000000 +0300
+++ gitup.c    2021-05-10 19:12:50.942962000 +0300
@@ -70,7 +70,8 @@
     char     *ref_delta_hash;
     uint32_t  pack_offset;
     char     *buffer;
-    uint32_t  buffer_size;
+    uint32_t  buffer_size,file_offset;
+    char   can_free;
 };
 
 struct file_node {
@@ -119,6 +120,7 @@
     int                  verbosity;
     uint8_t              display_depth;
     char                *updating;
+    int                back_store;
 } connector;
 
 static void     append(char **, unsigned int *, const char *, size_t);
@@ -165,7 +167,32 @@
 static void     unpack_objects(connector *);
 static uint32_t unpack_variable_length_integer(char *, uint32_t *);
 static void     usage(const char *);
+static void     load_buffer(connector *,struct object_node *);
+static void    release_buffer(struct object_node *);
 
+static void release_buffer(struct object_node *obj)
+{
+if(!obj->can_free) {
+ // dont release non file backed objects
+ free(obj->buffer);
+ obj->buffer = NULL;
+ }
+}
+
+static void load_buffer(connector * connection,struct object_node *obj)
+{
+ int rd;
+ if(!obj->buffer) {
+  obj->buffer = malloc(obj->buffer_size);
+  if(!obj->buffer)
+   err(EXIT_FAILURE, "load_buffer: malloc");
+  lseek(connection->back_store,obj->file_offset,SEEK_SET);
+  rd = read(connection->back_store,obj->buffer,obj->buffer_size);
+  if(rd != (int)obj->buffer_size) {
+   err(EXIT_FAILURE, "load_buffer: read %d %d",rd,obj->buffer_size);
+   }
+  }
+ }
 /*
  * node_compare
  *
@@ -1734,7 +1761,6 @@
     char               *hash = NULL;
 
     hash = calculate_object_hash(buffer, buffer_size, type);
-
     /* Check to make sure the object doesn't already exist. */
 
     find.hash = hash;
@@ -1762,6 +1788,8 @@
         object->ref_delta_hash = (ref_delta_hash ? legible_hash(ref_delta_hash) : NULL);
         object->buffer         = buffer;
         object->buffer_size    = buffer_size;
+                object->can_free       = 1;
+                object->file_offset    = -1;
        
         if (connection->verbosity > 1)
             fprintf(stdout,
@@ -1798,9 +1826,20 @@
     uint32_t       file_size = 0, file_bits = 0, pack_offset = 0;
     uint32_t       lookup_offset = 0, position = 4;
     unsigned char  zlib_out[16384];
+        int nobj_old,tot_len = 0;
+        char remote_files_tmp[BUFFER_UNIT_SMALL];
 
     /* Check the pack version number. */
+        snprintf(remote_files_tmp, BUFFER_UNIT_SMALL,
+                "%s.tmp",
+                connection->remote_files);
+        connection->back_store = open(remote_files_tmp, O_WRONLY | O_CREAT | O_TRUNC);
 
+        if (connection->back_store == -1)
+                err(EXIT_FAILURE,
+                        "save_tmp: write file failure %s",
+                        remote_files_tmp);                   
+
     version   = (unsigned char)connection->response[position + 3];
     position += 4;
 
@@ -1904,6 +1943,8 @@
 
         inflateEnd(&stream);
         position += stream.total_in;
+                write(connection->back_store,buffer,buffer_size);
+                nobj_old = connection->objects;
        
         store_object(connection,
             object_type,
@@ -1912,9 +1953,23 @@
             pack_offset,
             index_delta,
             ref_delta_hash);
-
+                if(nobj_old != connection->objects) {
+                 connection->object[nobj_old]->buffer = NULL;
+                 connection->object[nobj_old]->can_free = 0;
+                 connection->object[nobj_old]->file_offset = tot_len;
+                 }
+                tot_len += buffer_size;
+            free(buffer);   
         free(ref_delta_hash);
     }
+  close(connection->back_store);     
+  connection->back_store =  open(remote_files_tmp, O_RDONLY);
+  if (connection->back_store == -1)
+                err(EXIT_FAILURE,
+                 "open tmp ro:  failure %s",
+                 remote_files_tmp);
+
+  unlink(remote_files_tmp);   /* unlink now / dealocate when exit */
 }
 
 
@@ -2029,7 +2084,7 @@
         if ((merge_buffer = (char *)malloc(base->buffer_size)) == NULL)
             err(EXIT_FAILURE,
                 "apply_deltas: malloc");
-
+        load_buffer(connection,base);       
         memcpy(merge_buffer, base->buffer, base->buffer_size);
         merge_buffer_size = base->buffer_size;
 
@@ -2037,6 +2092,7 @@
 
         for (x = delta_count - 1; x >= 0; x--) {
             delta         = connection->object[deltas[x]];
+            load_buffer(connection,delta);   
             position      = 0;
             new_position  = 0;
             old_file_size = unpack_variable_length_integer(delta->buffer, &position);
@@ -2101,10 +2157,11 @@
              */
 
             memcpy(merge_buffer, layer_buffer, new_file_size);
+            release_buffer(delta);
         }
 
         /* Store the completed object. */
-
+        release_buffer(base);
         store_object(connection,
             base->type,
             merge_buffer,
@@ -2175,7 +2232,7 @@
             object.hash);
 
     /* Remove the base path from the list of upcoming deletions. */
-
+        load_buffer(connection,tree);
     file.path  = base_path;
     found_file = RB_FIND(Tree_Local_Path, &Local_Path, &file);
 
@@ -2291,7 +2348,7 @@
     }
 
     /* Add the tree data to the remote files list. */
-
+    release_buffer(tree);
     write(remote_descriptor, buffer, buffer_size);
     write(remote_descriptor, "\n", 1);
 
@@ -2346,6 +2403,7 @@
              */
 
             if (missing == false) {
+                load_buffer(connection,found_object);
                 check_hash = calculate_file_hash(
                     found_file->path,
                     found_file->mode);
@@ -2354,19 +2412,20 @@
                     found_object->buffer,
                     found_object->buffer_size,
                     3);
-
+                release_buffer(found_object);   
                 if (strncmp(check_hash, buffer_hash, 40) == 0)
                     update = false;
             }
 
             if (update == true) {
+                    load_buffer(connection,found_object);
                 save_file(found_file->path,
                     found_file->mode,
                     found_object->buffer,
                     found_object->buffer_size,
                     connection->verbosity,
                     connection->display_depth);
-
+                release_buffer(found_object);   
                 if (strstr(found_file->path, "UPDATING"))
                     extend_updating_list(connection,
                         found_file->path);
@@ -2409,13 +2468,14 @@
             "save_objects: cannot find %s",
             connection->want);
 
+    load_buffer(connection,found_object);                   
     if (memcmp(found_object->buffer, "tree ", 5) != 0)
         errc(EXIT_FAILURE, EINVAL,
             "save_objects: first object is not a commit");
 
     memcpy(tree, found_object->buffer + 5, 40);
     tree[40] = '\0';
-
+    release_buffer(found_object);
     /* Recursively start processing the tree. */
 
     snprintf(remote_files_new, BUFFER_UNIT_SMALL,
@@ -2460,14 +2520,14 @@
             errc(EXIT_FAILURE, EINVAL,
                 "save_objects: cannot find %s",
                 found_file->hash);
-
+        load_buffer(connection,found_object);       
         save_file(found_file->path,
             found_file->mode,
             found_object->buffer,
             found_object->buffer_size,
             connection->verbosity,
             connection->display_depth);
-
+                release_buffer(found_object);
         if (strstr(found_file->path, "UPDATING"))
             extend_updating_list(connection, found_file->path);
     }
```


----------



## edinilsonjs (May 10, 2021)

covacat said:


> the below patch makes a clone command for ports to use 300mb less ram
> creates a backing file during unpack_objects() and only keeps the offset in the file in the stored object. (object->buffer is kept null)
> when the object buffer is needed it is loaded from the backing file (/var/db/gitup/$repo.tmp)
> 
> ...


Is this code in version 0.93 ? Or will be merged in a next release?


----------



## covacat (May 10, 2021)

edinilsonjs said:


> Is this code in version 0.93 ? Or will be merged in a next release?


no idea; i'm not the port author, nor the maintainer


----------



## jmehr (May 12, 2021)

covacat said:


> the below patch makes a clone command for ports to use 300mb less ram
> creates a backing file during unpack_objects() and only keeps the offset in the file in the stored object. (object->buffer is kept null)
> when the object buffer is needed it is loaded from the backing file (/var/db/gitup/$repo.tmp)
> 
> ...


I just committed this.  Thank you very much!


----------



## jmehr (May 12, 2021)

covacat said:


> can you make prune_tree errors non fatal (warnings) ?


Done!


----------



## edinilsonjs (Jun 9, 2021)

Is there a way to use a specific source IP Address in gitup (something like environment var FETCH_BIND_ADDRESS in fetch or -S option in ping) ?


----------



## astyle (Jul 13, 2021)

@OP: Can you make gitup watch a specific group of ports like KDE? Right now, upgrading in-place is a major pain. The ports system has at least a dozen directories/categories where ports _kf5-*_ ports are hiding, and just as many directories/categories where  _plasma5-*_ ports are distributed. One idea that I have is doing that as an option for gitup.conf file...


----------



## grahamperrin@ (Aug 14, 2021)

jmehr <https://lists.freebsd.org/pipermail/freebsd-questions/2021-August/294564.html> _gitup is dumping core_


----------



## astyle (Aug 14, 2021)

grahamperrin said:


> jmehr <https://lists.freebsd.org/pipermail/freebsd-questions/2021-August/294564.html> _gitup is dumping core_


Nice to know... I'm gonna stick with portsnap for now... even though it's getting deprecated.


----------



## grahamperrin@ (Aug 14, 2021)

astyle said:


> Nice to know …



It's a report from one person, probably not a reason to avoid gitup.

For me, gitup runs normally (without crashing) on FreeBSD 14.0-CURRENT.


----------



## Mayhem30 (Aug 14, 2021)

grahamperrin said:


> It's a report from one person, probably not a reason to avoid gitup.
> 
> For me, gitup runs normally (without crashing) on FreeBSD 14.0-CURRENT.



Yeah, it works great here too.

I'm not having any issues with gitup on my FreeBSD 12.2 or 13.0 machines.


----------



## scottro (Aug 14, 2021)

No problem here with 13.0-p3


----------



## jmos (Aug 14, 2021)

grahamperrin said:


> It's a report from one person


Had core dumps from gitup more than once on three machines over the last days. I just fetched the entire ports tree from scratch, and then it worked again.


----------



## grahamperrin@ (Aug 14, 2021)

Thanks, 



jmos said:


> fetched the entire ports tree from scratch



– how, exactly?


----------



## jmos (Aug 14, 2021)

grahamperrin said:


> – how, exactly?


Deleted the content of /usr/ports/ (distfiles remained) as well as the directory /var/db/gitup/, then executed `gitup ports` again - worked.


----------



## grahamperrin@ (Aug 20, 2021)

grahamperrin said:


> > … gitup is dumping core …











						Segmentation fault · Issue #73 · johnmehr/gitup
					

Attempts to update the ports tree are failing with a segmentation fault. I've emptied /usr/ports and /var/db/gitup and tried using the clone option with the same result: tab@v1:~ % freebsd-vers...




					github.com


----------



## grahamperrin@ (Aug 21, 2021)

jmehr said:


> … Cloning/pulling the source can take up to 2GB of memory. …



Is this the likeliest cause of "Killed", where the OS has no swap?


----------



## aafbsd (Nov 16, 2021)

Just gave gitup a first try and started playing.  I started with:


```
{
  "defaults" : {
    "proxy_host"        : "proxy",
    "proxy_port"        : "3128",
    "port"              : 443,
    "host"              : "git.freebsd.org",
    "low_memory"        : false,
    "display_depth"     : 0,
    "verbosity"         : 1,
    "work_directory"    : "/tmp/gu",
  },

  "ports" : {
    "repository_path"  : "/ports.git",
    "branch"           : "main",
    "target_directory" : "/tmp/g",
    "ignores"          : [ "INDEX-11", "INDEX-12", "INDEX-13", "INDEX-14" ],
  }

}
```

and non-existing dirs "/tmp/g" and "/tmp/gu". Now I run "gitup ports" and it fills "/tmp/g" with the
current ports-tree. Out of curiosity I now run "gitup -r ports" and stuff in "/tmp/g" gets deleted.
Either I didn't get the logic or this is somewhat buggy .


----------



## grahamperrin@ (Nov 16, 2021)

Repair seems to cause excessive removals when there's a custom target · Issue #75 · johnmehr/gitup


----------



## astyle (Nov 16, 2021)

Looks like I'll stick with portsnap until I see it no longer working. Or I might learn straight git (not a priority for me, got other stuff on my plate, I just like the idea)


----------



## aafbsd (Nov 17, 2021)

Not much to learn. "git clone" followed by regular "git fetch", "git diff" (to see what would be done) and "git merge".
Can be combined into "git pull" but I like to do the single steps.

However, this gives you a bloated ports directory. Mine is 2,2GB with 50% sitting in .git. You will laugh but I'm
still a ctm-user (since 25+ years). But ports is no more delivered via ctm and Julian has no time to fix it. So
I was happy to discover gitup which seems small but maybe not that mature yet .

Another thing is that I have a few local modifications in my ports tree (stuff that can't be addresse via EXTRA_PATCH_TREE)
and I used to have a wrapper which restores the original files, applies the ctm files, and reapplies my own mods -- telling me
if their original versions were touched.

I have adopted this wrapper for the above git commands but I'd prefer to use something like gitup.


----------



## grahamperrin@ (Nov 27, 2021)

<https://github.com/johnmehr/gitup/issues/75#issuecomment-980531548>

Not yet ported; aafbsd would you like to also confirm the fix? Thank you.


----------



## aafbsd (Nov 29, 2021)

Yep, looks good...


----------



## aafbsd (Nov 29, 2021)

Now, what I'd like to have is an update mode where gitup leaves unknown files alone.
E.g., something like "ignores" for everything that does not get deleted or modified
actively from upstream. An example:

1. I run "gitup ports" to check out all for the first time

2. Now I add "/usr/ports/x/y/myfile"
3. Now I modify "/usr/ports/a/b/Makefile"

4. Someone modifies "/usr/ports/a/b/Makefile" upstream
5. Someone deletes"/usr/ports/c/p/distfile" upstream

6. Now I run "gitup ports" again. Normally it would now modify "Makefile",
delete "distfile" and "myfile". But I would like to have "myfile" left alone...


----------



## SirDice (Nov 29, 2021)

aafbsd said:


> Now, what I'd like to have is an update mode where gitup leaves unknown files alone.


Use devel/git in that case.


----------



## aafbsd (Nov 29, 2021)

SirDice said:


> Use devel/git in that case.


That's exactly what I don't want 
Please see https://forums.freebsd.org/threads/gitup.77863/post-542396


----------



## astyle (Nov 29, 2021)

aafbsd said:


> That's exactly what I don't want
> Please see https://forums.freebsd.org/threads/gitup.77863/post-542396


So why not script the chores you want done to the effect that you want?


----------



## aafbsd (Nov 29, 2021)

Because gitup will not let mentioned files alone. So I will have to live with git (and its bloat)
or hack gitup to do what I want. My hope was that gitup can be used as is for that but apparently
this is not the case.


----------



## SirDice (Nov 29, 2021)

gitup(1) is meant as a simple tool to only checkout the ports or source tree. It's not meant to do anything fancy.


----------



## covacat (Nov 29, 2021)

add your file(s) to ignore list


----------



## Jose (Nov 29, 2021)

aafbsd said:


> However, this gives you a bloated ports directory. Mine is 2,2GB with 50% sitting in .git.


Have you tried a shallow clone?








						How to Use Git Shallow Clone to Improve Performance | Perforce Software
					

Using git shallow clone can help you clone repos faster. Learn how to execute git shallow clone and prune your repos to accelerate CI pipelines.




					www.perforce.com


----------



## aafbsd (Nov 30, 2021)

covacat said:


> add your file(s) to ignore list


As the ignore list doesn't honour wildcards (or even regexes) this would be quite a
mess in my case. But since my files follow a very simple naming scheme, I've simply
hardcoded a matching test to ignore_file() and this should be enough.


----------



## aafbsd (Nov 30, 2021)

Jose said:


> Have you tried a shallow clone?
> 
> 
> 
> ...


Yep. Bloat was about 25% smaller but growing. I am happy now with the modified gitup ;-).


----------



## Jose (Nov 30, 2021)

aafbsd said:


> Yep. Bloat was about 25% smaller but growing. I am happy now with the modified gitup ;-).


Not sure what you're talking about.

```
$ git clone --depth 1 -b 2021Q4 https://git.FreeBSD.org/ports.git
Cloning into 'ports'...
remote: Enumerating objects: 178456, done.
remote: Counting objects: 100% (178456/178456), done.
remote: Compressing objects: 100% (167126/167126), done.
remote: Total 178456 (delta 9323), reused 118397 (delta 6961), pack-reused 0
Receiving objects: 100% (178456/178456), 73.81 MiB | 9.18 MiB/s, done.
Resolving deltas: 100% (9323/9323), done.
Updating files: 100% (142020/142020), done.
$ du -sh ports                                                                          
1.0G    ports
$ cd ports
$ du -sh .git                                                                     
 94M    .git
```


----------



## aafbsd (Nov 30, 2021)

Jose said:


> Not sure what you're talking about.


I didn't use -b and tried it once a day for 20 days...


----------



## Jose (Nov 30, 2021)

aafbsd said:


> I didn't use -b and tried it once a day for 20 days...


Gitup gives you one branch at a time. Let's compare apples to apples. Not sure what doing it for 20 days would prove.


----------



## SirDice (Nov 30, 2021)

Make sure your /usr/ports isn't filling up with all the work/ directories.


----------



## aafbsd (Nov 30, 2021)

Jose said:


> Gitup gives you one branch at a time. Let's compare apples to apples. Not sure what doing it for 20 days would prove.


Yep, you suggested git shallow clones and I commented on it. To clarify:
I tried git shallow clones for approx. 20 days, syncing one or two times a day.


----------



## aafbsd (Nov 30, 2021)

SirDice said:


> Make sure your /usr/ports isn't filling up with all the work/ directories.


Yep, my WRKDIR is on /tmp


----------



## covacat (Nov 30, 2021)

gitup kills all work dirs anyway


----------



## aafbsd (Dec 1, 2021)

Another reason why regex or at least wildcard matching for ignore would be useful.


----------



## covacat (Dec 1, 2021)

here is a patch that adds regex support 
also adds -I ignore on command line (works like ignore in config file)
a string preceded with a percent sign (%) is considered a regex otherwise a path
i had to modify prune_tree to check for ignores, otherwise you had to ignore work work/port work/port/file to protect a modified file
the only test i've done is
./gitup -v1  -I "%gitup/work/.*/gitup.c$" ports


```
--- work/gitup-0.96/gitup.c    2021-09-05 19:58:01.000000000 +0300
+++ /home/titus/builds/gitup.c    2021-12-01 12:38:45.404988000 +0200
@@ -52,6 +52,7 @@
 #include <string.h>
 #include <unistd.h>
 #include <zlib.h>
+#include <regex.h>
 
 #define GITUP_VERSION     "0.96"
 #define BUFFER_UNIT_SMALL  4096
@@ -122,6 +123,7 @@
     char                *remote_data_file;
     char                *remote_history_file;
     char               **ignore;
+    regex_t           **reg_ignore;                 
     int                  ignores;
     bool                 keep_pack_file;
     bool                 use_pack_file;
@@ -381,11 +383,18 @@
             return (false);
 
     for (x = 0; x < session->ignores; x++) {
+        if(!session->reg_ignore[x]) {
           ignore = session->ignore[x];
                
          if (strncmp(path, ignore, strlen(ignore)) == 0)
                return (true);
+         } else {
+           if(!regexec(session->reg_ignore[x],path,0,NULL,0)) {
+               printf("skipped %s with %s\n",path,session->ignore[x]);
+               return (true);
                }
+          }     
+    }
 
     return (false);
 }
@@ -511,6 +520,8 @@
 
             prune_tree(session, full_path);
         } else {
+       
+          if(!ignore_file(session,full_path,IGNORE_DELETE))
             if (remove(full_path) == -1)
                 err(EXIT_FAILURE,
                     "prune_tree: cannot remove %s",
@@ -3073,6 +3084,50 @@
 }
 
 
+static void
+add_ignore(connector *session,const char *string)
+{
+    char                temp[BUFFER_UNIT_SMALL];
+    int    ret_temp, length;
+    regex_t reg_temp;
+
+    session->ignore = (char **)realloc(
+        session->ignore,
+        (session->ignores + 1)*sizeof(char *));
+
+    session->reg_ignore = (regex_t **)realloc(
+        session->reg_ignore,
+        (session->ignores + 1)*sizeof(regex_t *));
+    if (session->ignore == NULL || session->reg_ignore == NULL)
+        err(EXIT_FAILURE,
+            "load_config_section: malloc");
+    session->reg_ignore[session->ignores] = NULL;       
+    snprintf(temp, sizeof(temp),
+        "%s",
+        string);
+
+    if (temp[0] != '/')
+        snprintf(temp, sizeof(temp),
+            "%s/%s",
+            session->path_target,
+            string);
+    if(string[0] == '%') {
+      ret_temp = regcomp(&reg_temp,string + 1,REG_EXTENDED);
+      if(ret_temp) {
+       warnx("warning: can't compile %s, ignoring\n",string + 1);
+       } else {
+       session->reg_ignore[session->ignores] = malloc(sizeof(regex_t));
+       if(!session->reg_ignore[session->ignores]) {
+       err(EXIT_FAILURE,
+            "load_config_section: malloc2");
+        }
+       memcpy(session->reg_ignore[session->ignores],&reg_temp,sizeof(regex_t));
+       }
+     }
+    length = session->ignores++;
+    session->ignore[length] = strdup(temp);
+       
+}
 /*
  * load_config_section
  *
@@ -3088,7 +3143,6 @@
     char                temp[BUFFER_UNIT_SMALL];
     int                 integer, length = 0;
     bool                boolean, target, path, ignores;
-
     its = ucl_object_iterate_new(section);
 
     while ((pair = ucl_object_iterate_safe(its, true))) {
@@ -3154,27 +3208,7 @@
 
             while ((ignore = ucl_object_iterate_safe(iti, true))) {
                 string = ucl_object_tostring(ignore);
-
-                session->ignore = (char **)realloc(
-                    session->ignore,
-                    (session->ignores + 1)*sizeof(char *));
-
-                if (session->ignore == NULL)
-                    err(EXIT_FAILURE,
-                        "load_config_section: malloc");
-
-                snprintf(temp, sizeof(temp),
-                    "%s",
-                    string);
-
-                if (temp[0] != '/')
-                    snprintf(temp, sizeof(temp),
-                        "%s/%s",
-                        session->path_target,
-                        string);
-
-                length = session->ignores++;
-                session->ignore[length] = strdup(temp);
+                add_ignore(session,string);   
             }
 
             ucl_object_iterate_free(iti);
@@ -3541,6 +3575,7 @@
         .remote_data_file    = NULL,
         .remote_history_file = NULL,
         .ignore              = NULL,
+        .reg_ignore         = NULL,
         .ignores             = 0,
         .commit_history      = false,
         .keep_pack_file      = false,
@@ -3581,7 +3616,7 @@
 
     /* Process the command line parameters. */
 
-    while ((option = getopt(argc, argv, "C:cd:h:klrS:t:u:v:w:")) != -1) {
+    while ((option = getopt(argc, argv, "I:C:cd:h:klrS:t:u:v:w:")) != -1) {
         switch (option) {
             case 'C':
                 if (session.verbosity)
@@ -3589,6 +3624,9 @@
                         "# Configuration file: %s\n",
                         configuration_file);
                 break;
+            case 'I':
+                add_ignore(&session,optarg);
+                break;   
             case 'c':
                 session.clone = true;
                 break;
@@ -4011,9 +4049,10 @@
             "important changes.\n%s#\n",
             session.updating);
 
-    for (x = 0; x < session.ignores; x++)
+    for (x = 0; x < session.ignores; x++) {
         free(session.ignore[x]);
-
+                free(session.reg_ignore[x]);
+                 }
     free(session.ignore);
     free(session.response);
     free(session.object);
```


----------



## grahamperrin@ (Dec 2, 2021)

Performance

7th April: 



jmehr said:


> … When pulling… net/gitup will be much slower than the official Git client. … net/gitup must walk the local tree and calculate the checksum for every file to do this.



For readers who might have missed this later comment: 



> … I'll be circling back to disk performance after I get the memory footprint issues addressed. …


----------



## jmehr (Dec 7, 2021)

covacat said:


> here is a patch that adds regex support
> also adds -I ignore on command line (works like ignore in config file)
> a string preceded with a percent sign (%) is considered a regex otherwise a path
> i had to modify prune_tree to check for ignores, otherwise you had to ignore work work/port work/port/file to protect a modified file
> ...



Thank you!  I committed a simplified version yesterday (everything in the ignores list goes through regcomp() and if we run into any problems, we can add a tuneable to include the REG_NOSPEC flag).

I also added a commented-out entry to the ports ignores list in gitup.conf:

"/work/|/work$"

which will preserve any /work/ directories in your ports tree.


----------



## aafbsd (Dec 7, 2021)

When using an ignore entry which matches a self created file (e.g. "sysutils/bareos17-server/lala")
and a subsequent gitup removes "sysutils/bareos17-server",  our to-be-ignored file is removed
as well. This can be circumvented with the following patch (warning: copy and paste):

`--- gitup.c.ORI 2021-12-07 09:47:08.645516000 +0100
+++ gitup.c     2021-12-07 09:47:02.493263000 +0100
@@ -514,6 +514,8 @@
                                "prune_tree: cannot stat() %s",
                                full_path);
 
+  if( ignore_file( session, full_path, IGNORE_DELETE ))
+    continue;
                if (S_ISDIR(sb.st_mode) != 0) {
                        if ((entry->d_namlen == 1) && (strcmp(entry->d_name, "." ) == 0))
                                continue;`

Of course this results in a warning

" ! cannot remove /tmp/g/sysutils/bareos17-server"

and has to be dealt with manually but it's still better than silently deleting it.


----------



## jmehr (Dec 11, 2021)

aafbsd said:


> When using an ignore entry which matches a self created file (e.g. "sysutils/bareos17-server/lala")
> and a subsequent gitup removes "sysutils/bareos17-server",  our to-be-ignored file is removed
> as well. This can be circumvented with the following patch (warning: copy and paste):
> 
> ...



I'm not having any luck recreating the scenario you've described.  When I create custom directories and files and add them to the ignores list, they survive subsequent runs.

When I clone an old commit of the https://github.com/freebsd/freebsd-ci repository where scripts/build/config-11 still exists, add a custom directory scripts/build/config-11/stuff to it and add another file scripts/build/config-11/stuff/test, add "scripts/build/config-11/stuff" to the ignores list and then pull the latest commit, the contents of scripts/build/config-11 are removed except for the custom directory and file.

Could you please send me a specific case where the custom files are deleted?  Thanks!


----------



## covacat (Dec 11, 2021)

if you ignore a file in a non repository dir eg /usr/ports/archivers/zip/work/file.c it will get nuked because /usr/ports/archivers/zip/work/
will be checked first and pruned


----------



## aafbsd (Dec 15, 2021)

jmehr said:


> I'm not having any luck recreating the scenario you've described.  When I create custom directories and files and add them to the ignores list, they survive subsequent runs.
> 
> When I clone an old commit of the https://github.com/freebsd/freebsd-ci repository where scripts/build/config-11 still exists, add a custom directory scripts/build/config-11/stuff to it and add another file scripts/build/config-11/stuff/test, add "scripts/build/config-11/stuff" to the ignores list and then pull the latest commit, the contents of scripts/build/config-11 are removed except for the custom directory and file.
> 
> Could you please send me a specific case where the custom files are deleted?  Thanks!


I have an old (as of 2021/12/03) ports directory in /tmp/g. In my gitup.conf I have an
"ignores" entry including "math/eigen2/bla". And I have this file:

buildbox:/tmp>ll g/math/eigen2/bla 
-rw-r-----  1 user  wheel  2852 15 Dec 06:40 g/math/eigen2/bla

Now I run gitup (where the eigen2 port is gone) and my file gets deleted:

...
 - /tmp/g/lang/ruby27/files/patch-include_ruby_ruby.h
 - /tmp/g/math/eigen2
 - /tmp/g/math/eigen2/Makefile
 - /tmp/g/math/eigen2/distinfo
 - /tmp/g/math/eigen2/pkg-descr
 - /tmp/g/math/eigen2/pkg-plist
 - /tmp/g/misc/freebsd-release-manifests/files/MANIFESTS/amd64-amd64-12.3-RC1
...

buildbox:/tmp>ll g/math/eigen2/bla     
ls: g/math/eigen2/bla: No such file or directory

With my patch from above the file won't get touched (and the dir not deleted). This is with gitup
from ports, but including the fix from https://forums.freebsd.org/posts/542233.

So in order to reproduce:

1. checkout ports with gitup
2. tar it to some place
3. run gitup until you find some port getting deleted
4. restore from the tar
5. copy something to "category/deletedport/myfile"
6. add "category/deletedport/myfile" to "ignores"
7. run gitup


----------



## jmehr (Dec 20, 2021)

aafbsd said:


> I have an old (as of 2021/12/03) ports directory in /tmp/g. In my gitup.conf I have an
> "ignores" entry including "math/eigen2/bla". And I have this file:
> 
> buildbox:/tmp>ll g/math/eigen2/bla
> ...


I'm still not having any luck reproducing the issue.  I have a copy of the ports tree from commit b298802b56c2f85df0cc28426b8612ea01751bd8 on December 5 (which has the math/eigen2 port) and if I add "math/eigen2/bla" to the ignores list, execute `touch math/eigen2/bla` and then run `gitup ports` to fetch the latest revision, the contents of math/eigen2 are removed except for math/eigen2/bla.

Could you please let me see the contents of your gitup.conf and could you try using the latest revision from gitup's repository?  Thanks!


----------



## covacat (Dec 20, 2021)

add math/eigen2/bar/bla.txt and exclude the file bla.txt
it will be erased because math/eigen2/bar comes first, it is not excluded and it is pruned (which deletes bla.txt)


----------



## jmehr (Dec 25, 2021)

covacat said:


> add math/eigen2/bar/bla.txt and exclude the file bla.txt it will be erased because math/eigen2/bar comes first, it is not excluded and it is pruned (which deletes bla.txt)



I still can't reproduce it but I'm thinking this might be due to differences in the ordering of files returned by the opendir() and readdir() calls.  I just committed aafbsd's patch (thank you!).  How does it work for both of you now?


----------



## grahamperrin@ (Jan 2, 2022)

jmehr please: is option `-c` intended for cases where, for example, a repair fails?

Spun off from <https://forums.freebsd.org/posts/549178> and with reference to the attached file, I had a peculiar experience. It seemed that:

`gitup -c release` (15:04, 549 in history) followed, not immediately, by `gitup release` (15:20, 560) resulted in repair, and a prompt to re-run.
It's _possible_ that I keyed Control-C during the first of the commands, although I can't imagine doing so. The linked post confirms that the first command ran to completion; _Done_.

Can you think of any other explanation for the second command resulting in a repair?


Below, I can not reproduce the peculiarity.


```
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # gitup -V
gitup version 0.96
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # freebsd-version -kru
13.0-RELEASE-p4
13.0-RELEASE-p4
13.0-RELEASE-p5
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # pkg -vv | grep -e url -e enabled
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest",
    enabled         : yes,
    url             : "https://alpha.pkgbase.live/current/FreeBSD:13:amd64/latest",
    enabled         : no,
    url             : "file:///usr/local/poudriere/data/packages/13-default",
    enabled         : yes,
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # gitup -c release
# Scanning local repository...
# Host: git.freebsd.org
# Port: 443
# Repository Path: /src.git
# Target Directory: /usr/src
# Have: 2646dd665909e60a369015c17cb602515e6025dc
# Want: 2646dd665909e60a369015c17cb602515e6025dc
# Branch: releng/13.0
# Done.
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # gitup release
# Scanning local repository...
# Host: git.freebsd.org
# Port: 443
# Repository Path: /src.git
# Target Directory: /usr/src
# Have: 2646dd665909e60a369015c17cb602515e6025dc
# Want: 2646dd665909e60a369015c17cb602515e6025dc
# Branch: releng/13.0
# Done.
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # grep -v \# /etc/freebsd-update.conf | sort


















Components src world kernel
IDSIgnorePaths /usr/share/man/cat
IDSIgnorePaths /usr/share/man/whatis
IDSIgnorePaths /var/db/locate.database
IDSIgnorePaths /var/log
IgnorePaths
KeyPrint 800651ef4b4c71c27e60786d7b487188970f4b4169cc055784e21eb71d410cc5
MergeChanges /etc/ /boot/device.hints
ServerName update.FreeBSD.org
UpdateIfUnmodified /etc/ /var/ /root/ /.cshrc /.profile
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # setenv PAGER cat
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # freebsd-update fetch install
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 13.0-RELEASE-p5.
No updates are available to install.
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # history -S
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # history | tail -n 21
   556  15:16   etcupdate -B
…
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # history | tail -n 29
   549  15:04   gitup -c release
   550  15:04   setenv PAGER cat ; freebsd-update fetch install
   551  15:10   cat /etc/freebsd-update.conf
   552  15:11   ls -hl /etc/freebsd-update.conf
   553  15:11   grep -v \# /etc/freebsd-update.conf
   554  15:11   ls -hl /etc/freebsd-update.conf
   555  15:11   grep -v \# /etc/freebsd-update.conf | sort
   556  15:16   etcupdate -B
   557  15:16   etcupdate -B
   558  15:16   etcupdate -p
   559  15:16   etcupdate resolve
   560  15:20   gitup release
   561  15:24   history | grep gitup
   562  15:24   history -S
   563  15:30   pkg install qterminal
   564  15:33   gitup -v
   565  15:33   gitup --version
   566  15:33   man gitup
   567  15:34   gitup -V
   568  15:34   freebsd-version -kru
   569  15:34   pkg -vv | grep -e url -e enabled
   570  15:34   gitup -c release
   571  15:41   gitup release
   572  15:42   grep -v \# /etc/freebsd-update.conf | sort
   573  15:43   setenv PAGER cat
   574  15:43   freebsd-update fetch install
   575  15:51   history -S
   576  15:53   history | tail -n 21
   577  15:54   history | tail -n 29
root@mowa219-gjp4-vm-freebsd-13-zfs:~ #
```

<https://cgit.freebsd.org/src/log/?h=releng/13.0> as expected, nothing since 2021-11-03.

If you wonder about the four unnecessary etcupdate(8) commands:

I was clutching at straws (in the context of the other topic, _freebsd-update fetching but failing to install updates_)
they seem to have no impact.


The screenshot below is to remind me that I have a VirtualBox snapshot of the machine as it was thirty days ago.

jmehr I'm inclined to treat the peculiarity above as negligible, a one-off without explanation. Alternatively: if you feel that it deserves further investigation, I might find time later in the month to thrash around with the 2021-12-03 snapshot as a basis.


----------



## grahamperrin@ (Jan 2, 2022)

grahamperrin said:


> It's _possible_ that I keyed Control-C during the first of the commands, although I can't imagine doing so. The linked post confirms that the first command ran to completion; _Done_.



More specifically, quoting from the linked post: 


```
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # gitup -c release                                     # Scanning local repository...
# Host: git.freebsd.org
# Port: 443
# Repository Path: /src.git
# Target Directory: /usr/src
# Have: 2646dd665909e60a369015c17cb602515e6025dc
# Want: 2646dd665909e60a369015c17cb602515e6025dc
# Branch: releng/13.0
# Done.
root@mowa219-gjp4-vm-freebsd-13-zfs:~ # …
```

after the word _Done_ appeared, there was a long wait
I *did* wonder whether something had got stuck
I *might have* impatiently keyed Control-C to regain the command prompt.
Please: what exactly happens during that waiting period after the word _Done_ appears?


----------



## astyle (Jan 2, 2022)

A possibility that comes to my mind is that gitup(1) _could be_ not exiting very gracefully, or the machine was waiting on a different mutex before deciding that `sh` and `gitup` merit attention from the processor. As in, there could be a hung process somewhere. Try running `# ps -ax` in a different terminal/tty to verify my assumptions.


----------



## aafbsd (Jan 3, 2022)

jmehr said:


> I still can't reproduce it but I'm thinking this might be due to differences in the ordering of files returned by the opendir() and readdir() calls.  I just committed aafbsd's patch (thank you!).  How does it work for both of you now?


I don't have that old test tree anymore but I just created a new one and will see what happens
in the next days.

BTW, I had trouble regarding the new regex feature: The man pages simply say:

"Regular expressions are supported"

So, is everything here treated as regex or do they have to be prefixed (and optionally postfixed) with
something (e.g. '/')? This should be clarified, especially for people coming from older versions...
In my case, I didn't know if I have to escape the '.' or not .


----------

