# Problem with pkg upgrade



## Friend Of Jolly Devil (Jan 22, 2022)

From time to time I had problems with pkg download speed, but everything else was ok, and starting it again was solving the issue. At the beginning of this week 'pkg upgrade' didn't work, after downloading pkg complained about package size mismatch error. Starting it again and again didn't help, when left alone it was just downloading one package in a loop, but it was increasing downloaded package count. Question is, what happened? How to repeat that behaviour on my own infrastructure? Thanks in advance.


----------



## 3301 (Jan 23, 2022)

Looks like some hardware problem with network card (is it wifi interface?) - try using another interface (maybe some usb stick?)
or with memory - run memtest and check if your RAM is ok.


----------



## eternal_noob (Jan 23, 2022)

Maybe try another pkg mirror. See https://forums.freebsd.org/threads/slow-package-fetching.83230/page-2#post-546049


----------



## grahamperrin@ (Jan 23, 2022)

Friend Of Jolly Devil said:


> … when left alone it was just downloading one package in a loop, but it was increasing downloaded package count. …



That's a peculiar symptom. Please share outputs from the commands below. 

`tunefs -p /`

`freebsd-version -kru ; uname -aKU`

`pkg -vv | grep -e url -e enabled`

`pkg prime-origins | sort`


----------



## Friend Of Jolly Devil (Jan 23, 2022)

3301 said:


> Looks like some hardware problem with network card (is it wifi interface?) - try using another interface (maybe some usb stick?)
> or with memory - run memtest and check if your RAM is ok.



No, it's Realtek 8168/8111 ethernet card. I don't have any wifi card, I have a 3g gsm modem but last time I checked it didn't work at all. Memtest returns no errors



eternal_noob said:


> Maybe try another pkg mirror. See https://forums.freebsd.org/threads/slow-package-fetching.83230/page-2#post-546049



According to this thread it is not a good idea, but maybe I'll try.



grahamperrin said:


> That's a peculiar symptom. Please share outputs from the commands below.
> 
> `tunefs -p /`
> 
> ...



I have zfs, not ufs.

```
fbsdzroot               9.26G   274G       96K  /fbsdzroot
fbsdzroot/ROOT          6.92G   274G       96K  none
fbsdzroot/ROOT/default  6.92G   274G     6.92G  /
fbsdzroot/tmp            280K   274G      280K  /tmp
fbsdzroot/usr           2.33G   274G       96K  /usr
fbsdzroot/usr/home       648M   274G      648M  /usr/home
fbsdzroot/usr/ports     1.01G   274G     1.01G  /usr/ports
fbsdzroot/usr/src        702M   274G      702M  /usr/src
fbsdzroot/var           1.01M   274G       96K  /var
fbsdzroot/var/audit       96K   274G       96K  /var/audit
fbsdzroot/var/crash       96K   274G       96K  /var/crash
fbsdzroot/var/log        516K   274G      516K  /var/log
fbsdzroot/var/mail       120K   274G      120K  /var/mail
fbsdzroot/var/tmp        112K   274G      112K  /var/tmp

pool: fbsdzroot
 state: ONLINE
config:

    NAME           STATE     READ WRITE CKSUM
    fbsdzroot  ONLINE       0     0     0
      ada0p4       ONLINE       0     0     0

errors: No known data errors
```


```
13.0-RELEASE-p6
13.0-RELEASE-p6
13.0-RELEASE-p6
 
FreeBSD *** 13.0-RELEASE-p6 FreeBSD 13.0-RELEASE-p6 #0: Mon Jan 10 06:28:50 UTC 2022     root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64 1300139 1300139
```


```
url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly",
    enabled         : yes,
```

Maybe it should be better to change it to https? Now it's the default.


```
archivers/cabextract
archivers/php80-phar
archivers/php80-zip
archivers/php80-zlib
archivers/unzip
audio/faac
audio/nas
audio/pulseaudio
converters/php80-iconv
databases/mariadb105-client
databases/mariadb105-server
databases/pecl-redis
databases/php80-mysqli
databases/php80-pdo_mysql
databases/php80-pdo_sqlite
databases/php80-sqlite3
databases/phpmyadmin5
devel/automake
devel/git
devel/gmake
devel/libburn
devel/libtool
devel/libublio
devel/p5-Locale-gettext
devel/php80-readline
devel/php80-tokenizer
devel/pkgconf
editors/emacs
editors/libreoffice
editors/vim
emulators/linux_base-c7
ftp/php80-curl
graphics/drm-kmod
graphics/evince
graphics/feh
graphics/phototonic
graphics/php80-gd
graphics/ristretto
japanese/lynx
java/icedtea-web
lang/php80
mail/thunderbird
math/galculator
math/php80-bcmath
misc/freebsd-doc-en
misc/freebsd-doc-pl
misc/help2man
misc/mc-nox11
misc/xfce4-wm-themes
multimedia/mp4v2
multimedia/transcode
multimedia/vlc
net/ifstat
net/php80-soap
net/rsync
polish/libreoffice
ports-mgmt/dialog4ports
ports-mgmt/pkg
print/foomatic-db-engine
print/gutenprint
print/hplip
print/qpdfview
print/texinfo
security/openssl
security/php80-filter
security/php80-openssl
sysutils/automount
sysutils/cdrdao
sysutils/desktop-installer
sysutils/dmidecode
sysutils/dvd+rw-tools
sysutils/exfat-utils
sysutils/fusefs-exfat
sysutils/fusefs-ext2
sysutils/fusefs-hfsfuse
sysutils/fusefs-libs
sysutils/fusefs-ntfs
sysutils/fusefs-simple-mtpfs
sysutils/php80-fileinfo
sysutils/php80-posix
sysutils/xfburn
sysutils/xfce4-battery-plugin
sysutils/xfce4-wavelan-plugin
textproc/php80-ctype
textproc/php80-dom
textproc/php80-simplexml
textproc/php80-xml
textproc/php80-xmlwriter
www/chromium
www/elinks
www/firefox
www/links
www/nginx
www/php80-session
www/w3m-img
x11-drivers/xf86-video-intel
x11-drivers/xf86-video-mga
x11-drivers/xorg-drivers
x11-fm/pcmanfm
x11-fonts/bitstream-vera
x11-fonts/droid-fonts-ttf
x11-fonts/fira
x11-fonts/liberation-fonts-ttf
x11-fonts/webfonts
x11-themes/lumina-themes
x11-themes/lxappearance
x11-themes/qtcurve-gtk2
x11-themes/qtcurve-qt5
x11-toolkits/tk87
x11-wm/i3
x11-wm/lxqt
x11-wm/xfce4
x11/dmenu
x11/dzen2
x11/hs-xmobar
x11/i3blocks
x11/i3lock
x11/i3status
x11/j4-dmenu-desktop
x11/lemonbar
x11/lumina
x11/qterminal
x11/rofi
x11/slim
x11/xdm
x11/xfce4-screensaver
x11/xfce4-screenshooter-plugin
x11/xlockmore
x11/xorg
x11/xsm
x11/zenity
```


----------



## grahamperrin@ (Jan 23, 2022)

Thanks.



Friend Of Jolly Devil said:


> Maybe it should be better to change it to https?



No need.



Friend Of Jolly Devil said:


> zfs



OK. Try this:

create, activate then boot a new boot environment
`pkg upgrade -f  -y`
It will take some time, but it's a likely way of resolving issues. It's thorough.

bectl(8)


----------



## Friend Of Jolly Devil (Jan 25, 2022)

It could be my ethernet kernel driver, as in this thread, but I don't have watchdog timeouts in /var/log/messages. And isos downloading works fine. I've also found suggestion to force bootstrap of pkg, it may solve some issues. I think I'll make a fresh install on another computer first and see what'll happen.


----------



## SirDice (Jan 25, 2022)

Access to your particular mirror could be problematic. It pays to try another mirror every now and then, just to check. In some cases that might even make things worse though. Downloaded packages are cached in /var/cache/pkg/, especially with checksum errors it can help to remove that cached version. You can also use pkg-clean(8) to clean up the cache.


----------



## Friend Of Jolly Devil (Jan 25, 2022)

I think I've finally got it. It seems that it was a mixture of bad luck and some pkg imperfection. At the exact moment I was upgrading, there was a minor version change of openjdk8, and in the result pkg tried to download a frankensteined version, it started download with one version, and ended with another. It noticed the mismatch, but it didn't noticed version change, so it repeatedly tried to download the old version, while the contents were from the new one, hence the mismatch. I should pkg update after the first mismatch, it would solved it. Can someone confirm it is plausible?


----------



## grahamperrin@ (Jan 26, 2022)

Friend Of Jolly Devil said:


> … I should pkg update after the first mismatch, it would solved it. …



I doubt it. 

Update is part of the routine, for example:


```
root@mowa219-gjp4-8570p-freebsd:~ # pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating poudriere repository catalogue...
poudriere repository is up to date.
All repositories are up to date.
Checking for upgrades (2 candidates): 100%
Processing candidates (2 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
root@mowa219-gjp4-8570p-freebsd:~ #
```


----------



## grahamperrin@ (Jan 26, 2022)

Friend Of Jolly Devil said:


> From time to time I had problems with pkg download speed, … 'pkg upgrade' … mismatch … Starting it again and again didn't help, when left alone it was just downloading one package in a loop, …



In this recent case <https://old.reddit.com/comments/sbeekd/-/> the end user diagnosed a side effect with a socks5 proxy / ssh tunnel.



Friend Of Jolly Devil said:


> … increasing downloaded package count. …



This symptom is vaguely familiar. Probably a cosmetic bug, which I never bothered to report, because it's sometimes difficult to reproduce.


----------



## Friend Of Jolly Devil (Jan 26, 2022)

grahamperrin said:


> I doubt it.
> 
> Update is part of the routine, for example:
> 
> ...



But it does this only once at the beginning. When change in repositories happens after this updating, what then? Does pkg auto-detect it?



grahamperrin said:


> In this recent case <https://old.reddit.com/comments/sbeekd/-/> the end user diagnosed a side effect with a socks5 proxy / ssh tunnel.



I don't have any socks5 proxy / ssh tunnel. It seems that he had the exact same problem, but probably it solved itself independently while he was trying socks5 proxy / ssh tunnel setting, hence his statement. 
In my case I allowed the second fetch to complete, it was broken too. After SirDice post I checked /var/cache/pkg. I have two openjdk8 files there: 1-st version 8.302.08.1_2, its size matches the first download,  and 2-nd v. 8.312.07.1 which size matches the second download, so there was a change in package version after 'pkg update' completed itself. It looks like pkg saw this change only partially, and the second time it claimed downloading the exact same package - at least exactly the same version number in both cases was printed. I can't tell if it's just a cosmetic bug or not, it's hard to find more detailed info about how pkg exactly works. 
Is there any way to download both versions manually to compare them with the ones in my cache?



grahamperrin said:


> This symptom is vaguely familiar. Probably a cosmetic bug, which I never bothered to report, because it's sometimes difficult to reproduce.



I suspect it's because of combination of random factors, maybe the above explanation will help somehow to reproduce it.


----------



## grahamperrin@ (Jan 26, 2022)

Friend Of Jolly Devil said:


> When change in repositories happens after this updating, what then? Does pkg auto-detect it?



Yes.


----------



## Friend Of Jolly Devil (Jan 27, 2022)

I've installed FreeBSD 13.0 p6 on a different computer with intel network card, I have issues with pkg there too. At the beginning pkg didn't work at all, I had to manually set a working mirror like eternal_noob suggested. It seems that pkg insists on using isc mirror, which is completely down for quite some time now.


----------



## sko (Jan 27, 2022)

Try to manually set another mirror by editing /etc/pkg/FreeBSD.conf and changing 'pkg.freebsd.org' to one from the list at the bottom of the https://pkg.freebsd.org/ page. The page tells you which mirror you are hitting (e.g. pkg0.pkg.freebsd.org), so you can choose another one from the list.

If the problem persist with other mirrors and on different machines, there may be something fishy with your connection. E.g. proxies with TLS-termination and/or intrusive DPI come to mind...

Most likely you are just hitting the mirror during a resync. This sometimes can lead to inconsistencies between available packages and the database. Trying again a few hours later (or sometimes even days e.g. after a major release or some other event that causes lots of packages to be upgraded/rebuilt) and maybe cleaning the cache and rebuilding the catalogue (`pkg clean -a && pkg update -f`) usually works.
If the problem is only reproducible with one particular mirror (the pkg.freebsd.org page usually tells you which mirror is selected for your location) over a longer period of time, open a PR so someone with insights to that mirror can take a look at it.


----------



## SirDice (Jan 27, 2022)

sko said:


> Try to manually set another mirror by editing /etc/pkg/FreeBSD.conf and changing 'pkg.freebsd.org' to one from the list at the bottom of the https://pkg.freebsd.org/ page.


Please don't edit /etc/pkg/FreeBSD.conf, make a /usr/local/etc/pkg/repos/FreeBSD.conf and put the URL in there. No need to copy anything else:

```
FreeBSD: {
  url: <url of the mirror>
}
```

The reasoning for this is that /etc/pkg/FreeBSD.conf might get changed with an update and that could remove your changes. Setting it in /usr/local/etc/pkg/repos/FreeBSD.conf will make sure your changes are going to survive this.


----------



## sko (Jan 27, 2022)

SirDice said:


> The reasoning for this is that /etc/pkg/FreeBSD.conf might get changed with an update and that could remove your changes. Setting it in /usr/local/etc/pkg/repos/FreeBSD.conf will make sure your changes are going to survive this.


The changes I mentioned were only meant as a temporary measure to verify/locate the problem; I should have pointed that out.
But yes - good practice is to always use /usr/local/etc for such configuration overrides if possible.


----------



## Friend Of Jolly Devil (Jan 27, 2022)

I've tried few different mirrors (in /usr/local/etc/pkg/repos/FreeBSD.conf)
pkg0.pkt.FreeBSD.org works the best (often over 1 MB/s)
pkg0.bme.FreeBSD.org works, but is veeery slooow (even just as low as < 8KB/s)
pkg0.isc.FreeBSD.org doesn't work at all.
it's reproducible, so it's not my connection problem.
It seems that https://pkg.freebsd.org/ has no idea that isc doesn't work, and it redirects to it sometimes.


----------



## grahamperrin@ (Jan 28, 2022)

pkg0.isc.freebsd.org



Friend Of Jolly Devil said:


> … It seems that pkg insists on using isc mirror, …





sko said:


> … Most likely you are just hitting the mirror during a resync. …





Friend Of Jolly Devil said:


> …
> 
> pkg0.isc.FreeBSD.org doesn't work at all.
> 
> it's reproducible, …



<https://github.com/ehaupt/fastest_pkg/issues/2#issuecomment-1000897777> (2021-12-24), again on 2021-12-27:



> `pkg0.isc.freebsd.org: 0.0 B/s`



<https://github.com/ehaupt/fastest_pkg/issues/2#issuecomment-1001326332> (2021-12-27):



> `pkg0.isc.freebsd.org` seem to be unreachable ATM.



<https://github.com/ehaupt/fastest_pkg/issues/2#issuecomment-1002160401> (2021-12-28):



> That's correct. If you run it with `-v` you'll see the `cURL` error message.



01:09 today:



Spoiler: (28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2746 ms: Operation timed out')





```
% date ; fastest_pkg -v
Fri 28 Jan 2022 01:09:47 GMT
pkg0.tuk.freebsd.org: 69%
(28, 'Operation timed out after 5000 milliseconds with 4672768 out of 6699232 bytes received')

pkg0.tuk.freebsd.org: 934.5 kB/s
pkg0.nyi.freebsd.org: 88%
(28, 'Operation timed out after 5000 milliseconds with 5908905 out of 6699232 bytes received')

pkg0.nyi.freebsd.org: 1.2 MB/s
pkg0.isc.freebsd.org: 0%
(28, 'Failed to connect to pkg0.isc.freebsd.org port 80 after 2746 ms: Operation timed out')
pkg0.isc.freebsd.org: 0.0 B/s
pkg0.bme.freebsd.org: 100%
pkg0.bme.freebsd.org: 6.9 MB/s
pkg0.pkt.freebsd.org: 100%
pkg0.pkt.freebsd.org: 6.8 MB/s

Fastest:
pkg0.bme.freebsd.org: 6.9 MB/s


Write configuration:
mkdir -p /usr/local/etc/pkg/repos/
echo 'FreeBSD: { url: "http://pkg0.bme.freebsd.org/${ABI}/latest" }' \
        > /usr/local/etc/pkg/repos/FreeBSD.conf


%
```






Friend Of Jolly Devil said:


> … It seems that pkg insists on using isc mirror, …



pkg-update(8) has no option `-v` for verbosity.



Friend Of Jolly Devil said:


> … It seems that https://pkg.freebsd.org/ has no idea that isc doesn't work, and it redirects to it sometimes. …



When `http://pkg0.isc.freebsd.org/` occurs, in the web browser, does the page time out?


----------



## grahamperrin@ (Jan 28, 2022)

Friend Of Jolly Devil said:


> … info about how pkg exactly works. …



<https://github.com/freebsd/pkg#readme>


----------



## Friend Of Jolly Devil (Jan 28, 2022)

grahamperrin said:


> pkg-update(8) has no option `-v` for verbosity.
> 
> 
> 
> When `http://pkg0.isc.freebsd.org/` occurs, in the web browser, does the page time out?



It has not. Yes, the page times out.



grahamperrin said:


> <https://github.com/freebsd/pkg#readme>



It is outdated (for example link to pkg wikipage) and contains only general info. Probably I should try to understand its source code


----------

