# Problem upgrading jails (pkg, freebsd-update)



## poorandunlucky (Apr 28, 2021)

So I have these jails now, and I want to install Samba on my www jail.

I've upgraded to 12.2-p6, pkg -vv shows kernel 120200.

I've upgraded my jail with freebsd-update -b /jails/www

There was an error, some directories for the certificates under /usr/share/[certs]/[trusted/blacklisted] didn't exist, so freebsd-update failed up upgrade those files in there, I manually created the directories.  Re-running freebsd-update -b says there's nothing to do.

Going back to the jail, and running pkg -vv shows osversion 120100.

Here's the log from my last transactions before posting here...


```
root@www:/ # cd /usr/share/certs/
root@www:/usr/share/certs # ls
blacklisted     trusted
root@www:/usr/share/certs # cd ..
root@www:/usr/share # cd ..
root@www:/usr # cd ..
root@www:/ # exit
exit
tyler@M6500:/home/tyler # freebsd-update -b /jails/www/ fetch install
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 12.2-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 12.2-RELEASE-p6.
No updates are available to install.
tyler@M6500:/home/tyler # jexec www /bin/tcsh
root@www:/ # pkg -vv
Version                 : 1.14.5
PKG_DBDIR = "/var/db/pkg";
PKG_CACHEDIR = "/var/cache/pkg";
PORTSDIR = "/usr/ports";
INDEXDIR = "";
INDEXFILE = "INDEX-12";
HANDLE_RC_SCRIPTS = false;
DEFAULT_ALWAYS_YES = false;
ASSUME_ALWAYS_YES = false;
REPOS_DIR [
    "/etc/pkg/",
    "/usr/local/etc/pkg/repos/",
]
PLIST_KEYWORDS_DIR = "";
SYSLOG = true;
ABI = "FreeBSD:12:amd64";
ALTABI = "freebsd:12:x86:64";
DEVELOPER_MODE = false;
VULNXML_SITE = "http://vuxml.freebsd.org/freebsd/vuln.xml.bz2";
FETCH_RETRY = 3;
PKG_PLUGINS_DIR = "/usr/local/lib/pkg/";
PKG_ENABLE_PLUGINS = true;
PLUGINS [
]
DEBUG_SCRIPTS = false;
PLUGINS_CONF_DIR = "/usr/local/etc/pkg/";
PERMISSIVE = false;
REPO_AUTOUPDATE = true;
NAMESERVER = "";
HTTP_USER_AGENT = "pkg/1.14.5";
EVENT_PIPE = "";
FETCH_TIMEOUT = 30;
UNSET_TIMESTAMP = false;
SSH_RESTRICT_DIR = "";
PKG_ENV {
}
PKG_SSH_ARGS = "";
DEBUG_LEVEL = 0;
ALIAS {
    all-depends = "query %dn-%dv";
    annotations = "info -A";
    build-depends = "info -qd";
    cinfo = "info -Cx";
    comment = "query -i \"%c\"";
    csearch = "search -Cx";
    desc = "query -i \"%e\"";
    download = "fetch";
    iinfo = "info -ix";
    isearch = "search -ix";
    prime-list = "query -e '%a = 0' '%n'";
    prime-origins = "query -e '%a = 0' '%o'";
    leaf = "query -e '%#r == 0' '%n-%v'";
    list = "info -ql";
    noauto = "query -e '%a == 0' '%n-%v'";
    options = "query -i \"%n - %Ok: %Ov\"";
    origin = "info -qo";
    provided-depends = "info -qb";
    rall-depends = "rquery %dn-%dv";
    raw = "info -R";
    rcomment = "rquery -i \"%c\"";
    rdesc = "rquery -i \"%e\"";
    required-depends = "info -qr";
    roptions = "rquery -i \"%n - %Ok: %Ov\"";
    shared-depends = "info -qB";
    show = "info -f -k";
    size = "info -sq";
}
CUDF_SOLVER = "";
SAT_SOLVER = "";
RUN_SCRIPTS = true;
CASE_SENSITIVE_MATCH = false;
LOCK_WAIT = 1;
LOCK_RETRIES = 5;
SQLITE_PROFILE = false;
WORKERS_COUNT = 0;
READ_LOCK = false;
PLIST_ACCEPT_DIRECTORIES = false;
IP_VERSION = 0;
AUTOMERGE = true;
VERSION_SOURCE = "";
CONSERVATIVE_UPGRADE = true;
PKG_CREATE_VERBOSE = false;
AUTOCLEAN = false;
DOT_FILE = "";
REPOSITORIES {
}
VALID_URL_SCHEME [
    "pkg+http",
    "pkg+https",
    "https",
    "http",
    "file",
    "ssh",
    "ftp",
    "ftps",
    "pkg+ssh",
    "pkg+ftp",
    "pkg+ftps",
]
ALLOW_BASE_SHLIBS = false;
WARN_SIZE_LIMIT = 1048576;
METALOG = "";
OSVERSION = 1201000;
IGNORE_OSVERSION = false;
BACKUP_LIBRARIES = false;
BACKUP_LIBRARY_PATH = "/usr/local/lib/compat/pkg";


Repositories:
  FreeBSD: {
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
root@www:/ #
```

Where did I screw-up?

Also: hello  *waves*


----------



## SirDice (Apr 28, 2021)

`freebsd-update fetch install` only downloads and installs patches for the version it has now. Your jail appears to be 12.1-RELEASE, you probably never upgraded it. It's not automagically upgraded when you upgrade the host OS. You need to use `freebsd-update -r 12.2-RELEASE upgrade` on the jail.



poorandunlucky said:


> Going back to the jail, and running pkg -vv shows osversion 120100.


Run `freebsd-version -u` to see version.


----------



## poorandunlucky (Apr 29, 2021)

`freebsd-version -u`shows 12.2-RELEASE-p6 from both the base OS, and from inside the jail (through `jexec`).

Running pkg from inside the jail tells me

`pkg: http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly/All/samba411-4.11.8.txz: Not Found`

Running `pkg -j www samba411` install gives me

`Newer FreeBSD version for package php73-pear-channel-symfony:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1202000
- running kernel: 1201000`

I tried running `freebsd-update -b /jails/www -r 12.2-RELEASE` as well, and it's also telling me it's already at that version...

Related yet Unrelated Questions:

What's good practice with jails?  Working on them by opening a shell, or operating from the base OS through the --jail options when possible?  Like upgrading, should I do it from the jail, or with the freebsd-update -b option?

Howcome trying to install samba from a jail shell tells me the package can't be found, but works fine if I do it from the base OS with the -j option?  Seems like at that stage it should just be downloading a file, I don't understand the error...

And back to the main issue, considering both the base OS, and the jail are at 12.2-RELEASE, howcome I'm getting that error?

I made a snapshot, I could override, and rollback if it doesn't work, but I'd just like to understand, I'm not experienced with jails so I think I'm probably doing something wrong in my system administration, so I'd like to take good habits, rather than build a pile of mistakes...  I'm kinda proud of my OS right now, and aside my personal folder being a mess, my server is pretty nice, and I don't want to feel like I want to wipe it and start over again...  It's running well, it's useful, and I really like it (obviously, I'm still here)...

Like when you upgrade your jails, what do you do?  I assume they're on their own dataset, and all...  I build them with bsdinstall, and then I open shells inside with jexec, or I'll use the -j option that I think just chroots to the jail basedir when appropriate in the script, so I mix the two without caring much, but is there deeper stuff I'm missing?

I even ran `pkg update` from inside the jail, maybe it needed to update its file that tells it the kernel version (I don't know where it gets it from, that might acttually be a good place to start tracking backwards, if you can help)...

I'm just really stumped by that problem, everything was going so smoothly I thought everything was in order...  I don't want to update to the newest release yet because I don't want to go through all the config right now, I just want samba on the www server at this time...  I'll enjoy going that road later, but right now I just want to get this sorted out to make sure everything's fine, and I'm not going to screw everything up again... and make a little website just to refresh myself, and show something to someone (which is what I set out to do in the first place, I'm also trying to remain focussed on my objective)...

I have samba on another jail, and it's running fine, I use it every day, all my files are there now, besides the obligatory stuff I have to keep local to Windows, everything goes on my file server, even game recordings from Windows, everything's set to save to my Samba network drive that's my home folder on that jail...  been running for like a year like that now...  I want to put Nextcloud on it, re-setup my DNS, I lost a lot of files because of Windows and Microsoft-related issues, and carelessness, and disorganization, plus I'm kinda depressed, and I also took up electronics as a hobby, so it's not going very fast, but at least I think it's fairly solid now...  It could just be the package script that has a typo, or it could be something I've done improperly, and I want to know because I really like my server...

I don't understand why that error, and like I said, I don't want to screw things up anymore, or any more...  Just asking for jail best practices, and clues where to look for the problem...

FreeBSD is very important to me, not only is it cool to watch debug messages whiz through on a console window, but it also speaks "my language" it just makes sense to me, I'm actually very passionate about it, I didn't choose it over Ubuntu because I liked the logo, the way it's organized, and the way the people who make it write, the code, the structure, it just...  I think it's beautiful, I find beauty in it, and because it's so rare to find something like that, well, I keep... trying...  *shrugs*

... and yes, even though I've been absent for a while, I've thought about you a few times, if you wonder, , wondering, and hoping, that you're healthy, and happy...

You've always tried to help, and I appreciate, even if you weren't always helpful.  

(sure spilled my heart out there...  i hope i don't regret it in the morning  lol  )


----------



## ShelLuser (Apr 29, 2021)

poorandunlucky said:


> What's good practice with jails?  Working on them by opening a shell, or operating from the base OS through the --jail options when possible?  Like upgrading, should I do it from the jail, or with the freebsd-update -b option?
> 
> Howcome trying to install samba from a jail shell tells me the package can't be found, but works fine if I do it from the base OS with the -j option?


That sounds like a badly set up jail to me. When done properly then a jail will behave in the same way as its host, so it can install packages, build ports, etc.

Best practice is obviously to use jexec(8) to access the jail. Don't forget that it contains running processes and if you simply overrule any possible (soft)lock and start messing around you're in for a _really_ bad time.



poorandunlucky said:


> Like when you upgrade your jails, what do you do?


Depends, usually the same thing as you'd do on the host.

Either you upgrade from inside the jail or you shut it down and then perform the upgrade from the outside. I usually shut the whole thing down and then "forcefully" extract the archives for the latest version, during which I use mergemaster(8) to help me sort out the configuration (basically I roughly follow the same procedure as if I was upgrading using the source tree).

But your mileage varies. Note that this is all explained in the FreeBSD handbook.



poorandunlucky said:


> I assume they're on their own dataset, and all...  I build them with bsdinstall, and then I open shells inside with jexec, or I'll use the -j option that I think just chroots to the jail basedir when appropriate in the script, so I mix the two without caring much, but is there deeper stuff I'm missing?


Why assume?  You build them, so you should know. No, a dataset is definitely not implied nor required (but can be useful). You open shells? It does more than that, also see the manual pages. You don't just open a shell, you start a new child process within your jail. Try using `w` or `ps` from inside the jail to see what I mean.


----------



## poorandunlucky (Apr 29, 2021)

I might've upgraded it without shutting it down...

There's not much on there,I think just nginx, pgsql, php-fpm, ...  It's going to be a small hassle, but none of it is really configured, anyway...  I think I'm going to just rebuilld it to make sure, I think that's my best move...

Not having a set method to upgrade jails, and there being more than one way about it, I think I I must've upgraded it without shitting it down like you suggested...

I'll also try to install that package on another jail see if it's not that...

There doesn't seem to be a textbook solution to this, so I'm going to try and find out, and I'll let you know what comes out of it...

Thanks again


----------



## SirDice (Apr 30, 2021)

poorandunlucky said:


> I think I'm going to just rebuilld it to make sure, I think that's my best move...


You might want to use a tool to create the jails, most of them also have functions to update and upgrade them. I've been a long time sysutils/ezjail user but recently tried to change to sysutils/bastille.


----------



## poorandunlucky (May 31, 2021)

So I think the problem is that when I upgrade my jails, I use freebsd-update -b /jails/<jailname> fetch install and it works fine from the base OS,, however, if try from within the jail (whtouth the -b, obviously(, it says it can't find the files for my architecture, and I have to built it from source.

The original problem was that pkg was complaining that my kernel version was 12.1,but pkg was trying to install for 12.2.

I always thought it was a file like /var/pkg/version or something like that, but I looked at /boot/, and my files there are from 2019...  In /bin/, too, like freebsd-update -b /jail/<jailname> did nothing apparently...

Do you really always have to build jails from source?  Why can't you just extract the packages like freebsd-install does?

That's so weird to me...

I always expected freebsd-update -b to just be the way to update and upgrade jails, it's how the base OS gets updated/upgraded and it's just .txz files that get extracted for my arch...

Why do I have to upgrade jails from source?  Do I also have to update (patch) them from source, too?

That seems weird, and tednous, and there has to be a reason...  I would sincerely expect freebsd-update to udate jails, I installed them with bsdinstall, so it's just tar files thar get extracted with some scripts for permissions and whatnot...

Why is upgrading/updating a jail so darn complicated?

I don't undetstand why those other tools don't work from within/whoutout the jail since it's just package extraction, and doing it on the base system surely can't be less complicated than doing it in a jail...  A jail can be shurdown to the point where it's just inert folders with nothing running, no usernale apps to keep running until the next boot, no...  It could just extract the packages straight into the jail base directories!

Wjy?  Is it s way, like a milestone, to get me to built my kernel, and change the source?  Is there another technical reason?

Sure I have a trash app that I could add to /src/, and I guess I could remove dome default modules from the GENERIC kernel, but it's so light already, and ...

*hangs head*

Does anyone know why it's like that?

It's not pissing me off, but I'm only run buildworld installworld a few times, and I'm kind scared I'm going to screw things up, or that it's going to mess my configuration files, or something...


----------



## quicktrick (Jul 18, 2021)

ShelLuser said:


> Either you upgrade from inside the jail or you shut it down and then perform the upgrade from the outside.


Logically, you are right, but in the handbook they seem to suggest doing an upgrade on a running jail.  See 15.4.3. Keeping Jails Patched and up to Date

I am also interested in this question. What is the right thing to do?


----------



## michael_hackson (Jul 18, 2021)

Perhaps this thread can help: Thread 49079 especially the wiki link in the last comment.

Since the OS and jails are seperately upgraded one must specify what current version the jail world is when running the `freebsd-update fetch` and `freebsd-update upgrade` commands.

In a nutshell: Stop your jail, `#chroot /path/to/mountedjail/`, check version of world in jail with `freebsd-version -u`. Follow the thread above.

No need to reboot during upgrade since jail already share Kernel with OS.


----------



## quicktrick (Jul 18, 2021)

I meant upgrading the packages from the host. Do I first need to stop the jail?  The handbook seems to suggest doing all the upgrades (`pkg`, `freebsd-update`) on a running jail.


----------



## grahamperrin@ (Jul 19, 2021)

poorandunlucky said:


> … Why is upgrading/updating a jail so darn complicated? …



My first experiment with sysutils/mkjail: 


```
root@mowa219-gjp4-8570p:~ # nano /usr/local/etc/mkjail.conf
root@mowa219-gjp4-8570p:~ # uname -KrU
14.0-CURRENT 1400025 1400025
root@mowa219-gjp4-8570p:~ # mkjail create -j 13 -v 13.0-RELEASE
Creating copperbowl/jails/13...
Extracting base into /jails/13...
Extracting lib32 into /jails/13...
Updating 13 jail...

src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching public key from update2.freebsd.org... done.
Fetching metadata signature for 13.0-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 10 patches.....10 done.
Applying patches... done.
Fetching 6 files... ... done.
The following files will be added as part of updating to
13.0-RELEASE-p3:
/usr/include/c++/v1/barrier
/usr/include/c++/v1/concepts
/usr/include/c++/v1/execution
/usr/include/c++/v1/latch
/usr/include/c++/v1/numbers
/usr/include/c++/v1/semaphore
/usr/include/c++/v1/tr1/barrier
/usr/include/c++/v1/tr1/concepts
/usr/include/c++/v1/tr1/execution
/usr/include/c++/v1/tr1/latch
/usr/include/c++/v1/tr1/numbers
/usr/include/c++/v1/tr1/semaphore
The following files will be updated as part of updating to
13.0-RELEASE-p3:
/bin/freebsd-version
/lib/libcasper.so.1
/usr/bin/bc
/usr/bin/dc
/usr/lib/libradius.a
/usr/lib/libradius.so.4
/usr/lib/libradius_p.a
/usr/lib32/libcasper.so.1
/usr/lib32/libradius.a
/usr/lib32/libradius.so.4
/usr/lib32/libradius_p.a
Installing updates...Scanning /jails/13/usr/share/certs/trusted for certificates...
 done.

Now put something like the following in /etc/jail.conf:

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
path = /jails/$name;
securelevel = 2;

13 {
    host.hostname = "13";
    ip4.addr = 192.168.1.10;
    ip6.addr = 2001:⋯:⋯:⋯::2;
    persist;
}

and then you can start the jail like so:

# service jail start 13

root@mowa219-gjp4-8570p:~ # man mkjail
No manual entry for mkjail
root@mowa219-gjp4-8570p:~ # mkjail --help
Unknown command '--help'
usage: mkjail command [options]

Commands:
    create      -- Create jail
    getrelease  -- Get FreeBSD release set tarballs
    update      -- Update jail with latest FreeBSD security/errata patches
    upgrade     -- Upgrade jail to a newer RELEASE

mkjail.sh: 2019, feld@FreeBSD.org

root@mowa219-gjp4-8570p:~ # mkjail update -h
usage: mkjail update [-a] | [-j JAILNAME]

        -a Update all running jails
        -h Show help
        -j Jail name

mkjail.sh: 2019, feld@FreeBSD.org

root@mowa219-gjp4-8570p:~ # service jail start 13
Cannot 'start' jail. Set jail_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.
root@mowa219-gjp4-8570p:~ # service jail onestart 13
Starting jails: cannot start jail  "13":
jail: 13: path: variable "name" not found
.
root@mowa219-gjp4-8570p:~ # nano /etc/jail.conf
root@mowa219-gjp4-8570p:~ # service jail onestart 13
Starting jails: 13.
root@mowa219-gjp4-8570p:~ # mkjail update -a
Updating all jails...

Updating 13 jail...

src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 13.0-RELEASE-p3.
No updates are available to install.

root@mowa219-gjp4-8570p:~ #
```

(What's above is slightly sanitised to exclude <https://github.com/mkjail/mkjail/issues/17>.)



michael_hackson said:


> … Thread 49079 …





SirDice said:


> … there's only one kernel (the host's). …



I wondered whether a binary for 13 might work with a 14 kernel.


```
root@mowa219-gjp4-8570p:~ # chroot /jails/13
root@mowa219-gjp4-8570p:/ # pkg install nano
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly/Latest/pkg.txz: No address record
Address resolution failed for http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly.
Consider changing PACKAGESITE.
root@mowa219-gjp4-8570p:/ # pkg install nano
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
⋯
[3/3] Extracting nano-5.8: 100%
root@mowa219-gjp4-8570p:/ # which nano
/usr/local/bin/nano
root@mowa219-gjp4-8570p:/ # file /usr/local/bin/nano
/usr/local/bin/nano: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300139), FreeBSD-style, stripped
root@mowa219-gjp4-8570p:/ # freebsd-version
13.0-RELEASE-p3
root@mowa219-gjp4-8570p:/ # uname -KrU
14.0-CURRENT 1400025 1300139
root@mowa219-gjp4-8570p:/ # ls -hl /etc/resolv.conf
-rw-r--r--  1 root  wheel    24B Jul 18 23:28 /etc/resolv.conf
root@mowa219-gjp4-8570p:/ # cat /etc/resolv.conf
nameserver 192.168.1.1

root@mowa219-gjp4-8570p:/ # ifconfig
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ⋯
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        ⋯
wlan0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ⋯
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1480
        ⋯
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ⋯
root@mowa219-gjp4-8570p:/ # route -6 show freshports.org
   route to: 2600:1f18:461f:1a17:372a:8b3e:c58e:af1b
destination: default
       mask: default
    gateway: ⋯.tunnel.⋯.⋯.ipv6.he.net
        fib: 0
  interface: gif0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1480         1         0
root@mowa219-gjp4-8570p:/ # ping -6 freshports.org
PING6(56=40+8+8 bytes) 2001:⋯:⋯:⋯::2 --> 2600:1f18:461f:1a17:372a:8b3e:c58e:af1b
16 bytes from 2600:1f18:461f:1a17:372a:8b3e:c58e:af1b, icmp_seq=0 hlim=243 time=80.215 ms
16 bytes from 2600:1f18:461f:1a17:372a:8b3e:c58e:af1b, icmp_seq=1 hlim=243 time=79.663 ms
^C
--- freshports.org ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 79.663/79.939/80.215/0.276 ms
root@mowa219-gjp4-8570p:/ # exit
exit
root@mowa219-gjp4-8570p:~ #
```

(After the initial address resolution failure l found the /etc/resolv.conf hint under <https://www.freebsd.org/cgi/man.cgi?query=jail&sektion=8&manpath=FreeBSD+13.0-RELEASE#EXAMPLES>.)


----------



## SirDice (Jul 19, 2021)

grahamperrin said:


> I wondered whether a binary for 13 might work with a 14 kernel.


It should. That's why COMPAT_FREEBSD11, COMPAT_FREEBSD12, COMPAT_FREEBSD13, etc are in the GENERIC kernel. And why there's a misc/compat10x, misc/compat11x and misc/compat12x  (you don't need those for running an N-1 jail on version N).


----------

