Mark Felder and I have been working on a minimalist set of jail scripts for creating and updating jails. All jail management is accomplished via standard vanilla FreeBSD jails. No jail managers are involved. Everything goes through jail.conf.
In this post:
- FreeBSD 13.0 host, recently update from FreeBSD 12.2
- FreeBSD 12.s jail on that host, about to be updated to FreeBSD 13
- These jails were NOT created with mkjail
- Jails are mounted at /jails
- Jails are in the zpool named system.
Follow along with me after I’ve just updated this host from FreeBSD 12.2 to FreeBSD 13 as I upgrade one of the jails to FreeBSD 13.
Getting mkjail
I downloaded mkjail locally, it is not available as a FreeBSD port. Perhaps one day.
[dan@knew:~/src] git clone https://github.com/mkjail/mkjail.git Cloning into 'mkjail'... remote: Enumerating objects: 111, done. remote: Counting objects: 100% (111/111), done. remote: Compressing objects: 100% (72/72), done. remote: Total 111 (delta 38), reused 88 (delta 27), pack-reused 0 Receiving objects: 100% (111/111), 19.16 KiB | 19.16 MiB/s, done. Resolving deltas: 100% (38/38), done. [dan@knew:~/src] $
I modified mkjail/src/etc/mkjail.conf:
# mkjail config file # Set your zpool name ZPOOL="system" # Set jail root filesystem path JAILROOT="/jails" # Sets you want extracted into new jail # options include: base, doc, games (deprecated), kernel, lib32, ports, src SETS="base lib32"
I changed ZPOOL to the zpool which contains my JAILROOT datasets.
I also added lib32 to the SETS which determines which tarballs mkjail will download for a given release of FreeBSD.
Modifying the jail pkg repo
I use my own jail repos. The repos must be explicitly named for each release. It’s just how I do this. You don’t have to. This step is not for you if you use FreeBSD package repos.
[dan@knew:~/src/mkjail] $ sudo joe /jails/*/usr/local/etc/pkg/repos/local.conf
I changed this:
- url: "pkg+http://fedex.int.unixathome.org/packages/122amd64-default-master-list/" + url: "pkg+http://fedex.int.unixathome.org/packages/13amd64-default-master-list/"
in all 18 files.
Updating the jail – first attempt
[dan@knew:~/src/mkjail] $ sudo ./src/bin/mkjail upgrade -v 13.0-RELEASE -j ansible cannot open 'system/mkjail': dataset does not exist Missing required sets for 13.0-RELEASE. Please run 'mkjail getrelease' for the version you want to upgrade to. [dan@knew:~/src/mkjail] $
Oh, OK, first time on this host. Fair enough:
[dan@knew:~/src/mkjail] $ sudo ./src/bin/mkjail upgrade -v 13.0-RELEASE Fetching release manifest... MANIFEST 1046 B 17 MBps 00s Fetching release tarballs... base.txz 180 MB 12 MBps 14s lib32.txz 67 MB 12 MBps 05s src.txz 153 MB 12 MBps 12s base.txz: sha256 verified lib32.txz: sha256 verified src.txz: sha256 verified Extracting src for use in jail upgrades... [dan@knew:~/src/mkjail] $
Let’s look at what’s in this new filesystem:
[dan@knew:~/src/mkjail] $ zfs list -r system/mkjail NAME USED AVAIL REFER MOUNTPOINT system/mkjail 1.91G 20.0T 219K /mkjail system/mkjail/13.0-RELEASE 1.91G 20.0T 1.91G /mkjail/13.0-RELEASE [dan@knew:~/src/mkjail] $ [dan@knew:~/src/mkjail] $ ls -l /mkjail total 1 drwxr-xr-x 3 root wheel 3 May 31 18:56 13.0-RELEASE [dan@knew:~/src/mkjail] $ ls -l /mkjail/13.0-RELEASE/ total 1 drwxr-xr-x 3 root wheel 3 May 31 18:56 usr [dan@knew:~/src/mkjail] $ ls -l /mkjail/13.0-RELEASE/usr/src/ total 1392 -rw-r--r-- 1 root wheel 6109 Apr 9 00:24 COPYRIGHT -rw-r--r-- 1 root wheel 500 Apr 9 00:24 LOCKS -rw-r--r-- 1 root wheel 7130 Apr 9 00:24 MAINTAINERS -rw-r--r-- 1 root wheel 30349 Apr 9 00:24 Makefile -rw-r--r-- 1 root wheel 122062 Apr 9 00:24 Makefile.inc1 -rw-r--r-- 1 root wheel 3654 Apr 9 00:24 Makefile.libcompat -rw-r--r-- 1 root wheel 1951 Apr 9 00:24 Makefile.sys.inc -rw-r--r-- 1 root wheel 575815 Apr 9 00:24 ObsoleteFiles.inc -rw-r--r-- 1 root wheel 2426 Apr 9 00:24 README -rw-r--r-- 1 root wheel 2464 Apr 9 00:24 README.md -rw-r--r-- 1 root wheel 11851 Apr 9 00:24 RELNOTES -rw-r--r-- 1 root wheel 103604 Apr 9 00:24 UPDATING drwxr-xr-x 41 root wheel 43 Apr 9 00:24 bin drwxr-xr-x 9 root wheel 11 Apr 9 00:24 cddl drwxr-xr-x 89 root wheel 89 Apr 9 00:24 contrib drwxr-xr-x 5 root wheel 6 Apr 9 00:24 crypto drwxr-xr-x 7 root wheel 12 Apr 9 00:24 etc drwxr-xr-x 5 root wheel 9 Apr 9 00:24 gnu drwxr-xr-x 8 root wheel 106 Apr 9 00:24 include drwxr-xr-x 8 root wheel 11 Apr 9 00:24 kerberos5 drwxr-xr-x 141 root wheel 143 Apr 9 00:24 lib drwxr-xr-x 38 root wheel 42 Apr 9 00:24 libexec drwxr-xr-x 12 root wheel 23 Apr 9 00:24 release drwxr-xr-x 4 root wheel 6 Apr 9 00:24 rescue drwxr-xr-x 90 root wheel 97 Apr 9 00:24 sbin drwxr-xr-x 8 root wheel 10 Apr 9 00:24 secure drwxr-xr-x 27 root wheel 29 Apr 9 00:24 share drwxr-xr-x 27 root wheel 36 Apr 9 00:24 stand drwxr-xr-x 53 root wheel 54 Apr 9 00:24 sys drwxr-xr-x 3 root wheel 6 Apr 9 00:24 targets drwxr-xr-x 5 root wheel 10 Apr 9 00:24 tests drwxr-xr-x 19 root wheel 23 Apr 9 00:24 tools drwxr-xr-x 284 root wheel 290 Apr 9 00:24 usr.bin drwxr-xr-x 225 root wheel 234 Apr 9 00:24 usr.sbin
That’s the source code, used during jail upgrades.
There’s more, over here:
[dan@knew:~/src/mkjail] $ ls -l /var/db/mkjail/releases/amd64/13.0-RELEASE/ total 410511 -rw-r--r-- 1 root wheel 1046 Apr 9 07:14 MANIFEST -rw-r--r-- 1 root wheel 188782444 Apr 9 07:14 base.txz -rw-r--r-- 1 root wheel 70848856 Apr 9 07:14 lib32.txz -rw-r--r-- 1 root wheel 160454124 Apr 9 07:14 src.txz [dan@knew:~/src/mkjail] $
These are used to create jails.
Updating the jail – 2nd attempt
NOTE: Don’t be tempted to interrupt the upgrade. The script will do a snapshot of your jail and then rollback to that snapshot should an error occur. Interrupting the script will avoid that rollback. You probably don’t want that. You can also go in manually and apply that rollback. I checked and found this snapshot:
system/jails/ansible@mkjail-202105311920
Then I ran the command a second time and was successful.
[dan@knew:~/src/mkjail] $ sudo ./src/bin/mkjail upgrade -v 13.0-RELEASE -j ansible
Upgrading ansible jail from - to 13.0-RELEASE...
D /etc/amd.map
D /etc/newsyslog.conf.d/amd.conf
D /etc/rc.d/amd
D /etc/rc.d/nsswitch
D /etc/rc.d/timed
[snip]
A /etc/periodic/daily/222.backup-gmirror
A /etc/periodic/daily/223.backup-zfs
A /etc/rc.d/zpool
Warnings:
Modified regular file remains: /etc/motd
Non-empty directory remains: /usr/share/openssl/man
Non-empty directory remains: /usr/share/openssl
Non-empty directory remains: /usr/share/man
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):
Installed packages to be REMOVED:
pkg: 1.16.3
Number of packages to be removed: 1
The operation will free 29 MiB.
[ansible.int.unixathome.org] [1/1] Deinstalling pkg-1.16.3...
[ansible.int.unixathome.org] [1/1] Deleting files for pkg-1.16.3: 100%
You may need to manually remove /usr/local/etc/pkg.conf if it is no longer needed.
Bootstrapping pkg from pkg+http://fedex.int.unixathome.org/packages/13amd64-default-master-list/, please wait...
Verifying signature with public key /etc/ssl/slocum.unixathome.org.cert... done
[ansible.int.unixathome.org] Installing pkg-1.16.3...
[ansible.int.unixathome.org] Extracting pkg-1.16.3: 100%
Updating local repository catalogue...
pkg-static: Repository local has a wrong packagesite, need to re-create database
[ansible.int.unixathome.org] Fetching meta.conf: 100% 163 B 0.2kB/s 00:01
[ansible.int.unixathome.org] Fetching packagesite.txz: 100% 258 KiB 263.8kB/s 00:01
Processing entries: 100%
local repository update completed. 989 packages processed.
All repositories are up to date.
Updating local repository catalogue...
local repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
Checking for upgrades (111 candidates): 100%
Processing candidates (111 candidates): 100%
The following 110 package(s) will be affected (of 0 checked):
Installed packages to be UPGRADED:
mysql57-client: 5.7.33 -> 5.7.34
p5-HTML-Parser: 3.75 -> 3.76
p5-MIME-Types: 2.18 -> 2.20
Installed packages to be REINSTALLED:
SamDruckerClientShell-0.2.6
apr-1.7.0.1.6.1_1 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:13:x86:64')
bash-5.1.8 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:13:x86:64')
bind-tools-9.16.16 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:13:x86:64')
[snip]
utf8proc-2.6.1 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:13:x86:64')
Number of packages to be upgraded: 3
Number of packages to be reinstalled: 107
The process will require 2 MiB more space.
154 MiB to be downloaded.
[ansible.int.unixathome.org] [1/110] Fetching pkg-1.16.3.txz: 100% 8 MiB 8.7MB/s 00:01
[snip]
[ansible.int.unixathome.org] [110/110] Fetching SamDruckerClientShell-0.2.6.txz: 100% 2 KiB 2.3kB/s 00:01
Checking integrity... done (0 conflicting)
[ansible.int.unixathome.org] [1/110] Reinstalling indexinfo-0.3.1...
[snip]
[ansible.int.unixathome.org] [110/110] Reinstalling SamDruckerClientShell-0.2.6...
[ansible.int.unixathome.org] [110/110] Extracting SamDruckerClientShell-0.2.6: 100%
You may need to manually remove /usr/local/etc/sudoers if it is no longer needed.
You may need to manually remove /usr/local/etc/postfix/main.cf if it is no longer needed.
=====
Message from postfix-3.6.0,1:
--
If you are upgrading from prior postfix version, please see the README
files for recommended changes to your configuration and additional
http://www.postfix.org/COMPATIBILITY_README.html
Incompatible change with postfix 3.5.x
==========================================
Internal protocols have changed. You need to "postfix stop" before
updating, or before backing out to an earlier release, otherwise
long-running daemons (pickup, qmgr, verify, tlsproxy, postscreen)
may fail to communicate with the rest of Postfix (warnings, timeouts).
The purpose of this change is to produce better error messages, for
example, when someone configures the discard daemon as a bounce
service in master.cf, or vice versa.
This change will break third-party code that implements a
Postfix-internal protocol such as qpsmtpd. Programs that depend on
Postfix internal details are not supported.
You may need to manually remove /usr/local/etc/nrpe.cfg if it is no longer needed.
You may need to manually remove /usr/local/etc/logcheck/logcheck.conf if it is no longer needed.
You may need to manually remove /usr/local/etc/logcheck/logcheck.logfiles if it is no longer needed.
You may need to manually remove /usr/local/etc/samdrucker/samdrucker.conf if it is no longer needed.
make[1] warning: /usr/src/: Read-only file system.
make[2] warning: /usr/src/: Read-only file system.
>>> Removing old files (only deletes safe to delete libs)
make[2] warning: /usr/src/: Read-only file system.
make[3] warning: /usr/src/: Read-only file system.
remove /boot/boot1.efifat? remove /boot/gptboot.efifat? remove /boot/lua/logo-beastie.lua? remove /boot/lua/logo-beastiebw.lua? remove /boot/lua/logo-fbsdbw.lua? remove /boot/lua/logo-orb.lua? remove /boot/lua/logo-orbbw.lua? remove /etc/gnats/freefall? remove /etc/host.conf? remove /etc/mtree/BIND.chroot.dist? remove /etc/pam.d/kde?
[snip]
/usr/share/pc-sysinstall/doc/help-start-autoinstall? remove /var/named/etc/namedb/master/empty.db? remove /var/named/etc/namedb/master/localhost-forward.db? remove /var/named/etc/namedb/master/localhost-reverse.db? remove /var/named/etc/namedb/named.root? >>> Old files removed
>>> Removing old directories
make[2] warning: /usr/src/: Read-only file system.
make[3] warning: /usr/src/: Read-only file system.
/var/run/named
/var/named/var/stats
/var/named/var/run/named
/var/named/var/run
[snip]
/usr/include/dev/bktr
/usr/games
/etc/gnats
>>> Old directories removed
To remove old libraries run 'make delete-old-libs'.
make[1] warning: /usr/src/: Read-only file system.
make[2] warning: /usr/src/: Read-only file system.
>>> Removing old libraries
Please be sure no application still uses those libraries, else you
can not start such an application. Consult UPDATING for more
information regarding how to cope with the removal/revision bump
of a specific library.
[snip]
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 3 patches.. done.
Applying patches... done.
Fetching 10 files... ....10 done.
The following files will be updated as part of updating to
13.0-RELEASE-p1:
/bin/freebsd-version
/usr/bin/bc
/usr/bin/dc
/usr/lib/libradius.a
/usr/lib/libradius.so.4
/usr/lib/libradius_p.a
/usr/lib32/libradius.a
/usr/lib32/libradius.so.4
/usr/lib32/libradius_p.a
/usr/src/contrib/bc/src/program.c
/usr/src/lib/libradius/radlib.c
/usr/src/sys/amd64/amd64/support.S
/usr/src/sys/amd64/linux/linux_support.s
/usr/src/sys/amd64/linux32/linux32_support.s
/usr/src/sys/conf/newvers.sh
/usr/src/sys/dev/mpt/mpt_cam.c
/usr/src/sys/dev/pms/freebsd/driver/ini/src/agdef.h
/usr/src/sys/dev/pms/freebsd/driver/ini/src/agtiapi.c
/usr/src/sys/netinet/ip_divert.c
Installing updates...Scanning /jails/ansible/usr/share/certs/blacklisted for certificates...
Scanning /jails/ansible/usr/share/certs/trusted for certificates...
Scanning /jails/ansible/usr/local/share/certs for certificates...
done.
This is a heavily reduced output and you can view the full output in a gist.
Things to note:
- The jail was updated without intervention
- freebsd-update was run to apply all available OS patches
- total time: 2m15.629s
After the upgrade
After the upgrade, I restart that jail:
[dan@knew:~/src/mkjail] $ sudo service jail restart ansible Stopping jails: ansible. Starting jails: ansible.
Then I proceed with the next jail. upgrade does have a -a option to “Upgrade all running jails” but I am choosing to run this one-at-a-time. I have some database servers on here which need to be coordinated with other jails.
Errors I’ve seen
mkjail strives to upgrade the jail with minimal input. This can lead to situations where the default answers give rise to error. For example:
Checking integrity...Assertion failed: (strcmp(uid, p->uid) != 0), function pkg_conflicts_check_local_path, file pkg_jobs_conflicts.c, line 386.
This arose when the jail had both python37 and python38 installed. My solution:
sudo pkg delete python37 sudo pkg autoremove
You can also go into src/share/mkjail/upgrade.sh and modify this line:
- jexec ${JAILNAME} /usr/local/sbin/pkg-static upgrade -fy || _cleanup
+ jexec ${JAILNAME} /usr/local/sbin/pkg-static upgrade -f || _cleanup
This changes the pkg upgrade process so that you get manually prompted to resolve issues. That has also worked for me.
