May 312021
 

Mark Felder and I have been working on a minimalist set of jail scripts for creating and updating jails. All jail management is accomplished via standard vanilla FreeBSD jails. No jail managers are involved. Everything goes through jail.conf.

In this post:

  • FreeBSD 13.0 host, recently update from FreeBSD 12.2
  • FreeBSD 12.s jail on that host, about to be updated to FreeBSD 13
  • These jails were NOT created with mkjail
  • Jails are mounted at /jails
  • Jails are in the zpool named system.

Follow along with me after I’ve just updated this host from FreeBSD 12.2 to FreeBSD 13 as I upgrade one of the jails to FreeBSD 13.

Getting mkjail

I downloaded mkjail locally, it is not available as a FreeBSD port. Perhaps one day.

[dan@knew:~/src] git clone https://github.com/mkjail/mkjail.git
Cloning into 'mkjail'...
remote: Enumerating objects: 111, done.
remote: Counting objects: 100% (111/111), done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 111 (delta 38), reused 88 (delta 27), pack-reused 0
Receiving objects: 100% (111/111), 19.16 KiB | 19.16 MiB/s, done.
Resolving deltas: 100% (38/38), done.
[dan@knew:~/src] $ 

I modified mkjail/src/etc/mkjail.conf:

# mkjail config file

# Set your zpool name
ZPOOL="system"

# Set jail root filesystem path
JAILROOT="/jails"

# Sets you want extracted into new jail
# options include: base, doc, games (deprecated), kernel, lib32, ports, src
SETS="base lib32"

I changed ZPOOL to the zpool which contains my JAILROOT datasets.

I also added lib32 to the SETS which determines which tarballs mkjail will download for a given release of FreeBSD.

Modifying the jail pkg repo

I use my own jail repos. The repos must be explicitly named for each release. It’s just how I do this. You don’t have to. This step is not for you if you use FreeBSD package repos.

[dan@knew:~/src/mkjail] $ sudo joe /jails/*/usr/local/etc/pkg/repos/local.conf

I changed this:

-   url: "pkg+http://fedex.int.unixathome.org/packages/122amd64-default-master-list/"
+   url: "pkg+http://fedex.int.unixathome.org/packages/13amd64-default-master-list/"

in all 18 files.

Updating the jail – first attempt

[dan@knew:~/src/mkjail] $ sudo ./src/bin/mkjail upgrade -v 13.0-RELEASE -j ansible
cannot open 'system/mkjail': dataset does not exist
Missing required sets for 13.0-RELEASE.
Please run 'mkjail getrelease' for the version you want to upgrade to.
[dan@knew:~/src/mkjail] $ 

Oh, OK, first time on this host. Fair enough:

[dan@knew:~/src/mkjail] $ sudo ./src/bin/mkjail upgrade -v 13.0-RELEASE
Fetching release manifest...
MANIFEST                                              1046  B   17 MBps    00s
Fetching release tarballs...
base.txz                                               180 MB   12 MBps    14s
lib32.txz                                               67 MB   12 MBps    05s
src.txz                                                153 MB   12 MBps    12s
base.txz: sha256 verified
lib32.txz: sha256 verified
src.txz: sha256 verified
Extracting src for use in jail upgrades...
[dan@knew:~/src/mkjail] $ 

Let’s look at what’s in this new filesystem:

[dan@knew:~/src/mkjail] $ zfs list -r system/mkjail
NAME                         USED  AVAIL     REFER  MOUNTPOINT
system/mkjail               1.91G  20.0T      219K  /mkjail
system/mkjail/13.0-RELEASE  1.91G  20.0T     1.91G  /mkjail/13.0-RELEASE
[dan@knew:~/src/mkjail] $ 

[dan@knew:~/src/mkjail] $ ls -l /mkjail
total 1
drwxr-xr-x  3 root  wheel  3 May 31 18:56 13.0-RELEASE
[dan@knew:~/src/mkjail] $ ls -l /mkjail/13.0-RELEASE/
total 1
drwxr-xr-x  3 root  wheel  3 May 31 18:56 usr
[dan@knew:~/src/mkjail] $ ls -l /mkjail/13.0-RELEASE/usr/src/
total 1392
-rw-r--r--    1 root  wheel    6109 Apr  9 00:24 COPYRIGHT
-rw-r--r--    1 root  wheel     500 Apr  9 00:24 LOCKS
-rw-r--r--    1 root  wheel    7130 Apr  9 00:24 MAINTAINERS
-rw-r--r--    1 root  wheel   30349 Apr  9 00:24 Makefile
-rw-r--r--    1 root  wheel  122062 Apr  9 00:24 Makefile.inc1
-rw-r--r--    1 root  wheel    3654 Apr  9 00:24 Makefile.libcompat
-rw-r--r--    1 root  wheel    1951 Apr  9 00:24 Makefile.sys.inc
-rw-r--r--    1 root  wheel  575815 Apr  9 00:24 ObsoleteFiles.inc
-rw-r--r--    1 root  wheel    2426 Apr  9 00:24 README
-rw-r--r--    1 root  wheel    2464 Apr  9 00:24 README.md
-rw-r--r--    1 root  wheel   11851 Apr  9 00:24 RELNOTES
-rw-r--r--    1 root  wheel  103604 Apr  9 00:24 UPDATING
drwxr-xr-x   41 root  wheel      43 Apr  9 00:24 bin
drwxr-xr-x    9 root  wheel      11 Apr  9 00:24 cddl
drwxr-xr-x   89 root  wheel      89 Apr  9 00:24 contrib
drwxr-xr-x    5 root  wheel       6 Apr  9 00:24 crypto
drwxr-xr-x    7 root  wheel      12 Apr  9 00:24 etc
drwxr-xr-x    5 root  wheel       9 Apr  9 00:24 gnu
drwxr-xr-x    8 root  wheel     106 Apr  9 00:24 include
drwxr-xr-x    8 root  wheel      11 Apr  9 00:24 kerberos5
drwxr-xr-x  141 root  wheel     143 Apr  9 00:24 lib
drwxr-xr-x   38 root  wheel      42 Apr  9 00:24 libexec
drwxr-xr-x   12 root  wheel      23 Apr  9 00:24 release
drwxr-xr-x    4 root  wheel       6 Apr  9 00:24 rescue
drwxr-xr-x   90 root  wheel      97 Apr  9 00:24 sbin
drwxr-xr-x    8 root  wheel      10 Apr  9 00:24 secure
drwxr-xr-x   27 root  wheel      29 Apr  9 00:24 share
drwxr-xr-x   27 root  wheel      36 Apr  9 00:24 stand
drwxr-xr-x   53 root  wheel      54 Apr  9 00:24 sys
drwxr-xr-x    3 root  wheel       6 Apr  9 00:24 targets
drwxr-xr-x    5 root  wheel      10 Apr  9 00:24 tests
drwxr-xr-x   19 root  wheel      23 Apr  9 00:24 tools
drwxr-xr-x  284 root  wheel     290 Apr  9 00:24 usr.bin
drwxr-xr-x  225 root  wheel     234 Apr  9 00:24 usr.sbin

That’s the source code, used during jail upgrades.

There’s more, over here:

[dan@knew:~/src/mkjail] $ ls -l /var/db/mkjail/releases/amd64/13.0-RELEASE/
total 410511
-rw-r--r--  1 root  wheel       1046 Apr  9 07:14 MANIFEST
-rw-r--r--  1 root  wheel  188782444 Apr  9 07:14 base.txz
-rw-r--r--  1 root  wheel   70848856 Apr  9 07:14 lib32.txz
-rw-r--r--  1 root  wheel  160454124 Apr  9 07:14 src.txz
[dan@knew:~/src/mkjail] $ 

These are used to create jails.

Updating the jail – 2nd attempt

NOTE: Don’t be tempted to interrupt the upgrade. The script will do a snapshot of your jail and then rollback to that snapshot should an error occur. Interrupting the script will avoid that rollback. You probably don’t want that. You can also go in manually and apply that rollback. I checked and found this snapshot:

system/jails/ansible@mkjail-202105311920

Then I ran the command a second time and was successful.

[dan@knew:~/src/mkjail] $ sudo ./src/bin/mkjail upgrade -v 13.0-RELEASE -j ansible
Upgrading ansible jail from - to 13.0-RELEASE...

  D /etc/amd.map
  D /etc/newsyslog.conf.d/amd.conf
  D /etc/rc.d/amd
  D /etc/rc.d/nsswitch
  D /etc/rc.d/timed
[snip]
  A /etc/periodic/daily/222.backup-gmirror
  A /etc/periodic/daily/223.backup-zfs
  A /etc/rc.d/zpool
Warnings:
  Modified regular file remains: /etc/motd
  Non-empty directory remains: /usr/share/openssl/man
  Non-empty directory remains: /usr/share/openssl
  Non-empty directory remains: /usr/share/man
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        pkg: 1.16.3

Number of packages to be removed: 1

The operation will free 29 MiB.
[ansible.int.unixathome.org] [1/1] Deinstalling pkg-1.16.3...
[ansible.int.unixathome.org] [1/1] Deleting files for pkg-1.16.3: 100%
You may need to manually remove /usr/local/etc/pkg.conf if it is no longer needed.
Bootstrapping pkg from pkg+http://fedex.int.unixathome.org/packages/13amd64-default-master-list/, please wait...
Verifying signature with public key /etc/ssl/slocum.unixathome.org.cert... done
[ansible.int.unixathome.org] Installing pkg-1.16.3...
[ansible.int.unixathome.org] Extracting pkg-1.16.3: 100%
Updating local repository catalogue...
pkg-static: Repository local has a wrong packagesite, need to re-create database
[ansible.int.unixathome.org] Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
[ansible.int.unixathome.org] Fetching packagesite.txz: 100%  258 KiB 263.8kB/s    00:01    
Processing entries: 100%
local repository update completed. 989 packages processed.
All repositories are up to date.
Updating local repository catalogue...
local repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
Checking for upgrades (111 candidates): 100%
Processing candidates (111 candidates): 100%
The following 110 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        mysql57-client: 5.7.33 -> 5.7.34
        p5-HTML-Parser: 3.75 -> 3.76
        p5-MIME-Types: 2.18 -> 2.20

Installed packages to be REINSTALLED:
        SamDruckerClientShell-0.2.6
        apr-1.7.0.1.6.1_1 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:13:x86:64')
        bash-5.1.8 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:13:x86:64')
        bind-tools-9.16.16 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:13:x86:64')
[snip]
        utf8proc-2.6.1 (ABI changed: 'freebsd:12:x86:64' -> 'freebsd:13:x86:64')

Number of packages to be upgraded: 3
Number of packages to be reinstalled: 107

The process will require 2 MiB more space.
154 MiB to be downloaded.
[ansible.int.unixathome.org] [1/110] Fetching pkg-1.16.3.txz: 100%    8 MiB   8.7MB/s    00:01    

[snip]   

[ansible.int.unixathome.org] [110/110] Fetching SamDruckerClientShell-0.2.6.txz: 100%    2 KiB   2.3kB/s    00:01    
Checking integrity... done (0 conflicting)
[ansible.int.unixathome.org] [1/110] Reinstalling indexinfo-0.3.1...

[snip]

[ansible.int.unixathome.org] [110/110] Reinstalling SamDruckerClientShell-0.2.6...
[ansible.int.unixathome.org] [110/110] Extracting SamDruckerClientShell-0.2.6: 100%

You may need to manually remove /usr/local/etc/sudoers if it is no longer needed.
You may need to manually remove /usr/local/etc/postfix/main.cf if it is no longer needed.
=====
Message from postfix-3.6.0,1:

--
If you are upgrading from prior postfix version, please see the README
files for recommended changes to your configuration and additional
http://www.postfix.org/COMPATIBILITY_README.html

Incompatible change with postfix 3.5.x
==========================================
Internal protocols have changed. You need to "postfix stop" before
updating, or before backing out to an earlier release, otherwise
long-running daemons (pickup, qmgr, verify, tlsproxy, postscreen)
may fail to communicate with the rest of Postfix (warnings, timeouts).

The purpose of this change is to produce better error messages, for
example, when someone configures the discard daemon as a bounce
service in master.cf, or vice versa.

This change will break third-party code that implements a
Postfix-internal protocol such as qpsmtpd. Programs that depend on
Postfix internal details are not supported.
You may need to manually remove /usr/local/etc/nrpe.cfg if it is no longer needed.
You may need to manually remove /usr/local/etc/logcheck/logcheck.conf if it is no longer needed.
You may need to manually remove /usr/local/etc/logcheck/logcheck.logfiles if it is no longer needed.
You may need to manually remove /usr/local/etc/samdrucker/samdrucker.conf if it is no longer needed.
make[1] warning: /usr/src/: Read-only file system.
make[2] warning: /usr/src/: Read-only file system.
>>> Removing old files (only deletes safe to delete libs)
make[2] warning: /usr/src/: Read-only file system.
make[3] warning: /usr/src/: Read-only file system.
remove /boot/boot1.efifat? remove /boot/gptboot.efifat? remove /boot/lua/logo-beastie.lua? remove /boot/lua/logo-beastiebw.lua? remove /boot/lua/logo-fbsdbw.lua? remove /boot/lua/logo-orb.lua? remove /boot/lua/logo-orbbw.lua? remove /etc/gnats/freefall? remove /etc/host.conf? remove /etc/mtree/BIND.chroot.dist? remove /etc/pam.d/kde? 

[snip]

 /usr/share/pc-sysinstall/doc/help-start-autoinstall? remove /var/named/etc/namedb/master/empty.db? remove /var/named/etc/namedb/master/localhost-forward.db? remove /var/named/etc/namedb/master/localhost-reverse.db? remove /var/named/etc/namedb/named.root? >>> Old files removed
>>> Removing old directories
make[2] warning: /usr/src/: Read-only file system.
make[3] warning: /usr/src/: Read-only file system.
/var/run/named
/var/named/var/stats
/var/named/var/run/named
/var/named/var/run

[snip]

/usr/include/dev/bktr
/usr/games
/etc/gnats
>>> Old directories removed
To remove old libraries run 'make delete-old-libs'.
make[1] warning: /usr/src/: Read-only file system.
make[2] warning: /usr/src/: Read-only file system.
>>> Removing old libraries
Please be sure no application still uses those libraries, else you
can not start such an application. Consult UPDATING for more
information regarding how to cope with the removal/revision bump
of a specific library.

[snip]

Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 3 patches.. done.
Applying patches... done.
Fetching 10 files... ....10 done.
The following files will be updated as part of updating to
13.0-RELEASE-p1:
/bin/freebsd-version
/usr/bin/bc
/usr/bin/dc
/usr/lib/libradius.a
/usr/lib/libradius.so.4
/usr/lib/libradius_p.a
/usr/lib32/libradius.a
/usr/lib32/libradius.so.4
/usr/lib32/libradius_p.a
/usr/src/contrib/bc/src/program.c
/usr/src/lib/libradius/radlib.c
/usr/src/sys/amd64/amd64/support.S
/usr/src/sys/amd64/linux/linux_support.s
/usr/src/sys/amd64/linux32/linux32_support.s
/usr/src/sys/conf/newvers.sh
/usr/src/sys/dev/mpt/mpt_cam.c
/usr/src/sys/dev/pms/freebsd/driver/ini/src/agdef.h
/usr/src/sys/dev/pms/freebsd/driver/ini/src/agtiapi.c
/usr/src/sys/netinet/ip_divert.c
Installing updates...Scanning /jails/ansible/usr/share/certs/blacklisted for certificates...
Scanning /jails/ansible/usr/share/certs/trusted for certificates...
Scanning /jails/ansible/usr/local/share/certs for certificates...
 done.

This is a heavily reduced output and you can view the full output in a gist.

Things to note:

  • The jail was updated without intervention
  • freebsd-update was run to apply all available OS patches
  • total time: 2m15.629s

After the upgrade

After the upgrade, I restart that jail:

[dan@knew:~/src/mkjail] $ sudo service jail restart ansible
Stopping jails: ansible.
Starting jails: ansible.

Then I proceed with the next jail. upgrade does have a -a option to “Upgrade all running jails” but I am choosing to run this one-at-a-time. I have some database servers on here which need to be coordinated with other jails.

Errors I’ve seen

mkjail strives to upgrade the jail with minimal input. This can lead to situations where the default answers give rise to error. For example:

Checking integrity...Assertion failed: (strcmp(uid, p->uid) != 0), function pkg_conflicts_check_local_path, file pkg_jobs_conflicts.c, line 386.

This arose when the jail had both python37 and python38 installed. My solution:

sudo pkg delete python37
sudo pkg autoremove

You can also go into src/share/mkjail/upgrade.sh and modify this line:

-    jexec ${JAILNAME} /usr/local/sbin/pkg-static upgrade -fy || _cleanup
+    jexec ${JAILNAME} /usr/local/sbin/pkg-static upgrade -f  || _cleanup

This changes the pkg upgrade process so that you get manually prompted to resolve issues. That has also worked for me.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive