Feb 282021

Tonight I got blocked by iocage and handling a ZFS filesystem from within an iocage jail.

These are the steps I followed to convert that jail from iocage to a vanilla jail.

The variable

To make this solution easier to use, at least for me, on future jail migrations, I have these variables:

$ export MYJAIL=devgit-nginx01      # the name of your jail
$ export IOCAGE_ZPOOL=system/iocage/jails # the zfs filesystem where your iocage jails reside
$ export VANILLA_ZPOOL=system/jails # the zfs filesystem where the vanilla jails will reside

For example, my devgit-nginx01 jail is at:

[dan@knew:~] $ zfs list system/iocage/jails/mysql56
NAME                          USED  AVAIL  REFER  MOUNTPOINT
system/iocage/jails/mysql56  11.5G  20.6T   210K  /iocage/jails/mysql56

In my case, I have to create the new destination filesystem first.

$ sudo zfs create -o compression=on -o atime=off -o mountpoint=/jails $VANILLA_ZPOOL

Copy the jail

First, I turn off the jail, just to be clean.

EDIT: On 2021-03-20 I amended this post to use a variable MYJAIL. This will make it easier for me to copy & paste the commands as I go along.

$ sudo iocage stop $MYJAIL
* Stopping devgit-nginx01
  + Executing prestop OK
  + Stopping services OK
  + Removing devfs_ruleset: 1021 OK
  + Removing jail process OK
  + Executing poststop OK

Copying the jail is not required, but it is cleaner. It leaves you a fallback position.

I snapshot the jail and send it away:

$ sudo zfs snapshot $IOCAGE_ZPOOL/$MYJAIL/root@vanilla
$ sudo zfs send $IOCAGE_ZPOOL/$MYJAIL/root@vanilla | sudo zfs recv $VANILLA_ZPOOL/$MYJAIL

This is what we wind up with:

NAME                          USED  AVAIL  REFER  MOUNTPOINT
system/jails/devgit-nginx01  8.09G  12.6T  8.09G  /jails/devgit-nginx01

$ cd /jails/$MYJAIL
$ ls -l
total 202
-r--r--r--   1 root    wheel  6177 Dec 13 12:51 COPYRIGHT
drwxr-xr-x   2 root    wheel    46 Feb 24 14:25 bin
drwxr-xr-x  10 root    wheel    63 Feb 24 14:25 boot
drwxr-xr-x   7 root    wheel     7 Jul  6  2020 cache
dr-xr-xr-x   2 root    wheel     2 Nov  1  2019 dev
drwxr-xr-x  25 root    wheel   109 Feb 24 14:25 etc
drwxr-xr-x   5 root    wheel    62 Feb 24 14:25 lib
drwxr-xr-x   3 root    wheel     5 Dec 13 12:50 libexec
drwxr-xr-x   2 root    wheel     2 Nov  1  2019 media
drwxr-xr-x   2 root    wheel     2 Nov  1  2019 mnt
drwxr-xr-x   2 root    wheel     2 Nov  1  2019 net
dr-xr-xr-x   2 root    wheel     2 Nov  1  2019 proc
drwxr-xr-x   2 root    wheel   150 Feb 24 14:25 rescue
drwxr-xr-x   3 root    wheel    11 Feb 12 00:44 root
drwxr-xr-x   2 root    wheel   137 Feb 24 14:25 sbin
drwxr-xr-x   2 root    wheel     2 Jul  6  2020 signals
lrwxr-xr-x   1 root    wheel    11 May  7  2020 sys -> usr/src/sys
drwxrwxrwt  23 root    wheel   119 Feb 28 18:39 tmp
drwxr-xr-x  15 root    wheel    15 Jun 21  2020 usr
drwxr-xr-x  28 nagios  wheel    28 Feb 28 18:39 var

Disable the iocage jail

$ sudo iocage set boot=off $MYJAIL
boot: 1 -> 0

Create an entry for jail.conf

Add a new entry to /etc/jail.conf, based on an existing entry, stagegit-nginx01:

devgit-nginx01 {
    host.hostname = "devgit-nginx01.int.unixathome.org";
    ip4.addr = "ix2|";

    # You don't need the rest of this you're doing zfs management
    # or mounting stuff in your jail


    # because want to mount zfs, we do that before the rc start up
    # not sure if we MUST do it in that order.
    exec.start="zfs mount -a";
    exec.start+="/bin/sh /etc/rc";

    exec.created+="zfs set jailed=on nvd/freshports/devgit-nginx01/var/db/freshports/cache";
    exec.created+="zfs jail $name    nvd/freshports/devgit-nginx01/var/db/freshports/cache";

The fstab file

You possibly won’t need an fstab file, but I did.

$ cd /etc/
$ sudo cp -i fstab.stagegit-nginx01 fstab.$MYJAIL

I updated that file to look like this, mostly mount points etc.

$ cat /etc/fstab.$MYJAIL
# The ingress jail generates and maintains some static HTML for use by the nginx jail
/iocage/jails/devgit-ingress01/root/var/db/freshports/cache/html /jails/devgit-nginx01/var/db/freshports/cache/html   nullfs  ro,nosuid,noexec  0   0

You probably don’t need that.

Start the jail

$ sudo service jail start $MYJAIL
Starting jails: devgit-nginx01.

ssh into the jail

Let’s go see

$ zfs list
NAME                                                               USED  AVAIL  REFER  MOUNTPOINT
nvd                                                                185G  29.6G    23K  /nvd
nvd/freshports                                                     185G  29.6G    23K  none
nvd/freshports/devgit-nginx01                                      336K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var                                  312K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db                               288K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports                    264K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache              240K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/categories    24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits       24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/daily         24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/general       24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/news          24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/packages      24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/pages         24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/ports         24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/spooling      24K  29.6G    24K  none


Changes for backups

This change of location requires a change to the snapshot backup script. Fortunately, it was just a rearrangement and an addition of a for loop.


You might want to go back and delete the snapshots.

sudo zfs destroy $IOCAGE_ZPOOL/$MYJAIL/root@vanilla
sudo zfs destroy $VANILLA_ZPOOL/$MYJAIL@vanilla

You can do this recursively later. Notice that I am running this command:

  • as non-root
  • dry-run mode: n
  • recursively: r
  • verbose: v

If all looks well, I will prefix the command with sudo, and remove the -n flag.

[dan@slocum:~] $ zfs destroy -nrv system@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/pg01/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/librenms/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/nsnotify/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/git-dev/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/fileserver/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/dns-hidden-master/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/git.langille.org/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/samdrucker/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/talos/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/cliff2/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/mydev/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/webserver/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/certs/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/unifi01/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/bacula/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/bacula-sd-03/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/certs-rsync/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/serpico/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/mysql01/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/besser/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/svn/root@vanilla
would destroy system/jails/talos@vanilla
would destroy system/jails/fileserver@vanilla
would destroy system/jails/librenms@vanilla
would destroy system/jails/unifi01@vanilla
would destroy system/jails/mydev@vanilla
would destroy system/jails/nsnotify@vanilla
would destroy system/jails/certs@vanilla
would destroy system/jails/bacula-sd-03@vanilla
would destroy system/jails/svn@vanilla
would destroy system/jails/pg01@vanilla
would destroy system/jails/mysql01@vanilla
would destroy system/jails/serpico@vanilla
would destroy system/jails/webserver@vanilla
would destroy system/jails/cliff2@vanilla
would destroy system/jails/bacula@vanilla
would destroy system/jails/dns-hidden-master@vanilla
would destroy system/jails/certs-rsync@vanilla
would destroy system/jails/samdrucker@vanilla
would destroy system/jails/git.langille.org@vanilla
would destroy system/jails/besser@vanilla
would reclaim 133M

Not shown is my renaming of the system/iocage filesystem to system/TRASH/iocage.DELETE.ME.after.2021.06.01. I did that rename because:

  1. It moves away /iocage which makes sure nothing is using it.
  2. It renames it so I know when I plan to delete it.

Deleting iocage

Once all the jails were converted, I removed iocage.

[dan@slocum:~] $ sudo pkg delete py37-iocage
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	py37-iocage: 1.2_6

Number of packages to be removed: 1

The operation will free 1 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling py37-iocage-1.2_6...
[1/1] Deleting files for py37-iocage-1.2_6: 100%
[dan@slocum:~] $ sudo pkg autoremove
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 25 packages:

Installed packages to be REMOVED:
	py37-GitPython: 3.1.11
	py37-certifi: 2020.12.5
	py37-cffi: 1.14.5
	py37-chardet: 3.0.4_3,1
	py37-click: 7.1.2
	py37-coloredlogs: 15.0
	py37-cryptography: 3.3.2
	py37-ddt: 1.4.1
	py37-dnspython: 1.16.0
	py37-gitdb2: 2.0.6
	py37-humanfriendly: 9.1
	py37-idna: 2.10
	py37-libzfs: 1.0.2020090900
	py37-netifaces: 0.10.9
	py37-openssl: 20.0.1
	py37-pycparser: 2.20
	py37-pysocks: 1.7.1
	py37-pytest-runner: 2.11.1
	py37-requests: 2.22.0_2
	py37-six: 1.15.0
	py37-smmap2: 2.0.5
	py37-texttable: 1.6.3
	py37-tqdm: 4.58.0
	py37-urllib3: 1.25.11,1
	rcs57: 5.7

Number of packages to be removed: 25

The operation will free 16 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/25] Deinstalling py37-requests-2.22.0_2...
[1/25] Deleting files for py37-requests-2.22.0_2: 100%
[2/25] Deinstalling py37-urllib3-1.25.11,1...
[2/25] Deleting files for py37-urllib3-1.25.11,1: 100%
[3/25] Deinstalling py37-openssl-20.0.1...
[3/25] Deleting files for py37-openssl-20.0.1: 100%
[4/25] Deinstalling py37-GitPython-3.1.11...
[4/25] Deleting files for py37-GitPython-3.1.11: 100%
[5/25] Deinstalling py37-cryptography-3.3.2...
[5/25] Deleting files for py37-cryptography-3.3.2: 100%
[6/25] Deinstalling py37-cffi-1.14.5...
[6/25] Deleting files for py37-cffi-1.14.5: 100%
[7/25] Deinstalling py37-coloredlogs-15.0...
[7/25] Deleting files for py37-coloredlogs-15.0: 100%
[8/25] Deinstalling py37-gitdb2-2.0.6...
[8/25] Deleting files for py37-gitdb2-2.0.6: 100%
[9/25] Deinstalling py37-certifi-2020.12.5...
[9/25] Deleting files for py37-certifi-2020.12.5: 100%
[10/25] Deinstalling py37-chardet-3.0.4_3,1...
[10/25] Deleting files for py37-chardet-3.0.4_3,1: 100%
[11/25] Deinstalling py37-click-7.1.2...
[11/25] Deleting files for py37-click-7.1.2: 100%
[12/25] Deinstalling py37-ddt-1.4.1...
[12/25] Deleting files for py37-ddt-1.4.1: 100%
[13/25] Deinstalling py37-dnspython-1.16.0...
[13/25] Deleting files for py37-dnspython-1.16.0: 100%
[14/25] Deinstalling py37-humanfriendly-9.1...
[14/25] Deleting files for py37-humanfriendly-9.1: 100%
[15/25] Deinstalling py37-idna-2.10...
[15/25] Deleting files for py37-idna-2.10: 100%
[16/25] Deinstalling py37-libzfs-1.0.2020090900...
[16/25] Deleting files for py37-libzfs-1.0.2020090900: 100%
[17/25] Deinstalling py37-netifaces-0.10.9...
[17/25] Deleting files for py37-netifaces-0.10.9: 100%
[18/25] Deinstalling py37-pycparser-2.20...
[18/25] Deleting files for py37-pycparser-2.20: 100%
[19/25] Deinstalling py37-pysocks-1.7.1...
[19/25] Deleting files for py37-pysocks-1.7.1: 100%
[20/25] Deinstalling py37-pytest-runner-2.11.1...
[20/25] Deleting files for py37-pytest-runner-2.11.1: 100%
[21/25] Deinstalling py37-tqdm-4.58.0...
[21/25] Deleting files for py37-tqdm-4.58.0: 100%
[22/25] Deinstalling py37-texttable-1.6.3...
[22/25] Deleting files for py37-texttable-1.6.3: 100%
[23/25] Deinstalling py37-six-1.15.0...
[23/25] Deleting files for py37-six-1.15.0: 100%
[24/25] Deinstalling py37-smmap2-2.0.5...
[24/25] Deleting files for py37-smmap2-2.0.5: 100%
[25/25] Deinstalling rcs57-5.7...
[25/25] Deleting files for rcs57-5.7: 100%
[dan@slocum:~] $ 
Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive