Feb 282021
Tonight I got blocked by iocage and handling a ZFS filesystem from within an iocage jail.
These are the steps I followed to convert that jail from iocage to a vanilla jail.
The variable
To make this solution easier to use, at least for me, on future jail migrations, I have these variables:
$ export MYJAIL=devgit-nginx01 # the name of your jail $ export IOCAGE_ZPOOL=system/iocage/jails # the zfs filesystem where your iocage jails reside $ export VANILLA_ZPOOL=system/jails # the zfs filesystem where the vanilla jails will reside
For example, my devgit-nginx01 jail is at:
[dan@knew:~] $ zfs list system/iocage/jails/mysql56 NAME USED AVAIL REFER MOUNTPOINT system/iocage/jails/mysql56 11.5G 20.6T 210K /iocage/jails/mysql56
In my case, I have to create the new destination filesystem first.
$ sudo zfs create -o compression=on -o atime=off -o mountpoint=/jails $VANILLA_ZPOOL
Copy the jail
First, I turn off the jail, just to be clean.
EDIT: On 2021-03-20 I amended this post to use a variable MYJAIL. This will make it easier for me to copy & paste the commands as I go along.
$ sudo iocage stop $MYJAIL * Stopping devgit-nginx01 + Executing prestop OK + Stopping services OK + Removing devfs_ruleset: 1021 OK + Removing jail process OK + Executing poststop OK
Copying the jail is not required, but it is cleaner. It leaves you a fallback position.
I snapshot the jail and send it away:
$ sudo zfs snapshot $IOCAGE_ZPOOL/$MYJAIL/root@vanilla $ sudo zfs send $IOCAGE_ZPOOL/$MYJAIL/root@vanilla | sudo zfs recv $VANILLA_ZPOOL/$MYJAIL
This is what we wind up with:
$ zfs list $VANILLA_ZPOOL/$MYJAIL NAME USED AVAIL REFER MOUNTPOINT system/jails/devgit-nginx01 8.09G 12.6T 8.09G /jails/devgit-nginx01 $ cd /jails/$MYJAIL $ ls -l total 202 -r--r--r-- 1 root wheel 6177 Dec 13 12:51 COPYRIGHT drwxr-xr-x 2 root wheel 46 Feb 24 14:25 bin drwxr-xr-x 10 root wheel 63 Feb 24 14:25 boot drwxr-xr-x 7 root wheel 7 Jul 6 2020 cache dr-xr-xr-x 2 root wheel 2 Nov 1 2019 dev drwxr-xr-x 25 root wheel 109 Feb 24 14:25 etc drwxr-xr-x 5 root wheel 62 Feb 24 14:25 lib drwxr-xr-x 3 root wheel 5 Dec 13 12:50 libexec drwxr-xr-x 2 root wheel 2 Nov 1 2019 media drwxr-xr-x 2 root wheel 2 Nov 1 2019 mnt drwxr-xr-x 2 root wheel 2 Nov 1 2019 net dr-xr-xr-x 2 root wheel 2 Nov 1 2019 proc drwxr-xr-x 2 root wheel 150 Feb 24 14:25 rescue drwxr-xr-x 3 root wheel 11 Feb 12 00:44 root drwxr-xr-x 2 root wheel 137 Feb 24 14:25 sbin drwxr-xr-x 2 root wheel 2 Jul 6 2020 signals lrwxr-xr-x 1 root wheel 11 May 7 2020 sys -> usr/src/sys drwxrwxrwt 23 root wheel 119 Feb 28 18:39 tmp drwxr-xr-x 15 root wheel 15 Jun 21 2020 usr drwxr-xr-x 28 nagios wheel 28 Feb 28 18:39 var
Disable the iocage jail
$ sudo iocage set boot=off $MYJAIL boot: 1 -> 0
Create an entry for jail.conf
Add a new entry to /etc/jail.conf, based on an existing entry, stagegit-nginx01:
devgit-nginx01 {
host.hostname = "devgit-nginx01.int.unixathome.org";
ip4.addr = "ix2|10.0.0.77/32";
persist;
# You don't need the rest of this you're doing zfs management
# or mounting stuff in your jail
allow.mount=true;
allow.mount.zfs=true;
enforce_statfs=1;
devfs_ruleset=5;
mount.fstab="/etc/fstab.$name";
# because want to mount zfs, we do that before the rc start up
# not sure if we MUST do it in that order.
exec.start="zfs mount -a";
exec.start+="/bin/sh /etc/rc";
exec.created+="zfs set jailed=on nvd/freshports/devgit-nginx01/var/db/freshports/cache";
exec.created+="zfs jail $name nvd/freshports/devgit-nginx01/var/db/freshports/cache";
}
The fstab file
You possibly won’t need an fstab file, but I did.
$ cd /etc/ $ sudo cp -i fstab.stagegit-nginx01 fstab.$MYJAIL
I updated that file to look like this, mostly mount points etc.
$ cat /etc/fstab.$MYJAIL # The ingress jail generates and maintains some static HTML for use by the nginx jail # /iocage/jails/devgit-ingress01/root/var/db/freshports/cache/html /jails/devgit-nginx01/var/db/freshports/cache/html nullfs ro,nosuid,noexec 0 0
You probably don’t need that.
Start the jail
$ sudo service jail start $MYJAIL Starting jails: devgit-nginx01.
ssh into the jail
Let’s go see
$ zfs list NAME USED AVAIL REFER MOUNTPOINT nvd 185G 29.6G 23K /nvd nvd/freshports 185G 29.6G 23K none nvd/freshports/devgit-nginx01 336K 29.6G 24K none nvd/freshports/devgit-nginx01/var 312K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db 288K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports 264K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache 240K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/categories 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/daily 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/general 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/news 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/packages 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/pages 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/ports 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/spooling 24K 29.6G 24K none $
PROFIT!
Changes for backups
This change of location requires a change to the snapshot backup script. Fortunately, it was just a rearrangement and an addition of a for loop.
Cleanup
You might want to go back and delete the snapshots.
sudo zfs destroy $IOCAGE_ZPOOL/$MYJAIL/root@vanilla sudo zfs destroy $VANILLA_ZPOOL/$MYJAIL@vanilla
You can do this recursively later. Notice that I am running this command:
- as non-root
- dry-run mode: n
- recursively: r
- verbose: v
If all looks well, I will prefix the command with sudo, and remove the -n flag.
[dan@slocum:~] $ zfs destroy -nrv system@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/pg01/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/librenms/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/nsnotify/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/git-dev/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/fileserver/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/dns-hidden-master/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/git.langille.org/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/samdrucker/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/talos/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/cliff2/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/mydev/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/webserver/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/certs/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/unifi01/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/bacula/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/bacula-sd-03/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/certs-rsync/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/serpico/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/mysql01/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/besser/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/svn/root@vanilla would destroy system/jails/talos@vanilla would destroy system/jails/fileserver@vanilla would destroy system/jails/librenms@vanilla would destroy system/jails/unifi01@vanilla would destroy system/jails/mydev@vanilla would destroy system/jails/nsnotify@vanilla would destroy system/jails/certs@vanilla would destroy system/jails/bacula-sd-03@vanilla would destroy system/jails/svn@vanilla would destroy system/jails/pg01@vanilla would destroy system/jails/mysql01@vanilla would destroy system/jails/serpico@vanilla would destroy system/jails/webserver@vanilla would destroy system/jails/cliff2@vanilla would destroy system/jails/bacula@vanilla would destroy system/jails/dns-hidden-master@vanilla would destroy system/jails/certs-rsync@vanilla would destroy system/jails/samdrucker@vanilla would destroy system/jails/git.langille.org@vanilla would destroy system/jails/besser@vanilla would reclaim 133M
Not shown is my renaming of the system/iocage filesystem to system/TRASH/iocage.DELETE.ME.after.2021.06.01. I did that rename because:
- It moves away /iocage which makes sure nothing is using it.
- It renames it so I know when I plan to delete it.
Deleting iocage
Once all the jails were converted, I removed iocage.
[dan@slocum:~] $ sudo pkg delete py37-iocage Checking integrity... done (0 conflicting) Deinstallation has been requested for the following 1 packages (of 0 packages in the universe): Installed packages to be REMOVED: py37-iocage: 1.2_6 Number of packages to be removed: 1 The operation will free 1 MiB. Proceed with deinstalling packages? [y/N]: y [1/1] Deinstalling py37-iocage-1.2_6... [1/1] Deleting files for py37-iocage-1.2_6: 100% [dan@slocum:~] $ sudo pkg autoremove Checking integrity... done (0 conflicting) Deinstallation has been requested for the following 25 packages: Installed packages to be REMOVED: py37-GitPython: 3.1.11 py37-certifi: 2020.12.5 py37-cffi: 1.14.5 py37-chardet: 3.0.4_3,1 py37-click: 7.1.2 py37-coloredlogs: 15.0 py37-cryptography: 3.3.2 py37-ddt: 1.4.1 py37-dnspython: 1.16.0 py37-gitdb2: 2.0.6 py37-humanfriendly: 9.1 py37-idna: 2.10 py37-libzfs: 1.0.2020090900 py37-netifaces: 0.10.9 py37-openssl: 20.0.1 py37-pycparser: 2.20 py37-pysocks: 1.7.1 py37-pytest-runner: 2.11.1 py37-requests: 2.22.0_2 py37-six: 1.15.0 py37-smmap2: 2.0.5 py37-texttable: 1.6.3 py37-tqdm: 4.58.0 py37-urllib3: 1.25.11,1 rcs57: 5.7 Number of packages to be removed: 25 The operation will free 16 MiB. Proceed with deinstalling packages? [y/N]: y [1/25] Deinstalling py37-requests-2.22.0_2... [1/25] Deleting files for py37-requests-2.22.0_2: 100% [2/25] Deinstalling py37-urllib3-1.25.11,1... [2/25] Deleting files for py37-urllib3-1.25.11,1: 100% [3/25] Deinstalling py37-openssl-20.0.1... [3/25] Deleting files for py37-openssl-20.0.1: 100% [4/25] Deinstalling py37-GitPython-3.1.11... [4/25] Deleting files for py37-GitPython-3.1.11: 100% [5/25] Deinstalling py37-cryptography-3.3.2... [5/25] Deleting files for py37-cryptography-3.3.2: 100% [6/25] Deinstalling py37-cffi-1.14.5... [6/25] Deleting files for py37-cffi-1.14.5: 100% [7/25] Deinstalling py37-coloredlogs-15.0... [7/25] Deleting files for py37-coloredlogs-15.0: 100% [8/25] Deinstalling py37-gitdb2-2.0.6... [8/25] Deleting files for py37-gitdb2-2.0.6: 100% [9/25] Deinstalling py37-certifi-2020.12.5... [9/25] Deleting files for py37-certifi-2020.12.5: 100% [10/25] Deinstalling py37-chardet-3.0.4_3,1... [10/25] Deleting files for py37-chardet-3.0.4_3,1: 100% [11/25] Deinstalling py37-click-7.1.2... [11/25] Deleting files for py37-click-7.1.2: 100% [12/25] Deinstalling py37-ddt-1.4.1... [12/25] Deleting files for py37-ddt-1.4.1: 100% [13/25] Deinstalling py37-dnspython-1.16.0... [13/25] Deleting files for py37-dnspython-1.16.0: 100% [14/25] Deinstalling py37-humanfriendly-9.1... [14/25] Deleting files for py37-humanfriendly-9.1: 100% [15/25] Deinstalling py37-idna-2.10... [15/25] Deleting files for py37-idna-2.10: 100% [16/25] Deinstalling py37-libzfs-1.0.2020090900... [16/25] Deleting files for py37-libzfs-1.0.2020090900: 100% [17/25] Deinstalling py37-netifaces-0.10.9... [17/25] Deleting files for py37-netifaces-0.10.9: 100% [18/25] Deinstalling py37-pycparser-2.20... [18/25] Deleting files for py37-pycparser-2.20: 100% [19/25] Deinstalling py37-pysocks-1.7.1... [19/25] Deleting files for py37-pysocks-1.7.1: 100% [20/25] Deinstalling py37-pytest-runner-2.11.1... [20/25] Deleting files for py37-pytest-runner-2.11.1: 100% [21/25] Deinstalling py37-tqdm-4.58.0... [21/25] Deleting files for py37-tqdm-4.58.0: 100% [22/25] Deinstalling py37-texttable-1.6.3... [22/25] Deleting files for py37-texttable-1.6.3: 100% [23/25] Deinstalling py37-six-1.15.0... [23/25] Deleting files for py37-six-1.15.0: 100% [24/25] Deinstalling py37-smmap2-2.0.5... [24/25] Deleting files for py37-smmap2-2.0.5: 100% [25/25] Deinstalling rcs57-5.7... [25/25] Deleting files for rcs57-5.7: 100% [dan@slocum:~] $
