Feb 282021
Tonight I got blocked by iocage and handling a ZFS filesystem from within an iocage jail.
These are the steps I followed to convert that jail from iocage to a vanilla jail.
The variable
To make this solution easier to use, at least for me, on future jail migrations, I have these variables:
$ export MYJAIL=devgit-nginx01 # the name of your jail $ export IOCAGE_ZPOOL=system/iocage/jails # the zfs filesystem where your iocage jails reside $ export VANILLA_ZPOOL=system/jails # the zfs filesystem where the vanilla jails will reside
For example, my devgit-nginx01 jail is at:
[dan@knew:~] $ zfs list system/iocage/jails/mysql56 NAME USED AVAIL REFER MOUNTPOINT system/iocage/jails/mysql56 11.5G 20.6T 210K /iocage/jails/mysql56
In my case, I have to create the new destination filesystem first.
$ sudo zfs create -o compression=on -o atime=off -o mountpoint=/jails $VANILLA_ZPOOL
Copy the jail
First, I turn off the jail, just to be clean.
EDIT: On 2021-03-20 I amended this post to use a variable MYJAIL. This will make it easier for me to copy & paste the commands as I go along.
$ sudo iocage stop $MYJAIL * Stopping devgit-nginx01 + Executing prestop OK + Stopping services OK + Removing devfs_ruleset: 1021 OK + Removing jail process OK + Executing poststop OK
Copying the jail is not required, but it is cleaner. It leaves you a fallback position.
I snapshot the jail and send it away:
$ sudo zfs snapshot $IOCAGE_ZPOOL/$MYJAIL/root@vanilla $ sudo zfs send $IOCAGE_ZPOOL/$MYJAIL/root@vanilla | sudo zfs recv $VANILLA_ZPOOL/$MYJAIL
This is what we wind up with:
$ zfs list $VANILLA_ZPOOL/$MYJAIL NAME USED AVAIL REFER MOUNTPOINT system/jails/devgit-nginx01 8.09G 12.6T 8.09G /jails/devgit-nginx01 $ cd /jails/$MYJAIL $ ls -l total 202 -r--r--r-- 1 root wheel 6177 Dec 13 12:51 COPYRIGHT drwxr-xr-x 2 root wheel 46 Feb 24 14:25 bin drwxr-xr-x 10 root wheel 63 Feb 24 14:25 boot drwxr-xr-x 7 root wheel 7 Jul 6 2020 cache dr-xr-xr-x 2 root wheel 2 Nov 1 2019 dev drwxr-xr-x 25 root wheel 109 Feb 24 14:25 etc drwxr-xr-x 5 root wheel 62 Feb 24 14:25 lib drwxr-xr-x 3 root wheel 5 Dec 13 12:50 libexec drwxr-xr-x 2 root wheel 2 Nov 1 2019 media drwxr-xr-x 2 root wheel 2 Nov 1 2019 mnt drwxr-xr-x 2 root wheel 2 Nov 1 2019 net dr-xr-xr-x 2 root wheel 2 Nov 1 2019 proc drwxr-xr-x 2 root wheel 150 Feb 24 14:25 rescue drwxr-xr-x 3 root wheel 11 Feb 12 00:44 root drwxr-xr-x 2 root wheel 137 Feb 24 14:25 sbin drwxr-xr-x 2 root wheel 2 Jul 6 2020 signals lrwxr-xr-x 1 root wheel 11 May 7 2020 sys -> usr/src/sys drwxrwxrwt 23 root wheel 119 Feb 28 18:39 tmp drwxr-xr-x 15 root wheel 15 Jun 21 2020 usr drwxr-xr-x 28 nagios wheel 28 Feb 28 18:39 var
Disable the iocage jail
$ sudo iocage set boot=off $MYJAIL boot: 1 -> 0
Create an entry for jail.conf
Add a new entry to /etc/jail.conf, based on an existing entry, stagegit-nginx01:
devgit-nginx01 {
host.hostname = "devgit-nginx01.int.unixathome.org";
ip4.addr = "ix2|10.0.0.77/32";
persist;
# You don't need the rest of this you're doing zfs management
# or mounting stuff in your jail
allow.mount=true;
allow.mount.zfs=true;
enforce_statfs=1;
devfs_ruleset=5;
mount.fstab="/etc/fstab.$name";
# because want to mount zfs, we do that before the rc start up
# not sure if we MUST do it in that order.
exec.start="zfs mount -a";
exec.start+="/bin/sh /etc/rc";
exec.created+="zfs set jailed=on nvd/freshports/devgit-nginx01/var/db/freshports/cache";
exec.created+="zfs jail $name nvd/freshports/devgit-nginx01/var/db/freshports/cache";
}
The fstab file
You possibly won’t need an fstab file, but I did.
$ cd /etc/ $ sudo cp -i fstab.stagegit-nginx01 fstab.$MYJAIL
I updated that file to look like this, mostly mount points etc.
$ cat /etc/fstab.$MYJAIL # The ingress jail generates and maintains some static HTML for use by the nginx jail # /iocage/jails/devgit-ingress01/root/var/db/freshports/cache/html /jails/devgit-nginx01/var/db/freshports/cache/html nullfs ro,nosuid,noexec 0 0
You probably don’t need that.
Start the jail
$ sudo service jail start $MYJAIL Starting jails: devgit-nginx01.
ssh into the jail
Let’s go see
$ zfs list NAME USED AVAIL REFER MOUNTPOINT nvd 185G 29.6G 23K /nvd nvd/freshports 185G 29.6G 23K none nvd/freshports/devgit-nginx01 336K 29.6G 24K none nvd/freshports/devgit-nginx01/var 312K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db 288K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports 264K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache 240K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/categories 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/daily 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/general 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/news 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/packages 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/pages 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/ports 24K 29.6G 24K none nvd/freshports/devgit-nginx01/var/db/freshports/cache/spooling 24K 29.6G 24K none $
PROFIT!
Changes for backups
This change of location requires a change to the snapshot backup script. Fortunately, it was just a rearrangement and an addition of a for loop.
Cleanup
You might want to go back and delete the snapshots.
sudo zfs destroy $IOCAGE_ZPOOL/$MYJAIL/root@vanilla sudo zfs destroy $VANILLA_ZPOOL/$MYJAIL@vanilla
You can do this recursively later. Notice that I am running this command:
- as non-root
- dry-run mode: n
- recursively: r
- verbose: v
If all looks well, I will prefix the command with sudo, and remove the -n flag.
[dan@slocum:~] $ zfs destroy -nrv system@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/pg01/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/librenms/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/nsnotify/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/git-dev/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/fileserver/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/dns-hidden-master/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/git.langille.org/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/samdrucker/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/talos/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/cliff2/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/mydev/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/webserver/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/certs/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/unifi01/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/bacula/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/bacula-sd-03/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/certs-rsync/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/serpico/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/mysql01/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/besser/root@vanilla would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/svn/root@vanilla would destroy system/jails/talos@vanilla would destroy system/jails/fileserver@vanilla would destroy system/jails/librenms@vanilla would destroy system/jails/unifi01@vanilla would destroy system/jails/mydev@vanilla would destroy system/jails/nsnotify@vanilla would destroy system/jails/certs@vanilla would destroy system/jails/bacula-sd-03@vanilla would destroy system/jails/svn@vanilla would destroy system/jails/pg01@vanilla would destroy system/jails/mysql01@vanilla would destroy system/jails/serpico@vanilla would destroy system/jails/webserver@vanilla would destroy system/jails/cliff2@vanilla would destroy system/jails/bacula@vanilla would destroy system/jails/dns-hidden-master@vanilla would destroy system/jails/certs-rsync@vanilla would destroy system/jails/samdrucker@vanilla would destroy system/jails/git.langille.org@vanilla would destroy system/jails/besser@vanilla would reclaim 133M
Not shown is my renaming of the system/iocage filesystem to system/TRASH/iocage.DELETE.ME.after.2021.06.01. I did that rename because:
- It moves away /iocage which makes sure nothing is using it.
- It renames it so I know when I plan to delete it.
Deleting iocage
Once all the jails were converted, I removed iocage.
[dan@slocum:~] $ sudo pkg delete py37-iocage Checking integrity... done (0 conflicting) Deinstallation has been requested for the following 1 packages (of 0 packages in the universe): Installed packages to be REMOVED: py37-iocage: 1.2_6 Number of packages to be removed: 1 The operation will free 1 MiB. Proceed with deinstalling packages? [y/N]: y [1/1] Deinstalling py37-iocage-1.2_6... [1/1] Deleting files for py37-iocage-1.2_6: 100% [dan@slocum:~] $ sudo pkg autoremove Checking integrity... done (0 conflicting) Deinstallation has been requested for the following 25 packages: Installed packages to be REMOVED: py37-GitPython: 3.1.11 py37-certifi: 2020.12.5 py37-cffi: 1.14.5 py37-chardet: 3.0.4_3,1 py37-click: 7.1.2 py37-coloredlogs: 15.0 py37-cryptography: 3.3.2 py37-ddt: 1.4.1 py37-dnspython: 1.16.0 py37-gitdb2: 2.0.6 py37-humanfriendly: 9.1 py37-idna: 2.10 py37-libzfs: 1.0.2020090900 py37-netifaces: 0.10.9 py37-openssl: 20.0.1 py37-pycparser: 2.20 py37-pysocks: 1.7.1 py37-pytest-runner: 2.11.1 py37-requests: 2.22.0_2 py37-six: 1.15.0 py37-smmap2: 2.0.5 py37-texttable: 1.6.3 py37-tqdm: 4.58.0 py37-urllib3: 1.25.11,1 rcs57: 5.7 Number of packages to be removed: 25 The operation will free 16 MiB. Proceed with deinstalling packages? [y/N]: y [1/25] Deinstalling py37-requests-2.22.0_2... [1/25] Deleting files for py37-requests-2.22.0_2: 100% [2/25] Deinstalling py37-urllib3-1.25.11,1... [2/25] Deleting files for py37-urllib3-1.25.11,1: 100% [3/25] Deinstalling py37-openssl-20.0.1... [3/25] Deleting files for py37-openssl-20.0.1: 100% [4/25] Deinstalling py37-GitPython-3.1.11... [4/25] Deleting files for py37-GitPython-3.1.11: 100% [5/25] Deinstalling py37-cryptography-3.3.2... [5/25] Deleting files for py37-cryptography-3.3.2: 100% [6/25] Deinstalling py37-cffi-1.14.5... [6/25] Deleting files for py37-cffi-1.14.5: 100% [7/25] Deinstalling py37-coloredlogs-15.0... [7/25] Deleting files for py37-coloredlogs-15.0: 100% [8/25] Deinstalling py37-gitdb2-2.0.6... [8/25] Deleting files for py37-gitdb2-2.0.6: 100% [9/25] Deinstalling py37-certifi-2020.12.5... [9/25] Deleting files for py37-certifi-2020.12.5: 100% [10/25] Deinstalling py37-chardet-3.0.4_3,1... [10/25] Deleting files for py37-chardet-3.0.4_3,1: 100% [11/25] Deinstalling py37-click-7.1.2... [11/25] Deleting files for py37-click-7.1.2: 100% [12/25] Deinstalling py37-ddt-1.4.1... [12/25] Deleting files for py37-ddt-1.4.1: 100% [13/25] Deinstalling py37-dnspython-1.16.0... [13/25] Deleting files for py37-dnspython-1.16.0: 100% [14/25] Deinstalling py37-humanfriendly-9.1... [14/25] Deleting files for py37-humanfriendly-9.1: 100% [15/25] Deinstalling py37-idna-2.10... [15/25] Deleting files for py37-idna-2.10: 100% [16/25] Deinstalling py37-libzfs-1.0.2020090900... [16/25] Deleting files for py37-libzfs-1.0.2020090900: 100% [17/25] Deinstalling py37-netifaces-0.10.9... [17/25] Deleting files for py37-netifaces-0.10.9: 100% [18/25] Deinstalling py37-pycparser-2.20... [18/25] Deleting files for py37-pycparser-2.20: 100% [19/25] Deinstalling py37-pysocks-1.7.1... [19/25] Deleting files for py37-pysocks-1.7.1: 100% [20/25] Deinstalling py37-pytest-runner-2.11.1... [20/25] Deleting files for py37-pytest-runner-2.11.1: 100% [21/25] Deinstalling py37-tqdm-4.58.0... [21/25] Deleting files for py37-tqdm-4.58.0: 100% [22/25] Deinstalling py37-texttable-1.6.3... [22/25] Deleting files for py37-texttable-1.6.3: 100% [23/25] Deinstalling py37-six-1.15.0... [23/25] Deleting files for py37-six-1.15.0: 100% [24/25] Deinstalling py37-smmap2-2.0.5... [24/25] Deleting files for py37-smmap2-2.0.5: 100% [25/25] Deinstalling rcs57-5.7... [25/25] Deleting files for rcs57-5.7: 100% [dan@slocum:~] $
Thanks Dan, this was exactly what I was looking for!
Good. I’m glad it helped. What prompted you to change jail styles?
well I was bopping along just fine for a long time, but then iocage got very slow. It would take 4 minutes to list 17 jails. I did some digging and found that some other people were having the same issue, and that the development has slowed to a crawl. I had been wanting to convert but didn’t know how, thought I would have to start from scratch which would have been a hassle, so finding this post was a real coup!
I am new to BSDs (less than 2 years) and I got excited about jails and jail management tools. I read Michael Lucas’s book and iocage seemed like the way to go, even though in linux, my motto is KISS, and to his credit he also has the regular jail management tools too, so the book helps in both ways. Anywhooz, now that I have a little bit of experience with jails and the mystery has been removed, I converted all my existing jails to vanilla and I’m good to go!