Converting an iocage jail to a vanilla jail

Tonight I got blocked by iocage and handling a ZFS filesystem from within an iocage jail.

These are the steps I followed to convert that jail from iocage to a vanilla jail.

The variable

To make this solution easier to use, at least for me, on future jail migrations, I have these variables:

$ export MYJAIL=devgit-nginx01      # the name of your jail
$ export IOCAGE_ZPOOL=system/iocage/jails # the zfs filesystem where your iocage jails reside
$ export VANILLA_ZPOOL=system/jails # the zfs filesystem where the vanilla jails will reside

For example, my devgit-nginx01 jail is at:

[dan@knew:~] $ zfs list system/iocage/jails/mysql56
NAME                          USED  AVAIL  REFER  MOUNTPOINT
system/iocage/jails/mysql56  11.5G  20.6T   210K  /iocage/jails/mysql56

In my case, I have to create the new destination filesystem first.

$ sudo zfs create -o compression=on -o atime=off -o mountpoint=/jails $VANILLA_ZPOOL

Copy the jail

First, I turn off the jail, just to be clean.

EDIT: On 2021-03-20 I amended this post to use a variable MYJAIL. This will make it easier for me to copy & paste the commands as I go along.

$ sudo iocage stop $MYJAIL
* Stopping devgit-nginx01
  + Executing prestop OK
  + Stopping services OK
  + Removing devfs_ruleset: 1021 OK
  + Removing jail process OK
  + Executing poststop OK

Copying the jail is not required, but it is cleaner. It leaves you a fallback position.

I snapshot the jail and send it away:

$ sudo zfs snapshot $IOCAGE_ZPOOL/$MYJAIL/root@vanilla
$ sudo zfs send $IOCAGE_ZPOOL/$MYJAIL/root@vanilla | sudo zfs recv $VANILLA_ZPOOL/$MYJAIL

This is what we wind up with:

NAME                          USED  AVAIL  REFER  MOUNTPOINT
system/jails/devgit-nginx01  8.09G  12.6T  8.09G  /jails/devgit-nginx01

$ cd /jails/$MYJAIL
$ ls -l
total 202
-r--r--r--   1 root    wheel  6177 Dec 13 12:51 COPYRIGHT
drwxr-xr-x   2 root    wheel    46 Feb 24 14:25 bin
drwxr-xr-x  10 root    wheel    63 Feb 24 14:25 boot
drwxr-xr-x   7 root    wheel     7 Jul  6  2020 cache
dr-xr-xr-x   2 root    wheel     2 Nov  1  2019 dev
drwxr-xr-x  25 root    wheel   109 Feb 24 14:25 etc
drwxr-xr-x   5 root    wheel    62 Feb 24 14:25 lib
drwxr-xr-x   3 root    wheel     5 Dec 13 12:50 libexec
drwxr-xr-x   2 root    wheel     2 Nov  1  2019 media
drwxr-xr-x   2 root    wheel     2 Nov  1  2019 mnt
drwxr-xr-x   2 root    wheel     2 Nov  1  2019 net
dr-xr-xr-x   2 root    wheel     2 Nov  1  2019 proc
drwxr-xr-x   2 root    wheel   150 Feb 24 14:25 rescue
drwxr-xr-x   3 root    wheel    11 Feb 12 00:44 root
drwxr-xr-x   2 root    wheel   137 Feb 24 14:25 sbin
drwxr-xr-x   2 root    wheel     2 Jul  6  2020 signals
lrwxr-xr-x   1 root    wheel    11 May  7  2020 sys -> usr/src/sys
drwxrwxrwt  23 root    wheel   119 Feb 28 18:39 tmp
drwxr-xr-x  15 root    wheel    15 Jun 21  2020 usr
drwxr-xr-x  28 nagios  wheel    28 Feb 28 18:39 var

Disable the iocage jail

$ sudo iocage set boot=off $MYJAIL
boot: 1 -> 0

Create an entry for jail.conf

Add a new entry to /etc/jail.conf, based on an existing entry, stagegit-nginx01:

devgit-nginx01 {
    host.hostname = "";
    ip4.addr = "ix2|";

    # You don't need the rest of this you're doing zfs management
    # or mounting stuff in your jail


    # because want to mount zfs, we do that before the rc start up
    # not sure if we MUST do it in that order.
    exec.start="zfs mount -a";
    exec.start+="/bin/sh /etc/rc";

    exec.created+="zfs set jailed=on nvd/freshports/devgit-nginx01/var/db/freshports/cache";
    exec.created+="zfs jail $name    nvd/freshports/devgit-nginx01/var/db/freshports/cache";

The fstab file

You possibly won’t need an fstab file, but I did.

$ cd /etc/
$ sudo cp -i fstab.stagegit-nginx01 fstab.$MYJAIL

I updated that file to look like this, mostly mount points etc.

$ cat /etc/fstab.$MYJAIL
# The ingress jail generates and maintains some static HTML for use by the nginx jail
/iocage/jails/devgit-ingress01/root/var/db/freshports/cache/html /jails/devgit-nginx01/var/db/freshports/cache/html   nullfs  ro,nosuid,noexec  0   0

You probably don’t need that.

Start the jail

$ sudo service jail start $MYJAIL
Starting jails: devgit-nginx01.

ssh into the jail

Let’s go see

$ zfs list
NAME                                                               USED  AVAIL  REFER  MOUNTPOINT
nvd                                                                185G  29.6G    23K  /nvd
nvd/freshports                                                     185G  29.6G    23K  none
nvd/freshports/devgit-nginx01                                      336K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var                                  312K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db                               288K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports                    264K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache              240K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/categories    24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits       24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/daily         24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/general       24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/news          24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/packages      24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/pages         24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/ports         24K  29.6G    24K  none
nvd/freshports/devgit-nginx01/var/db/freshports/cache/spooling      24K  29.6G    24K  none


Changes for backups

This change of location requires a change to the snapshot backup script. Fortunately, it was just a rearrangement and an addition of a for loop.


You might want to go back and delete the snapshots.

sudo zfs destroy $IOCAGE_ZPOOL/$MYJAIL/root@vanilla
sudo zfs destroy $VANILLA_ZPOOL/$MYJAIL@vanilla

You can do this recursively later. Notice that I am running this command:

  • as non-root
  • dry-run mode: n
  • recursively: r
  • verbose: v

If all looks well, I will prefix the command with sudo, and remove the -n flag.

[dan@slocum:~] $ zfs destroy -nrv system@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/pg01/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/librenms/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/nsnotify/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/git-dev/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/fileserver/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/dns-hidden-master/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/samdrucker/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/talos/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/cliff2/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/mydev/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/webserver/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/certs/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/unifi01/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/bacula/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/bacula-sd-03/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/certs-rsync/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/serpico/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/mysql01/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/besser/root@vanilla
would destroy system/TRASH/iocage.DELETE.ME.after.2021.06.01/jails/svn/root@vanilla
would destroy system/jails/talos@vanilla
would destroy system/jails/fileserver@vanilla
would destroy system/jails/librenms@vanilla
would destroy system/jails/unifi01@vanilla
would destroy system/jails/mydev@vanilla
would destroy system/jails/nsnotify@vanilla
would destroy system/jails/certs@vanilla
would destroy system/jails/bacula-sd-03@vanilla
would destroy system/jails/svn@vanilla
would destroy system/jails/pg01@vanilla
would destroy system/jails/mysql01@vanilla
would destroy system/jails/serpico@vanilla
would destroy system/jails/webserver@vanilla
would destroy system/jails/cliff2@vanilla
would destroy system/jails/bacula@vanilla
would destroy system/jails/dns-hidden-master@vanilla
would destroy system/jails/certs-rsync@vanilla
would destroy system/jails/samdrucker@vanilla
would destroy system/jails/
would destroy system/jails/besser@vanilla
would reclaim 133M

Not shown is my renaming of the system/iocage filesystem to system/TRASH/iocage.DELETE.ME.after.2021.06.01. I did that rename because:

  1. It moves away /iocage which makes sure nothing is using it.
  2. It renames it so I know when I plan to delete it.

Deleting iocage

Once all the jails were converted, I removed iocage.

[dan@slocum:~] $ sudo pkg delete py37-iocage
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	py37-iocage: 1.2_6

Number of packages to be removed: 1

The operation will free 1 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling py37-iocage-1.2_6...
[1/1] Deleting files for py37-iocage-1.2_6: 100%
[dan@slocum:~] $ sudo pkg autoremove
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 25 packages:

Installed packages to be REMOVED:
	py37-GitPython: 3.1.11
	py37-certifi: 2020.12.5
	py37-cffi: 1.14.5
	py37-chardet: 3.0.4_3,1
	py37-click: 7.1.2
	py37-coloredlogs: 15.0
	py37-cryptography: 3.3.2
	py37-ddt: 1.4.1
	py37-dnspython: 1.16.0
	py37-gitdb2: 2.0.6
	py37-humanfriendly: 9.1
	py37-idna: 2.10
	py37-libzfs: 1.0.2020090900
	py37-netifaces: 0.10.9
	py37-openssl: 20.0.1
	py37-pycparser: 2.20
	py37-pysocks: 1.7.1
	py37-pytest-runner: 2.11.1
	py37-requests: 2.22.0_2
	py37-six: 1.15.0
	py37-smmap2: 2.0.5
	py37-texttable: 1.6.3
	py37-tqdm: 4.58.0
	py37-urllib3: 1.25.11,1
	rcs57: 5.7

Number of packages to be removed: 25

The operation will free 16 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/25] Deinstalling py37-requests-2.22.0_2...
[1/25] Deleting files for py37-requests-2.22.0_2: 100%
[2/25] Deinstalling py37-urllib3-1.25.11,1...
[2/25] Deleting files for py37-urllib3-1.25.11,1: 100%
[3/25] Deinstalling py37-openssl-20.0.1...
[3/25] Deleting files for py37-openssl-20.0.1: 100%
[4/25] Deinstalling py37-GitPython-3.1.11...
[4/25] Deleting files for py37-GitPython-3.1.11: 100%
[5/25] Deinstalling py37-cryptography-3.3.2...
[5/25] Deleting files for py37-cryptography-3.3.2: 100%
[6/25] Deinstalling py37-cffi-1.14.5...
[6/25] Deleting files for py37-cffi-1.14.5: 100%
[7/25] Deinstalling py37-coloredlogs-15.0...
[7/25] Deleting files for py37-coloredlogs-15.0: 100%
[8/25] Deinstalling py37-gitdb2-2.0.6...
[8/25] Deleting files for py37-gitdb2-2.0.6: 100%
[9/25] Deinstalling py37-certifi-2020.12.5...
[9/25] Deleting files for py37-certifi-2020.12.5: 100%
[10/25] Deinstalling py37-chardet-3.0.4_3,1...
[10/25] Deleting files for py37-chardet-3.0.4_3,1: 100%
[11/25] Deinstalling py37-click-7.1.2...
[11/25] Deleting files for py37-click-7.1.2: 100%
[12/25] Deinstalling py37-ddt-1.4.1...
[12/25] Deleting files for py37-ddt-1.4.1: 100%
[13/25] Deinstalling py37-dnspython-1.16.0...
[13/25] Deleting files for py37-dnspython-1.16.0: 100%
[14/25] Deinstalling py37-humanfriendly-9.1...
[14/25] Deleting files for py37-humanfriendly-9.1: 100%
[15/25] Deinstalling py37-idna-2.10...
[15/25] Deleting files for py37-idna-2.10: 100%
[16/25] Deinstalling py37-libzfs-1.0.2020090900...
[16/25] Deleting files for py37-libzfs-1.0.2020090900: 100%
[17/25] Deinstalling py37-netifaces-0.10.9...
[17/25] Deleting files for py37-netifaces-0.10.9: 100%
[18/25] Deinstalling py37-pycparser-2.20...
[18/25] Deleting files for py37-pycparser-2.20: 100%
[19/25] Deinstalling py37-pysocks-1.7.1...
[19/25] Deleting files for py37-pysocks-1.7.1: 100%
[20/25] Deinstalling py37-pytest-runner-2.11.1...
[20/25] Deleting files for py37-pytest-runner-2.11.1: 100%
[21/25] Deinstalling py37-tqdm-4.58.0...
[21/25] Deleting files for py37-tqdm-4.58.0: 100%
[22/25] Deinstalling py37-texttable-1.6.3...
[22/25] Deleting files for py37-texttable-1.6.3: 100%
[23/25] Deinstalling py37-six-1.15.0...
[23/25] Deleting files for py37-six-1.15.0: 100%
[24/25] Deinstalling py37-smmap2-2.0.5...
[24/25] Deleting files for py37-smmap2-2.0.5: 100%
[25/25] Deinstalling rcs57-5.7...
[25/25] Deleting files for rcs57-5.7: 100%
[dan@slocum:~] $ 
Website Pin Facebook Twitter Myspace Friendfeed Technorati Digg Google StumbleUpon Premium Responsive

3 thoughts on “Converting an iocage jail to a vanilla jail”

      1. well I was bopping along just fine for a long time, but then iocage got very slow. It would take 4 minutes to list 17 jails. I did some digging and found that some other people were having the same issue, and that the development has slowed to a crawl. I had been wanting to convert but didn’t know how, thought I would have to start from scratch which would have been a hassle, so finding this post was a real coup!

        I am new to BSDs (less than 2 years) and I got excited about jails and jail management tools. I read Michael Lucas’s book and iocage seemed like the way to go, even though in linux, my motto is KISS, and to his credit he also has the regular jail management tools too, so the book helps in both ways. Anywhooz, now that I have a little bit of experience with jails and the mystery has been removed, I converted all my existing jails to vanilla and I’m good to go!

Leave a Comment

Scroll to Top