Jails with embedded, but not jailed, ZFS datasets – how to mount/umount – corrected

This is a rewrite of Jails with embedded, but not jailed, ZFS datasets – how to mount/umount.

First, this is not about jailed ZFS datasets. I use them for FreshPorts, but that’s not I’m going to talk about here.

In this post:

• FreeBSD 13.2

This is my example:

[22:25 r730-03 dvl ~] % zfs list | grep bacula-sd-04      
data01/bacula-volumes                         12.1T  4.18T       96K  /jails/bacula-sd-04/usr/local/bacula/volumes
data01/bacula-volumes/DiffFile                 109G   915G      109G  /jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile
data01/bacula-volumes/FullFile                4.00T  2.00T     3.89T  /jails/bacula-sd-04/usr/local/bacula/volumes/FullFile
data01/bacula-volumes/FullFileNoNextPool      7.80T  4.18T     7.80T  /jails/bacula-sd-04/usr/local/bacula/volumes/FullFileNoNextPool
data01/bacula-volumes/IncrFile                 207G  2.30T      105G  /jails/bacula-sd-04/usr/local/bacula/volumes/IncrFile
data01/jails/bacula-sd-04                     2.17G  4.18T     1.60G  /jails/bacula-sd-04
[22:25 r730-03 dvl ~] % 

The jail bacula-sd-04 has a lot of data, backups, which persists outside the jail. I didn’t want all this data inside the data01/jails/bacula-sd-04 dataset, because that’s for jails. In other hosts that data is in a entirely different zpool. In fact, I may move all the jails into a different zpool. Keeping the data separate is what I want to do.
The problem, if I want to umount/move/play, the fact that other datasets are mounted into it causes a problem. I can’t because:

[23:25 r730-03 dvl ~] % sudo service jail stop bacula-sd-04
Stopping jails: bacula-sd-04.
[23:27 r730-03 dvl ~] % sudo zfs umount data01/jails/bacula-sd-04
cannot unmount ‘/jails/bacula-sd-04’: pool or dataset is busy
[23:27 r730-03 dvl ~] %

I have a solution.

The background

A jail can have an fstab and that serves exactly my purpose. However,I want thefstab specified in the jail configuration so that everything to do with the jail is defined there. Either way is acceptable.

To use an external fstab file, I used this entry in the jail configuration:

mount.fstab="/etc/fstab.$name";

Given the jail filename (bacula-sd-04), this is what I have in /etc/fstab.bacula-sd-04

[22:28 r730-03 dvl ~] % cat /etc/fstab.bacula-sd-04                                  
data01/bacula-volumes/DiffFile           /jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile           zfs   rw,noatime        0 0
data01/bacula-volumes/FullFile           /jails/bacula-sd-04/usr/local/bacula/volumes/FullFile           zfs   rw,noatime        0 0
data01/bacula-volumes/FullFileNoNextPool /jails/bacula-sd-04/usr/local/bacula/volumes/FullFileNoNextPool zfs   rw,noatime        0 0
data01/bacula-volumes/IncrFile           /jails/bacula-sd-04/usr/local/bacula/volumes/IncrFile           zfs   rw,noatime        0 0

This is the jail configuration for that approach:

bacula-sd-04 {
        persist;
        mount.fstab="/etc/fstab.$name";
        ip4.addr = "$NIC|10.55.0.7";
}

That works as demonstrated here.

Start the jail, we have files:

[23:04 r730-03 dvl ~] % sudo service jail restart bacula-sd-04         
Stopping jails:.
Starting jails: bacula-sd-04.
[23:05 r730-03 dvl ~] % sudo find /jails/bacula-sd-04/usr/local/bacula/volumes | head
/jails/bacula-sd-04/usr/local/bacula/volumes
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile/DiffAuto-04-17748
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile/DiffAuto-04-18042
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile/DiffAuto-04-18138
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile/DiffAuto-04-19317
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile/DiffAuto-04-19289
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile/DiffAuto-04-17735
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile/DiffAuto-04-18048
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile/DiffAuto-04-18132
[23:05 r730-03 dvl ~] % 

Stop the jails, we don’t have files:

[23:05 r730-03 dvl ~] % sudo service jail stop bacula-sd-04   
Stopping jails: bacula-sd-04.
[23:05 r730-03 dvl ~] % sudo find /jails/bacula-sd-04/usr/local/bacula/volumes | head
/jails/bacula-sd-04/usr/local/bacula/volumes
/jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile
/jails/bacula-sd-04/usr/local/bacula/volumes/IncrFile
/jails/bacula-sd-04/usr/local/bacula/volumes/FullFileNoNextPool
/jails/bacula-sd-04/usr/local/bacula/volumes/FullFile
[23:05 r730-03 dvl ~] % 

Success.

Now, all in the configuration

Here is the approach I’m taking now, courtesy of Jan Bramkamp about this.

bacula-sd-04 {
        persist;
        ip4.addr = "$NIC|10.55.0.7";

        # keep this with tabs
        $fstab = "<< 'EOF'
                data01/bacula-volumes/DiffFile           $path/usr/local/bacula/volumes/DiffFile           zfs   rw,noatime        0 0
                data01/bacula-volumes/FullFile           $path/usr/local/bacula/volumes/FullFile           zfs   rw,noatime        0 0
                data01/bacula-volumes/FullFileNoNextPool $path/usr/local/bacula/volumes/FullFileNoNextPool zfs   rw,noatime        0 0
                data01/bacula-volumes/IncrFile           $path/usr/local/bacula/volumes/IncrFile           zfs   rw,noatime        0 0
        EOF";

        exec.prepare  += "/sbin/mount  -a -v    -F /dev/stdin $fstab";
        exec.poststop += "/sbin/umount -a -v -f -F /dev/stdin $fstab";
}

This works. It works well, and it let's me see the details all in one place. I may or may not take this approach, but it is especially useful if you are using one-jail-per-configuration file in /etc/jail.conf.d/.