This approach did not work, because I used the wrong set of filesystems. See below for Error.
This post has been replaced by Jails with embedded, but not jailed, ZFS datasets – how to mount/umount – corrected.
NOTE: on yesterday’s (2023-10-04) reboot, the file systems did not properly mount. zfs get mounted said they were mounted, but the directories were empty. Stopping, then starting the jail resulted in a proper mount. I wonder if this is some race condition during boot.
EDIT: 2023-10-22 This problem is persistent. The host was rebooted yesterday. Restarting the jail solves the problem. I’m going to add a monitoring check. Eventually, we need to solve this problem.
First, this is not about jailed ZFS datasets. I use them for FreshPorts, but that’s not I’m going to talk about here.
In this post:
- FreeBSD 13.2
This is my example:
[23:22 r730-03 dvl ~] % zfs list | grep bacula-sd-04 data01/bacula-volumes 7.40T 5.82T 96K /jails/bacula-sd-04/usr/local/bacula/volumes data01/bacula-volumes/DiffFile 71.6G 952G 71.6G /jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile data01/bacula-volumes/FullFile 3.48T 2.52T 3.42T /jails/bacula-sd-04/usr/local/bacula/volumes/FullFile data01/bacula-volumes/FullFileNoNextPool 3.30T 5.82T 3.30T /jails/bacula-sd-04/usr/local/bacula/volumes/FullFileNoNextPool data01/bacula-volumes/IncrFile 568G 1.95T 99.4G /jails/bacula-sd-04/usr/local/bacula/volumes/IncrFile data01/jails/bacula-sd-04 1.96G 5.82T 1.70G /jails/bacula-sd-04 [23:22 r730-03 dvl ~] %
The jail bacula-sd-04 has a lot of data, backups, which persists outside the jail. I didn’t want all this data inside the data01/jails/bacula-sd-04 dataset, because that’s for jails. In other hosts that data is in a entirely different zpool. In fact, I may move all the jails into a different zpool. Keeping the data separate is what I want to do.
The problem, if I want to umount/move/play, the fact that other datasets are mounted into it causes a problem. I can’t because:
[23:25 r730-03 dvl ~] % sudo service jail stop bacula-sd-04
Stopping jails: bacula-sd-04.
[23:27 r730-03 dvl ~] % sudo zfs umount data01/jails/bacula-sd-04
cannot unmount ‘/jails/bacula-sd-04’: pool or dataset is busy
[23:27 r730-03 dvl ~] %
I may have a solution.
The background
My dev-ingress01 jail configuration contains this:
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-ingress01";
exec.created+="zfs jail $name data02/freshports/jailed/dev-ingress01";
I do something like that for FreshPorts.
Where are these fancy directives defined? man 8 jail – enjoy the reading.
Why not use the same method to mount/umount?
I was talking to Jan Bramkamp about this, and the following is the example approach he provided.
$fstab = "<<- 'EOF'
devfs $path/dev devfs rw,ruleset=$devfs_ruleset 0 0
fdescfs $path/dev/fd fdescfs rw 0 0
tmpfs $path/tmp tmpfs rw,mode=1777,size=128m 0 0
tmpfs $path/var/cache tmpfs rw,mode=0755,size=128m 0 0
tmpfs $path/var/db mfs rw,size=256m,-k$base_path/var/db 0 0
tmpfs $path/var/run tmpfs rw,mode=0755,size=128m 0 0
tmpfs $path/var/tmp tmpfs rw,mode=1777,size=128m 0 0
EOF";
exec.prepare += "[ -d $snapshot_path ] || mount -F /dev/stdin -a -v $fstab";
How do I convert that to zfs? I’m not going to interpret the above for you, but I will show you what worked for me. My best clue was Alternate /etc/fstab in the Root On ZFS. Here is what I tried first.
The beauty of this: the same data ($fstab) is used for both mount and umount.
# path is defined globally and is the same for all jails.
# name will be bacula-sd-04, in this case.
path = /jails/$name;
bacula-sd-04 {
persist;
ip4.addr = "$NIC|10.55.0.7";
$fstab = "<<- 'EOF'
data01/bacula-volumes $path/usr/local/bacula/volumes zfs rw,noatime 0 0
data01/bacula-volumes/DiffFile $path/usr/local/bacula/volumes/DiffFile zfs rw,noatime 0 0
data01/bacula-volumes/FullFile $path/usr/local/bacula/volumes/FullFile zfs rw,noatime 0 0
data01/bacula-volumes/FullFileNoNextPool $path/usr/local/bacula/volumes/FullFileNoNextPool zfs rw,noatime 0 0
data01/bacula-volumes/IncrFile $path/usr/local/bacula/volumes/IncrFile zfs rw,noatime 0 0
EOF";
exec.prepare += "mount -F /dev/stdin -a -v $fstab";
exec.poststop += "umount -a -f -F /dev/stdin $fstab";
}
Why it didn’t work on boot
EDIT 2023-11-08 – There was a persistent problem with this approach. It would not work on reboot. It would work if I restarted the jail. Why? $path/usr/local/bacula/volumes should not be in that list.
[23:20 r730-03 dvl ~] % zfs list -r -o name,mounted,canmount,mountpoint data01/bacula-volumes NAME MOUNTED CANMOUNT MOUNTPOINT data01/bacula-volumes no off /jails/bacula-sd-04/usr/local/bacula/volumes data01/bacula-volumes/DiffFile no on /jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile data01/bacula-volumes/FullFile no on /jails/bacula-sd-04/usr/local/bacula/volumes/FullFile data01/bacula-volumes/FullFileNoNextPool no on /jails/bacula-sd-04/usr/local/bacula/volumes/FullFileNoNextPool data01/bacula-volumes/IncrFile no on /jails/bacula-sd-04/usr/local/bacula/volumes/IncrFile [23:20 r730-03 dvl ~] %
Although data01/bacula-volumes is specified as canmount off, that part is ignored with legacy mount, as opposed to zfs mount. The problem: data01/bacula-volumes would get mounted overtop of the other mount points, and you’d see nothing. That’s not helpful at all.
The solution, remove data01/bacula-volumes from the configuration. Then it worked just fine. Several people spent about 2 hours tracking that one down. It’s something we all knew about, but everyone suspected something else as the culprit.
We now return you to the original article
This takes advantage of the -Foption on mount
Specifying the mount points both here and zfs is a duplication. However, you could script that too.
I’m shocked to report that this worked first time.
Examples
I start the jail, everything is mounted:
[23:50 r730-03 dvl ~] % sudo service jail start bacula-sd-04 Starting jails: bacula-sd-04. [23:51 r730-03 dvl ~] % zfs get -r -t filesystem mounted data01/bacula-volumes NAME PROPERTY VALUE SOURCE data01/bacula-volumes mounted yes - data01/bacula-volumes/DiffFile mounted yes - data01/bacula-volumes/FullFile mounted yes - data01/bacula-volumes/FullFileNoNextPool mounted yes - data01/bacula-volumes/IncrFile mounted yes - [23:51 r730-03 dvl ~] %
I stop the jail, everything is unmounted:
[23:51 r730-03 dvl ~] % zfs get -r -t filesystem mounted data01/bacula-volumes NAME PROPERTY VALUE SOURCE data01/bacula-volumes mounted no - data01/bacula-volumes/DiffFile mounted no - data01/bacula-volumes/FullFile mounted no - data01/bacula-volumes/FullFileNoNextPool mounted no - data01/bacula-volumes/IncrFile mounted no - [23:52 r730-03 dvl ~] %
With the jail, stopped, I can manipulate the jail dataset:
[23:52 r730-03 dvl ~] % sudo zfs umount data01/jails/bacula-sd-04 [0:00 r730-03 dvl ~] % sudo zfs mount data01/jails/bacula-sd-04
Profit!
Oh wait, there’s more
I hit this problem:
[0:05 r730-03 dvl ~] % sudo service jail start bacula-sd-04
Starting jails: cannot start jail "bacula-sd-04":
mount: /jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile: No such file or directory
mount: /jails/bacula-sd-04/usr/local/bacula/volumes/FullFile: No such file or directory
mount: /jails/bacula-sd-04/usr/local/bacula/volumes/FullFileNoNextPool: No such file or directory
mount: /jails/bacula-sd-04/usr/local/bacula/volumes/IncrFile: No such file or directory
fstab: /dev/stdin:6: Inappropriate file type or format
jail: bacula-sd-04: /bin/sh -c mount -F /dev/stdin -a -v <<- 'EOF'
data01/bacula-volumes /jails/bacula-sd-04/usr/local/bacula/volumes zfs rw,noatime 0 0
data01/bacula-volumes/DiffFile /jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile zfs rw,noatime 0 0
data01/bacula-volumes/FullFile /jails/bacula-sd-04/usr/local/bacula/volumes/FullFile zfs rw,noatime 0 0
data01/bacula-volumes/FullFileNoNextPool /jails/bacula-sd-04/usr/local/bacula/volumes/FullFileNoNextPool zfs rw,noatime 0 0
data01/bacula-volumes/IncrFile /jails/bacula-sd-04/usr/local/bacula/volumes/IncrFile zfs rw,noatime 0 0
EOF: failed
.
I fixed it by creating the directories into the top level location.
[0:06 r730-03 dvl ~] % su Password: root@r730-03:/home/dvl # cd /jails/bacula-sd-04/usr/local/bacula/volumes root@r730-03:/jails/bacula-sd-04/usr/local/bacula/volumes # sudo mkdir DiffFile FullFile FullFileNoNextPool IncrFile root@r730-03:/jails/bacula-sd-04/usr/local/bacula/volumes # cd - root@r730-03:/home/dvl # exit
Then it worked as expected:
[0:07 r730-03 dvl ~] % sudo service jail start bacula-sd-04 Starting jails: bacula-sd-04. [0:07 r730-03 dvl ~] % sudo service jail stop bacula-sd-04 Stopping jails: bacula-sd-04. [0:07 r730-03 dvl ~] % sudo service jail start bacula-sd-04 Starting jails: bacula-sd-04. [0:07 r730-03 dvl ~] % sudo service jail stop bacula-sd-04 Stopping jails: bacula-sd-04. [0:07 r730-03 dvl ~] % zfs get -r -t filesystem mounted data01/bacula-volumes NAME PROPERTY VALUE SOURCE data01/bacula-volumes mounted no - data01/bacula-volumes/DiffFile mounted no - data01/bacula-volumes/FullFile mounted no - data01/bacula-volumes/FullFileNoNextPool mounted no - data01/bacula-volumes/IncrFile mounted no - [0:07 r730-03 dvl ~] %
In the end
I'm not sure how/why this might be better/worse than something the following. However, I know the above solution keeps the jail configuration all in one location. I think I like that better.
bacula-sd-04 {
persist;
ip4.addr = "$NIC|10.55.0.7";
mount.fstab = "/etc/fstab.$name"
}
Where {{/etc/fstab.bacula-sd-04}} contains:
data01/bacula-volumes /jails/bacula-sd-04/usr/local/bacula/volumes zfs rw,noatime 0 0 data01/bacula-volumes/DiffFile /jails/bacula-sd-04/usr/local/bacula/volumes/DiffFile zfs rw,noatime 0 0 data01/bacula-volumes/FullFile /jails/bacula-sd-04/usr/local/bacula/volumes/FullFile zfs rw,noatime 0 0 data01/bacula-volumes/FullFileNoNextPool /jails/bacula-sd-04/usr/local/bacula/volumes/FullFileNoNextPool zfs rw,noatime 0 0 data01/bacula-volumes/IncrFile /jails/bacula-sd-04/usr/local/bacula/volumes/IncrFile zfs rw,noatime 0 0
I know it's better than running these scripts:
[0:07 r730-03 dvl ~] % cat ~/bin/zfs-mount-bacula-sd-04-bacula-volumes #!/bin/sh zfs mount data01/bacula-volumes/DiffFile zfs mount data01/bacula-volumes/IncrFile zfs mount data01/bacula-volumes/FullFile zfs mount data01/bacula-volumes/FullFileNoNextPool [0:12 r730-03 dvl ~] % cat ~/bin/zfs-umount-bacula-sd-04-bacula-volumes #!/bin/sh zfs umount data01/bacula-volumes/DiffFile zfs umount data01/bacula-volumes/IncrFile zfs umount data01/bacula-volumes/FullFile zfs umount data01/bacula-volumes/FullFileNoNextPool
