snapshot ; ls ; destroy; dataset is busy – WTF?

I have found this bizarre situation where a FreeBSD 12.2 snapshot cannot be destroyed.

This is the simple example, which works:

[dan@pkg01:~] $ sudo zfs snapshot tank_fast/poudriere@snapshot-for-backup
[dan@pkg01:~] $ sudo zfs destroy tank_fast/poudriere@snapshot-for-backup
[dan@pkg01:~] $

This is the simple example, which fails:

[dan@pkg01:~] $ sudo zfs snapshot tank_fast/poudriere@snapshot-for-backup
[dan@pkg01:~] $ ls /usr/local/poudriere/.zfs/snapshot/snapshot-for-backup
cache data  jails ports
[dan@pkg01:~] $ sudo zfs destroy tank_fast/poudriere@snapshot-for-backup
cannot destroy snapshot tank_fast/poudriere@snapshot-for-backup: dataset is busy
[dan@pkg01:~] $

EDIT: we have confirmed the above is mounted.We have tried to umount it from within the jail.

[dan@pkg01:~] $ mount -v | grep snapshot-for-backup
tank_fast/poudriere@snapshot-for-backup on /usr/local/poudriere/.zfs/snapshot/snapshot-for-backup (zfs, local, noatime, nosuid, read-only, nfsv4acls)

[dan@pkg01:~] $ sudo umount tantank_fast/poudriere@snapshot-for-backup
umount: unmount of /usr/local/poudriere/.zfs/snapshot/snapshot-for-backup failed: Operation not permitted

[dan@pkg01:~] $ sudo umount /us/usr/local/poudriere/.zfs/snapshot/snapshot-for-backup
umount: unmount of /usr/local/poudriere/.zfs/snapshot/snapshot-for-backup failed: Operation not permitted
[dan@pkg01:~] $

Notes, some of which were penned by dch during our IRC debugging session:

The above was performed in the jail pkg01
The jail is on the host r720-01
Time does not change the above. I can wait hours and still cannot destroy the snapshot.
ls (the snapshot dir causes something to make the dataset get “busy”) – I have also tried cd into the directory and back out again with the same result.
but only when these activities are done from within the jail – on the host, the problem does not appear
there are no holds, nor visible mountpoints (zfs list -o mounted,… nor mount |grep poudriere) as seen either within the host, or the jail
there are no other non-zfs mountpoints stacked on top (tmpfs, devfs etc)
even a `zfs destroy -f` fails here – in the jail
if it’s not actually zfs, what else might be holding stuff open?
there are definitely no nullmounts involved – confirmed by mount | grep -i null on the host

As root in the jail, same problem:

[root@pkg01 ~]# zfs snapshot tank_fast/poudriere@snapshot-for-backup
[root@pkg01 ~]# ls /usr/local/poudriere/.zfs/snapshot/snapshot-for-backup
cache data  jails ports
[root@pkg01 ~]# zfs destroy tank_fast/poudriere@snapshot-for-backup
cannot destroy snapshot tank_fast/poudriere@snapshot-for-backup: dataset is busy
[root@pkg01 ~]#

configuation

The /etc/jail.conf configuration for this jail:

exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.consolelog="/var/tmp/$name";
mount.devfs;
path = /jails/$name;
# depreacted; see man jail
#allow.sysvipc;
allow.raw_sockets;
#securelevel = 2;

exec.prestart  = "logger trying to start jail $name...";
exec.poststart = "logger jail $name has started";
exec.prestop   = "logger shutting down jail $name";
exec.poststop  = "logger jail $name has shut down";

host.hostname = "$name.int.unixathome.org";

persist;

pkg01 {

    allow.mount.fdescfs;

    path = /jails/$name;
    vnet;
    allow.raw_sockets;
    allow.socket_af;
    allow.mount.linprocfs;
    allow.mount.nullfs;
    allow.mount.procfs;
    allow.mount.tmpfs;
    allow.mount.zfs;
    sysvmsg=new;
    sysvsem=new;
    sysvshm=new;
    allow.mount;
    allow.mount.devfs;
    host.domainname=none;
    allow.chflags;
    enforce_statfs=1;
    children.max=100;
    vnet.interface   = "e0b_$name";

    exec.prestart   += "/usr/local/sbin/jib addm    $name ix0";
    exec.poststop   += "/usr/local/sbin/jib destroy $name";

    exec.poststart  += "jail -m allow.mount.linprocfs=1 name=$name";

    exec.created+="zfs set jailed=on tank_fast/poudriere";
    exec.created+="zfs jail $name    tank_fast/poudriere";
}

Is it jailed?

Is it jailed? Yes, see also the exec.prestart in the configuration above.

[dan@r720-01:~] $ zfs get -r -t filesystem jailed tank_fast/poudriere
NAME                                          PROPERTY  VALUE   SOURCE
tank_fast/poudriere                           jailed    on      local
tank_fast/poudriere/ccache.amd64              jailed    on      local
tank_fast/poudriere/ccache.i386               jailed    on      local
tank_fast/poudriere/data                      jailed    on      local
tank_fast/poudriere/data/cache                jailed    on      local
tank_fast/poudriere/data/cronjob-logs         jailed    on      local
tank_fast/poudriere/data/packages             jailed    on      local
tank_fast/poudriere/distfiles                 jailed    on      local
tank_fast/poudriere/jails                     jailed    on      local
tank_fast/poudriere/jails/121amd64            jailed    on      inherited from tank_fast/poudriere/jails
tank_fast/poudriere/jails/121i386             jailed    on      inherited from tank_fast/poudriere/jails
tank_fast/poudriere/jails/122amd64            jailed    on      inherited from tank_fast/poudriere/jails
tank_fast/poudriere/jails/122i386             jailed    on      inherited from tank_fast/poudriere/jails
tank_fast/poudriere/ports                     jailed    on      local
tank_fast/poudriere/ports/2021Q2_DELETEME     jailed    on      inherited from tank_fast/poudriere/ports
tank_fast/poudriere/ports/default             jailed    on      inherited from tank_fast/poudriere/ports
tank_fast/poudriere/ports/default.subversion  jailed    on      local
tank_fast/poudriere/ports/main                jailed    on      local
tank_fast/poudriere/ports/testing             jailed    on      local
[dan@r720-01:~] $

For fun, I changed all the SOURCE=local to SOURCE=inherited from tank_fast/poudriere. It did not change the problem.

On the host, attempting to jail the filesystem gives:

[dan@r720-01:~] $ sudo zfs jail pkg01 tank_fast/poudriere
cannot jail 'tank_fast/poudriere': dataset already exists
[dan@r720-01:~] $

lsof

[dan@r720-01:~] $ sudo lsof | grep snapshot-for-backup
lsof: WARNING: compiled for FreeBSD release 12.2-RELEASE-p2; this is 12.2-RELEASE-p6.
lsof: WARNING: access /root/.lsof_r720-01: No such file or directory
lsof: WARNING: created device cache file: /root/.lsof_r720-01
lsof: no pwd entry for UID 770
lsof: no pwd entry for UID 770
...
lsof: no pwd entry for UID 770
lsof: no pwd entry for UID 1003
...
lsof: no pwd entry for UID 1003
lsof: no pwd entry for UID 243
...
lsof: no pwd entry for UID 243
lsof: no pwd entry for UID 223
...
lsof: no pwd entry for UID 223
lsof: no pwd entry for UID 223
lsof: no pwd entry for UID 1004
lsof: no pwd entry for UID 1004
...
lsof: no pwd entry for UID 1004
lsof: no pwd entry for UID 1004
lsof: no pwd entry for UID 1004
lsof: no pwd entry for UID 770
lsof: no pwd entry for UID 770
...

lsof: no pwd entry for UID 770
[dan@r720-01:~] $

Reproducing

I reproduced this with another newly created filesystem in this jail:

$ sudo zfs create tank_fast/poudriere/test
$ zfs list tank_fast/poudriere/test
NAME                       USED  AVAIL  REFER  MOUNTPOINT
tank_fast/poudriere/test   160K   494G    96K  /usr/local/poudriere/test
$ sudo touch /usr/local/poudriere/test/a
$ sudo zfs snapshot tank_fast/poudriere/test@test
$ sudo zfs destroy tank_fast/poudriere/test@test
$ sudo zfs snapshot tank_fast/poudriere/test@test
$ ls /usr/local/poudriere/test/.zfs/snapshot/test
a
$ sudo zfs destroy tank_fast/poudriere/test@test
cannot destroy snapshot tank_fast/poudriere/test@test: dataset is busy

It is interesting that these are being mounted:

[dan@pkg01:~] $ mount -v | grep tank_fast/poudriere/test
tank_fast/poudriere/test on /usr/local/poudriere/test (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/test@test on /usr/local/poudriere/test/.zfs/snapshot/test (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_21:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_21:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_20:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_20:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_16:40:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_16:40:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_17:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_17:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_18:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_18:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_16:40:00_daily on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_16:40:00_daily (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_19:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_19:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
[dan@pkg01:~] $ tank_fast/poudriere/test

We cannot unmount:

[dan@pkg01:~] $ sudo umount /us/usr/local/poudriere/test/.zfs/snapshot/test
umount: unmount of /usr/local/poudriere/test/.zfs/snapshot/test failed: Operation not permitted
[dan@pkg01:~] $

There is no way to recover from this situation from within the jail.

zfs get all

[dan@r720-01:~] $ zfs get all tank_fast/poudriere
NAME                 PROPERTY              VALUE                  SOURCE
tank_fast/poudriere  type                  filesystem             -
tank_fast/poudriere  creation              Mon Feb 12  1:30 2018  -
tank_fast/poudriere  used                  95.3G                  -
tank_fast/poudriere  available             494G                   -
tank_fast/poudriere  referenced            104K                   -
tank_fast/poudriere  compressratio         1.82x                  -
tank_fast/poudriere  mounted               yes                    -
tank_fast/poudriere  quota                 none                   default
tank_fast/poudriere  reservation           none                   default
tank_fast/poudriere  recordsize            128K                   default
tank_fast/poudriere  mountpoint            /usr/local/poudriere   local
tank_fast/poudriere  sharenfs              off                    default
tank_fast/poudriere  checksum              on                     default
tank_fast/poudriere  compression           lz4                    received
tank_fast/poudriere  atime                 off                    inherited from tank_fast
tank_fast/poudriere  devices               on                     default
tank_fast/poudriere  exec                  on                     default
tank_fast/poudriere  setuid                on                     default
tank_fast/poudriere  readonly              off                    default
tank_fast/poudriere  jailed                on                     local
tank_fast/poudriere  snapdir               hidden                 default
tank_fast/poudriere  aclmode               discard                default
tank_fast/poudriere  aclinherit            restricted             default
tank_fast/poudriere  createtxg             1384766                -
tank_fast/poudriere  canmount              on                     default
tank_fast/poudriere  xattr                 off                    temporary
tank_fast/poudriere  copies                1                      default
tank_fast/poudriere  version               5                      -
tank_fast/poudriere  utf8only              off                    -
tank_fast/poudriere  normalization         none                   -
tank_fast/poudriere  casesensitivity       sensitive              -
tank_fast/poudriere  vscan                 off                    default
tank_fast/poudriere  nbmand                off                    default
tank_fast/poudriere  sharesmb              off                    default
tank_fast/poudriere  refquota              none                   default
tank_fast/poudriere  refreservation        none                   default
tank_fast/poudriere  guid                  3778979138212040322    -
tank_fast/poudriere  primarycache          all                    default
tank_fast/poudriere  secondarycache        all                    default
tank_fast/poudriere  usedbysnapshots       56K                    -
tank_fast/poudriere  usedbydataset         104K                   -
tank_fast/poudriere  usedbychildren        95.3G                  -
tank_fast/poudriere  usedbyrefreservation  0                      -
tank_fast/poudriere  logbias               latency                default
tank_fast/poudriere  objsetid              22.8K                  -
tank_fast/poudriere  dedup                 off                    default
tank_fast/poudriere  mlslabel                                     -
tank_fast/poudriere  sync                  standard               default
tank_fast/poudriere  dnodesize             legacy                 default
tank_fast/poudriere  refcompressratio      1.00x                  -
tank_fast/poudriere  written               0                      -
tank_fast/poudriere  logicalused           158G                   -
tank_fast/poudriere  logicalreferenced     41K                    -
tank_fast/poudriere  volmode               default                default
tank_fast/poudriere  filesystem_limit      none                   default
tank_fast/poudriere  snapshot_limit        none                   default
tank_fast/poudriere  filesystem_count      none                   default
tank_fast/poudriere  snapshot_count        none                   default
tank_fast/poudriere  redundant_metadata    all                    default
tank_fast/poudriere  special_small_blocks  0                      default
[dan@r720-01:~] $

These are my jail configuration settings:

[dan@r720-01:~] $ jls -n -j 374 | xargs -n 1
devfs_ruleset=0
nodying
enforce_statfs=1
host=new
ip4=inherit
ip6=inherit
jid=374
linux=new
name=pkg01
osreldate=1202000
osrelease=12.2-RELEASE-p6
parent=0
path=/jails/pkg01
persist
securelevel=-1
sysvmsg=new
sysvsem=new
sysvshm=new
vnet=new
allow.chflags
allow.nomlock
allow.mount
allow.mount.devfs
allow.mount.fdescfs
allow.mount.linprocfs
allow.mount.nullfs
allow.mount.procfs
allow.mount.tmpfs
allow.mount.zfs
allow.noquotas
allow.raw_sockets
allow.noread_msgbuf
allow.reserved_ports
allow.set_hostname
allow.socket_af
allow.nosysvipc
children.cur=0
children.max=100
cpuset.id=7
host.domainname=none
host.hostid=0
host.hostname=pkg01.int.unixathome.org
host.hostuuid=00000000-0000-0000-0000-000000000000
ip4.addr=
ip4.saddrsel
ip6.addr=
ip6.saddrsel
linux.osname=Linux
linux.osrelease=3.2.0
linux.oss_version=198144
[dan@r720-01:~] $ jls -n -j 374 | xargs -n 1 | sort
allow.chflags
allow.mount
allow.mount.devfs
allow.mount.fdescfs
allow.mount.linprocfs
allow.mount.nullfs
allow.mount.procfs
allow.mount.tmpfs
allow.mount.zfs
allow.nomlock
allow.noquotas
allow.noread_msgbuf
allow.nosysvipc
allow.raw_sockets
allow.reserved_ports
allow.set_hostname
allow.socket_af
children.cur=0
children.max=100
cpuset.id=7
devfs_ruleset=0
enforce_statfs=1
host.domainname=none
host.hostid=0
host.hostname=pkg01.int.unixathome.org
host.hostuuid=00000000-0000-0000-0000-000000000000
host=new
ip4.addr=
ip4.saddrsel
ip4=inherit
ip6.addr=
ip6.saddrsel
ip6=inherit
jid=374
linux.osname=Linux
linux.osrelease=3.2.0
linux.oss_version=198144
linux=new
name=pkg01
nodying
osreldate=1202000
osrelease=12.2-RELEASE-p6
parent=0
path=/jails/pkg01
persist
securelevel=-1
sysvmsg=new
sysvsem=new
sysvshm=new
vnet=new
[dan@r720-01:~] $

Other snapshots are mounted

Added on 2021-04-26 14:19 UTC.

I noticed all these mounts (this is in the jail):

[dan@pkg01:~] $ mount -v
tank_fast/jails/pkg01 on / (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere on /usr/local/poudriere (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/distfiles on /usr/ports/distfiles (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/data on /usr/local/poudriere/data (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/ports on /usr/local/poudriere/ports (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/jails on /usr/local/poudriere/jails (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/data/packages on /usr/local/poudriere/data/packages (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/jails/121amd64 on /usr/local/poudriere/jails/121amd64 (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/jails/122amd64 on /usr/local/poudriere/jails/122amd64 (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/ports/main on /usr/local/poudriere/ports/main (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/jails/121i386 on /usr/local/poudriere/jails/121i386 (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/ports/default on /usr/local/poudriere/ports/default (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/data/cronjob-logs on /usr/local/poudriere/data/cronjob-logs (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/data/cache on /usr/local/poudriere/data/cache (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/jails/122i386 on /usr/local/poudriere/jails/122i386 (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/ports/default.subversion on /usr/local/poudriere/ports/default.subversion (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/ports/testing on /usr/local/poudriere/ports/testing (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/ports/2021Q2_DELETEME on /usr/local/poudriere/ports/2021Q2_DELETEME (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/ccache.i386 on /var/cache/ccache.i386 (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/ccache.amd64 on /var/cache/ccache.amd64 (zfs, local, noatime, nfsv4acls)
procfs on /proc (procfs, local)
tank_fast/poudriere@autosnap_2019-11-13_20:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_20:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_22:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_22:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_08:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_08:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_09:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_09:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_23:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_23:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_20:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_20:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_21:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_21:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_12:15:00_frequently on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_12:15:00_frequently (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_23:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_23:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_22:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_22:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_05:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_05:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_11:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_11:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_10:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_10:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_04:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_04:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_12:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_12:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_11:45:00_frequently on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_11:45:00_frequently (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_06:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_06:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_11:30:00_frequently on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_11:30:00_frequently (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_07:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_07:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_02:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_02:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_12:30:00_frequently on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_12:30:00_frequently (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_03:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_03:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_01:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_01:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_00:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_00:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_05:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_05:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_11:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_11:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_17:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_17:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_16:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_16:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_10:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_10:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-11_23:05:00_yearly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-11_23:05:00_yearly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_04:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_04:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_12:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_12:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_14:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_14:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_06:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_06:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_07:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_07:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_15:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_15:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_13:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_13:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_02:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_02:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_16:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_16:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_17:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_17:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_03:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_03:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_13:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_13:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_15:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_15:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_01:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_01:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_00:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_00:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_14:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_14:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_19:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_19:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-12_18:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-12_18:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-11_23:05:00_daily on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-11_23:05:00_daily (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_08:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_08:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_12:00:00_frequently on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_12:00:00_frequently (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-11_23:05:00_monthly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-11_23:05:00_monthly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_09:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_09:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_00:00:00_daily on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_00:00:00_daily (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_18:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_18:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-13_19:00:00_hourly on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_19:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere@autosnap_2019-11-14_00:00:00_daily on /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-14_00:00:00_daily (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test on /usr/local/poudriere/test (zfs, local, noatime, nfsv4acls)
tank_fast/poudriere/test@test on /usr/local/poudriere/test/.zfs/snapshot/test (zfs, local, noatime, nosuid, read-only, nfsv4acls)
devfs on /dev (devfs)
tank_fast/poudriere/test@autosnap_2021-04-25_21:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_21:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_20:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_20:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_16:40:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_16:40:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_17:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_17:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_18:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_18:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_16:40:00_daily on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_16:40:00_daily (zfs, local, noatime, nosuid, read-only, nfsv4acls)
tank_fast/poudriere/test@autosnap_2021-04-25_19:00:00_hourly on /usr/local/poudriere/test/.zfs/snapshot/autosnap_2021-04-25_19:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
[dan@pkg01:~] $

They cannot be unmounted from within the jail:

[dan@pkg01:~] $ sudo umount /us/usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_22:00:00_hourly
umount: unmount of /usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_22:00:00_hourly failed: Operation not permitted

The mounts are visible on the host, and they can be unmounted there:

[dan@r720-01:~] $ mount -v | grep autosnap_2019-11-13_22:00:00_hourly
tank_fast/poudriere@autosnap_2019-11-13_22:00:00_hourly on /jails/pkg01/usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_22:00:00_hourly (zfs, local, noatime, nosuid, read-only, nfsv4acls)
[dan@r720-01:~] $ sudo umount /ja/jails/pkg01/usr/local/poudriere/.zfs/snapshot/autosnap_2019-11-13_22:00:00_hourly
[dan@r720-01:~] $ mount -v | grep autosnap_2019-11-13_22:00:00_hourly
[dan@r720-01:~] $

Any ideas?