This post will be short on detail. I’m writing it for myself, to document what I’m doing.
I suspect Migrating data02/freshports/dev-ingress01 to another zpool will be more useful to you.
In this post:
- FreeBSD 15.0
I stopped the jail:
[15:30 r730-01 dvl ~] % sudo service jail stop dev-nginx01 Stopping jails: dev-nginx01.
I took dev.freshports.org offline via:
[15:32 r720-02-proxy01 dvl ~] % ~/bin/offline dev.freshports.org "$(date -R -v +2H)" "Offline for filesystem migration"
<html>
<head>
<title>Error 503 Service Unavailable</title>
<style>
<style>
body { text-align: center; padding: 20px; font: 25px Helvetica, sans-serif; color: #efe8e8; background-color:#2e2929}
@media (min-width: 768px){
body{ padding-top: 150px; }
}
h1 { font-size: 50px; }
h2 { font-size: 35px; }
h3 { font-size: 28px; }
article { display: block; text-align: left; max-width: 650px; margin: 0 auto; }
td { font-size: 25px; line-height: 1.5; margin: 20px 0; }
</style>
</style>
</head>
<body>
<h1>Server is offline for maintenance</h1>
<h2>503 Service Unavailable</h2>
<h3>Offline for filesystem migration</h3>
<table>
<tr>
<td>Started at:</td><td>Mon, 20 Apr 2026 15:32:42 +0000</td>
</tr>
<tr>
<td>Please retry after:</td><td> Mon, 20 Apr 2026 17:32:42 +0000</td>
</tr>
</table>
</body>
</html>
The source
[15:29 r730-01 dvl ~] % zfs list -r data02/freshports/dev-nginx01 NAME USED AVAIL REFER MOUNTPOINT data02/freshports/dev-nginx01 54.7M 338G 96K none data02/freshports/dev-nginx01/www 54.6M 338G 96K /jails/dev-nginx01/usr/local/www data02/freshports/dev-nginx01/www/freshports 51.8M 338G 51.8M /jails/dev-nginx01/usr/local/www/freshports data02/freshports/dev-nginx01/www/freshsource 2.71M 338G 2.71M /jails/dev-nginx01/usr/local/www/freshsource [15:33 r730-01 dvl ~] % zfs get -t filesystem mountpoint -r data02/freshports/dev-nginx01 | grep -v no NAME PROPERTY VALUE SOURCE data02/freshports/dev-nginx01/www mountpoint /jails/dev-nginx01/usr/local/www received data02/freshports/dev-nginx01/www/freshports mountpoint /jails/dev-nginx01/usr/local/www/freshports inherited from data02/freshports/dev-nginx01/www data02/freshports/dev-nginx01/www/freshsource mountpoint /jails/dev-nginx01/usr/local/www/freshsource inherited from data02/freshports/dev-nginx01/www [15:34 r730-01 dvl ~] % zfs get -t filesystem canmount -r data02/freshports/dev-nginx01 | grep -v on NAME PROPERTY VALUE SOURCE data02/freshports/dev-nginx01/www canmount off received data02/freshports/dev-nginx01/www/freshports canmount noauto received data02/freshports/dev-nginx01/www/freshsource canmount noauto received [15:35 r730-01 dvl ~] % zfs get -t filesystem mounted -r data02/freshports/dev-nginx01 | grep -v yes NAME PROPERTY VALUE SOURCE data02/freshports/dev-nginx01 mounted no - data02/freshports/dev-nginx01/www mounted no - data02/freshports/dev-nginx01/www/freshports mounted no - data02/freshports/dev-nginx01/www/freshsource mounted no -
Now, you might look at that, and wonder why the filesystems are offline.
This is why:
[15:30 r730-01 dvl ~] % cat /etc/jail.conf.d/dev-nginx01.conf
dev-nginx01 {
#
# start of standard settings for each jail
#
$bridge = "bridge0";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
path = /jails/$name;
allow.raw_sockets;
#securelevel = 2;
host.hostname = "$name.int.unixathome.org";
exec.consolelog="/var/tmp/jail-console-$name.log";
persist;
#
# end of standard settings for each jail
#
ip4.addr = "$bridge|10.55.0.39";
allow.mount=true;
allow.mount.zfs=true;
enforce_statfs=1;
devfs_ruleset=5;
mount.fstab="/etc/fstab.$name";
# because want to mount zfs, we do that before the rc start up
# not sure if we MUST do it in that order.
exec.start="zfs mount -a";
exec.start+="/bin/sh /etc/rc";
mount.fstab="/etc/fstab.$name";
# jail all the things.
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache";
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache/categories";
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache/commits";
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache/daily";
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache/general";
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache/news";
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache/packages";
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache/pages";
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache/ports";
exec.created+="zfs set jailed=on data02/freshports/jailed/dev-nginx01/cache/spooling";
exec.created+="zfs jail $name data02/freshports/jailed/dev-nginx01/cache";
# mount things
exec.created+="zfs mount data02/freshports/dev-nginx01/www/freshports";
exec.created+="zfs mount data02/freshports/dev-nginx01/www/freshsource";
# unjail and umount so we can get access to the underlying mount points
# when required/
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache";
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache/categories";
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache/commits";
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache/daily";
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache/general";
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache/news";
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache/packages";
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache/pages";
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache/ports";
exec.poststop+="zfs set jailed=off data02/freshports/jailed/dev-nginx01/cache/spooling";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache/categories";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache/commits";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache/daily";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache/general";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache/news";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache/packages";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache/pages";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache/ports";
exec.poststop+="zfs umount data02/freshports/jailed/dev-nginx01/cache/spooling";
exec.poststop+="zfs umount data02/freshports/dev-nginx01/www/freshports";
exec.poststop+="zfs umount data02/freshports/dev-nginx01/www/freshsource";
}
Start/stop the jail, and the filesystems are mounted/umounted.
Why? So I can manipulate the filesystems without having to umount them manually.
The snapshot and copy
Snapshot:
[15:35 r730-01 dvl ~] % sudo zfs snapshot -r data02/freshports/dev-nginx01@for.data04.0 [15:38 r730-01 dvl ~] %
Send:
root@r730-01:/home/dvl # zfs send -R data02/freshports/dev-nginx01@for.data04.0 | zfs receive data04/freshports/dev-nginx01 root@r730-01:/home/dvl #
Received:
root@r730-01:/home/dvl # zfs list -r data04/freshports/dev-nginx01 NAME USED AVAIL REFER MOUNTPOINT data04/freshports/dev-nginx01 92.7M 15.0T 205K /data04/freshports/dev-nginx01 data04/freshports/dev-nginx01/www 92.5M 15.0T 205K /jails/dev-nginx01/usr/local/www data04/freshports/dev-nginx01/www/freshports 87.1M 15.0T 87.1M /jails/dev-nginx01/usr/local/www/freshports data04/freshports/dev-nginx01/www/freshsource 5.17M 15.0T 5.17M /jails/dev-nginx01/usr/local/www/freshsource
Why does the destination take so much more space?
root@r730-01:/home/dvl # zfs get compression data02/freshports/dev-nginx01 data04/freshports/dev-nginx01 NAME PROPERTY VALUE SOURCE data02/freshports/dev-nginx01 compression zstd inherited from data02 data04/freshports/dev-nginx01 compression zstd inherited from data04 root@r730-01:/home/dvl # zfs get recordsize data02/freshports/dev-nginx01 data04/freshports/dev-nginx01 NAME PROPERTY VALUE SOURCE data02/freshports/dev-nginx01 recordsize 32K inherited from data02/freshports data04/freshports/dev-nginx01 recordsize 128K inherited from data04 root@r730-01:/home/dvl # zfs get -r -t filesystem recordsize data02/freshports/dev-nginx01 data04/freshports/dev-nginx01 NAME PROPERTY VALUE SOURCE data02/freshports/dev-nginx01 recordsize 32K inherited from data02/freshports data02/freshports/dev-nginx01/www recordsize 128K received data02/freshports/dev-nginx01/www/freshports recordsize 128K received data02/freshports/dev-nginx01/www/freshsource recordsize 128K received data04/freshports/dev-nginx01 recordsize 128K inherited from data04 data04/freshports/dev-nginx01/www recordsize 128K received data04/freshports/dev-nginx01/www/freshports recordsize 128K received data04/freshports/dev-nginx01/www/freshsource recordsize 128K received root@r730-01:/home/dvl #
I don’t know. Nothing clear in there to me.
Turning off the old filesystem
Set canmount:
root@r730-01:/home/dvl # zfs list -Hr data02/freshports/dev-nginx01 | cut -f 1 -w | tail -r | xargs -n 1 sudo zfs set canmount=off root@r730-01:/home/dvl #
rename:
root@r730-01:/home/dvl # zfs rename data02/freshports/dev-nginx01 data02/freshports/dev-nginx01.DELETE.ME.after.2026.05.15
Start the webserver
root@r730-01:/home/dvl # service jail start dev-nginx01 Starting jails: dev-nginx01. root@r730-01:/home/dvl #
Put the site back online.
[15:58 r720-02-proxy01 dvl ~] % sudo rm /usr/local/www/offline/dev.freshports.org-maintenance.html [16:01 r720-02-proxy01 dvl ~] %
Done. Looks good to me.
Edits
This was done to keep sanoid working with the new filesystems, not the old (and now renamed):
[16:03 r730-01 dvl ~] % sudoedit /usr/local/etc/sanoid/sanoid.conf
Old snapshots
Here go the old snapshots:
[16:08 r730-01 dvl ~] % zfs list -H -o name -r -t snapshot data02/freshports/dev-nginx01.DELETE.ME.after.2026.05.15 | grep -v for.data04.0 | xargs -n 1 sudo zfs destroy
and now we have:
[16:09 r730-01 dvl ~] % zfs list -r data02/freshports/dev-nginx01.DELETE.ME.after.2026.05.15 NAME USED AVAIL REFER MOUNTPOINT data02/freshports/dev-nginx01.DELETE.ME.after.2026.05.15 54.7M 338G 96K none data02/freshports/dev-nginx01.DELETE.ME.after.2026.05.15/www 54.6M 338G 96K /jails/dev-nginx01/usr/local/www data02/freshports/dev-nginx01.DELETE.ME.after.2026.05.15/www/freshports 51.8M 338G 51.8M /jails/dev-nginx01/usr/local/www/freshports data02/freshports/dev-nginx01.DELETE.ME.after.2026.05.15/www/freshsource 2.71M 338G 2.71M /jails/dev-nginx01/usr/local/www/freshsource
