It’s time.
Time to create the 8x 4T raidz2 zpool. That doesn’t happen in this post, hopefully the next one.
But first, I have to move some stuff around, to free up all the 4TB devices.
The host has 4 devices (NVMe) and I moved in two more, 4TB SSDs. From /var/log/messages:
Dec 9 00:24:29 r730-01 kernel: mrsas0: System PD created target ID: 0xc Dec 9 00:24:29 r730-01 kernel: mrsas0: System PD created target ID: 0xd Dec 9 00:24:30 r730-01 kernel: da12 at mrsas0 bus 1 scbus1 target 13 lun 0 Dec 9 00:24:30 r730-01 kernel: da12:Fixed Direct Access SPC-4 SCSI device Dec 9 00:24:30 r730-01 kernel: da12: Serial Number S757NS0Y700760R Dec 9 00:24:30 r730-01 kernel: da12: 150.000MB/s transfers Dec 9 00:24:30 r730-01 kernel: da12: 3815447MB (7814037168 512 byte sectors) Dec 9 00:24:30 r730-01 kernel: da12: quirks=0x8<4K> Dec 9 00:24:30 r730-01 kernel: da13 at mrsas0 bus 1 scbus1 target 12 lun 0 Dec 9 00:24:30 r730-01 kernel: da13: Fixed Direct Access SPC-4 SCSI device Dec 9 00:24:30 r730-01 kernel: da13: Serial Number S757NS0Y700758M Dec 9 00:24:30 r730-01 kernel: da13: 150.000MB/s transfers Dec 9 00:24:30 r730-01 kernel: da13: 3815447MB (7814037168 512 byte sectors) Dec 9 00:24:30 r730-01 kernel: da13: quirks=0x8<4K>
That makes the list now: disks=”nda0 nda1 nda2 nda3 nda5 nda8 da12 da13″
Oh, but two of those disks are involved in an exist zpool: data02
[0:36 r730-01 dvl ~/tmp] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 5.81T 5.10T 731G - - 3% 87% 1.00x ONLINE - data02 3.62T 688G 2.95T - - 33% 18% 1.00x ONLINE - data03 7.25T 1.28T 5.97T - - 32% 17% 1.00x ONLINE - zroot 107G 30.1G 76.9G - - 15% 28% 1.00x ONLINE -
It has 688G on it. However, it’s the main jail zpool.
[0:50 r730-01 dvl ~/tmp] % zfs list -d 2 -r data02 NAME USED AVAIL REFER MOUNTPOINT data02 868G 2.66T 96K none data02/freshports 307G 2.66T 88K none data02/freshports/dev-ingress01 244G 2.66T 88K none data02/freshports/dev-nginx01 55.9M 2.66T 96K none data02/freshports/dvl-ingress01 18.6G 2.66T 96K none data02/freshports/dvl-nginx01 23.4M 2.66T 96K none data02/freshports/jailed 449M 2.66T 96K none data02/freshports/stage-ingress01 19.6G 2.66T 96K none data02/freshports/test-ingress01 24.2G 2.66T 96K none data02/jails 309G 2.66T 9.54G /jails data02/jails/bacula 16.9G 2.66T 16.3G /jails/bacula data02/jails/bacula-sd-02 3.45G 2.66T 2.84G /jails/bacula-sd-02 data02/jails/bacula-sd-03 4.85G 2.66T 4.25G /jails/bacula-sd-03 data02/jails/besser 5.63G 2.66T 4.21G /jails/besser data02/jails/certs 3.02G 2.66T 2.42G /jails/certs data02/jails/certs-rsync 3.00G 2.66T 2.42G /jails/certs-rsync data02/jails/cliff2 3.05G 2.66T 2.47G /jails/cliff2 data02/jails/dev-ingress01 4.62G 2.66T 3.59G /jails/dev-ingress01 data02/jails/dev-nginx01 4.47G 2.66T 3.44G /jails/dev-nginx01 data02/jails/dev-pgeu 7.62G 2.66T 7.62G /jails/dev-pgeu data02/jails/dns-hidden-master 3.21G 2.66T 2.57G /jails/dns-hidden-master data02/jails/dns1 11.0G 2.66T 4.95G /jails/dns1 data02/jails/dvl-ingress01 7.81G 2.66T 5.28G /jails/dvl-ingress01 data02/jails/dvl-nginx01 2.15G 2.66T 1.17G /jails/dvl-nginx01 data02/jails/freebsd143 501M 2.66T 501M /jails/freebsd143 data02/jails/git 4.94G 2.66T 4.34G /jails/git data02/jails/jail_within_jail 1.28G 2.66T 750M /jails/jail_within_jail data02/jails/mqtt01 3.60G 2.66T 3.00G /jails/mqtt01 data02/jails/mydev 26.0G 2.66T 22.0G /jails/mydev data02/jails/mysql01 17.4G 2.66T 7.77G /jails/mysql01 data02/jails/nsnotify 3.28G 2.66T 2.61G /jails/nsnotify data02/jails/pg01 46.0G 2.66T 10.2G /jails/pg01 data02/jails/pg02 11.0G 2.66T 10.4G /jails/pg02 data02/jails/pg03 11.3G 2.66T 9.41G /jails/pg03 data02/jails/pkg01 16.9G 2.66T 13.8G /jails/pkg01 data02/jails/samdrucker 4.88G 2.66T 4.27G /jails/samdrucker data02/jails/serpico 3.10G 2.66T 2.49G /jails/serpico data02/jails/stage-ingress01 5.21G 2.66T 2.89G /jails/stage-ingress01 data02/jails/stage-nginx01 2.44G 2.66T 1.38G /jails/stage-nginx01 data02/jails/svn 10.5G 2.66T 9.77G /jails/svn data02/jails/talos 2.98G 2.66T 2.40G /jails/talos data02/jails/test-ingress01 2.31G 2.66T 1.35G /jails/test-ingress01 data02/jails/test-nginx01 2.38G 2.66T 1.30G /jails/test-nginx01 data02/jails/unifi01 30.2G 2.66T 12.2G /jails/unifi01 data02/jails/webserver 12.7G 2.66T 11.5G /jails/webserver data02/reserved 180G 2.84T 96K none data02/vm 70.8G 2.66T 7.24G /usr/local/vm data02/vm/freebsd-test 743M 2.66T 112K /usr/local/vm/freebsd-test data02/vm/hass 59.1G 2.66T 13.4G /usr/local/vm/hass data02/vm/home-assistant 351M 2.66T 351M /usr/local/vm/home-assistant data02/vm/myguest 3.38G 2.66T 3.38G /usr/local/vm/myguest
It seems I should move this elsewhere first. I know nda6 and nda7 are 1TB.
[0:31 r730-01 dvl ~/tmp] % sudo diskinfo -v /dev/nda6 /dev/nda6 512 # sectorsize 1000204886016 # mediasize in bytes (932G) 1953525168 # mediasize in sectors 0 # stripesize 0 # stripeoffset Samsung SSD 980 PRO with Heatsink 1TB # Disk descr. S6WSNJ0T208743F # Disk ident. nvme6 # Attachment Yes # TRIM/UNMAP support 0 # Rotation rate in RPM [0:31 r730-01 dvl ~/tmp] % sudo diskinfo -v /dev/nda7 /dev/nda7 512 # sectorsize 1000204886016 # mediasize in bytes (932G) 1953525168 # mediasize in sectors 0 # stripesize 0 # stripeoffset Samsung SSD 980 PRO with Heatsink 1TB # Disk descr. S6WSNJ0T207774T # Disk ident. nvme7 # Attachment Yes # TRIM/UNMAP support 0 # Rotation rate in RPM
I create the new zpool:
[0:54 r730-01 dvl ~] % sudo gpart create -s gpt nda6
nda6 created
[0:54 r730-01 dvl ~] % sudo gpart create -s gpt nda7
nda7 created
[0:56 r730-01 dvl ~] % sudo gpart add -i 1 -t freebsd-zfs -l S6WSNJ0T208743F nda6
nda6p1 added
[0:56 r730-01 dvl ~] % sudo gpart add -i 1 -t freebsd-zfs -l S6WSNJ0T207774T nda7
nda7p1 added
[0:56 r730-01 dvl ~] % sudo zpool create data02_new mirror /dev/gpt/S6WSNJ0T208743F /dev/gpt/S6WSNJ0T207774T
[0:57 r730-01 dvl ~] %
[0:57 r730-01 dvl ~] % zpool list data02_new
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
data02_new 928G 372K 928G - - 0% 0% 1.00x ONLINE -
[0:57 r730-01 dvl ~] % zpool status data02_new
pool: data02_new
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
data02_new ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/S6WSNJ0T208743F ONLINE 0 0 0
gpt/S6WSNJ0T207774T ONLINE 0 0 0
errors: No known data errors
[0:58 r730-01 dvl ~] % sudo zpool scrub data02_new
[0:58 r730-01 dvl ~] % gpart show nda6 nda7
=> 40 1953525088 nda6 GPT (932G)
40 1953525088 1 freebsd-zfs (932G)
=> 40 1953525088 nda7 GPT (932G)
40 1953525088 1 freebsd-zfs (932G)
[0:58 r730-01 dvl ~] %
I ran the scrub so that my Nagios monitoring does not warn me there is no recent scrub on that zpool.
Copying the data
Now I want to copy all the data from data02 into data02_new so I can repurpose those drives into an 8x 4T zpool array.
I’m heading to bed. I’ll come back to this another day.
Tuesday – the initial send | recv
I do a snapshot, then send | receive. I thought about doing this from a live-boot off a thumb drive.
Or, I could just do it now:
[16:24 r730-01 dvl ~] % sudo service jail stop Stopping jails: unifi01 nsnotify dns-hidden-master serpico jail_within_jail mydev bacula-sd-03 samdrucker talos bacula-sd-02 mqtt01 webserver svn git certs certs-rsync besser bacula stage-nginx01 stage-ingress01 test-nginx01 test-ingress01 dvl-nginx01 dvl-ingress01 dev-nginx01 dev-ingress01 pkg01 pg03 pg02 pg01 mysql01 cliff2 dns1. [16:26 r730-01 dvl ~] % sudo vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE freebsd-test default bhyveload 1 256M - No Stopped hass default uefi 4 8GB - Yes [1] Running (2133) home-assistant default uefi 1 1GB - No Stopped myguest default bhyveload 1 768M - No Stopped [16:26 r730-01 dvl ~] % sudo vm stop hass Sending ACPI shutdown to hass [16:26 r730-01 dvl ~] %
Here goes the snapshot and the send | recv:
root@r730-01:/home/dvl # zfs snapshot -r data02@for-transfer-to-data02_new
root@r730-01:/home/dvl # time zfs send -Rv data02@for-transfer-to-data02_new | zfs receive -uFv data02_new
full send of data02@delete-later-full2 estimated size is 43.1K
send from @delete-later-full2 to data02@delete-later-full3 estimated size is 624B
send from @delete-later-full3 to data02@for-transfer-to-data02_new estimated size is 624B
full send of data02/reserved@delete-later-full2 estimated size is 43.1K
send from @delete-later-full2 to data02/reserved@delete-later-full3 estimated size is 624B
send from @delete-later-full3 to data02/reserved@for-transfer-to-data02_new estimated size is 624B
full send of data02/freshports@delete-later-full2 estimated size is 39.3K
send from @delete-later-full2 to data02/freshports@delete-later-full3 estimated size is 624B
send from @delete-later-full3 to data02/freshports@for-transfer-to-data02_new estimated size is 624B
full send of data02/freshports/dev-ingress01@delete-later-full2 estimated size is 39.3K
send from @delete-later-full2 to data02/freshports/dev-ingress01@delete-later-full3 estimated size is 624B
...
TIME SENT SNAPSHOT data02/vm/hass@for-transfer-to-data02_new
received 248M stream in 0.45 seconds (553M/sec)
receiving incremental stream of data02/vm/hass@autosnap_2025-12-09_16:30:02_frequently into data02_new/vm/hass@autosnap_2025-12-09_16:30:02_frequently
received 312B stream in 0.02 seconds (13.0K/sec)
receiving incremental stream of data02/vm/hass@for-transfer-to-data02_new into data02_new/vm/hass@for-transfer-to-data02_new
received 312B stream in 0.02 seconds (13.2K/sec)
1393.06 real 0.76 user 847.86 sys
root@r730-01:/home/dvl #
A whole lot of export attempts and umounts
The following issues are a good view into why jails should mount and umount everything they need on start and stop, respectively. If I had that in place, the following would not have taken so long:
root@r730-01:/home/dvl # zpool export data02 cannot unmount '/jails/svn': pool or dataset is busy root@r730-01:/home/dvl # mount | grep svn data02/jails/svn on /jails/svn (zfs, local, noatime, nfsv4acls) data03/repos/subversion on /jails/svn/usr/local/svn/repos (zfs, local, noatime, nfsv4acls) root@r730-01:/home/dvl # zfs umount data03/repos/subversion root@r730-01:/home/dvl # zpool export data02 cannot unmount '/jails/pkg01': pool or dataset is busy root@r730-01:/home/dvl # mount | grep pkg01 data02/jails/pkg01 on /jails/pkg01 (zfs, local, noatime, nfsv4acls) data03/poudriere/ccache on /jails/pkg01/var/ccache (zfs, local, noatime, nfsv4acls) data03/poudriere/distfiles on /jails/pkg01/usr/ports/distfiles (zfs, local, noatime, nfsv4acls) data03/poudriere on /jails/pkg01/usr/local/poudriere (zfs, local, noatime, nfsv4acls) data03/poudriere/data on /jails/pkg01/usr/local/poudriere/data (zfs, local, noatime, nfsv4acls) data03/poudriere/jails on /jails/pkg01/usr/local/poudriere/jails (zfs, local, noatime, nfsv4acls) data03/poudriere/test on /jails/pkg01/usr/local/poudriere/test (zfs, local, noatime, nfsv4acls) data03/poudriere/ports on /jails/pkg01/usr/local/poudriere/ports (zfs, local, noatime, nfsv4acls) data03/poudriere/jails/143amd64 on /jails/pkg01/usr/local/poudriere/jails/143amd64 (zfs, local, noatime, nfsv4acls) data03/poudriere/data/cronjob-logs on /jails/pkg01/usr/local/poudriere/data/cronjob-logs (zfs, local, noatime, nfsv4acls) data03/poudriere/data/cache on /jails/pkg01/usr/local/poudriere/data/cache (zfs, local, noatime, nfsv4acls) data03/poudriere/data/packages on /jails/pkg01/usr/local/poudriere/data/packages (zfs, local, noatime, nfsv4acls) data03/poudriere/jails/142amd64 on /jails/pkg01/usr/local/poudriere/jails/142amd64 (zfs, local, noatime, nfsv4acls) data03/poudriere/ports/default on /jails/pkg01/usr/local/poudriere/ports/default (zfs, local, noatime, nfsv4acls) data03/poudriere/ports/testing on /jails/pkg01/usr/local/poudriere/ports/testing (zfs, local, noatime, nfsv4acls) data03/poudriere/ports/main on /jails/pkg01/usr/local/poudriere/ports/main (zfs, local, noatime, nfsv4acls) data03/poudriere/ports/pgeu_system on /jails/pkg01/usr/local/poudriere/ports/pgeu_system (zfs, local, noatime, nfsv4acls) root@r730-01:/home/dvl # zfs list -r data03 NAME USED AVAIL REFER MOUNTPOINT data03 1.64T 5.48T 96K none data03/acme-certs 11.9M 5.48T 96K none data03/acme-certs/certs 6.58M 5.48T 5.67M /jails/certs/var/db/acme/certs data03/acme-certs/certs-for-rsync 5.26M 5.48T 4.03M /jails/certs/var/db/certs-for-rsync data03/dvl 120K 5.48T 120K /jails/dev-ingress01/usr/home/dvl/src data03/jail_within_jail 1.08G 5.48T 96K none data03/jail_within_jail/jails 1.08G 5.48T 96K none data03/jail_within_jail/jails/freshports 1.08G 5.48T 1.08G none data03/librenms-rrd 117G 5.48T 25.5G /jails/besser/var/db/librenms/rrd data03/pg01 71.2G 5.48T 96K none data03/pg01/freshports.dvl 37.1G 5.48T 27.6G /jails/pg01/var/db/postgres.freshports.dvl data03/pg01/postgres 34.1G 5.48T 27.6G /jails/pg01/var/db/postgres data03/pg02 78.5G 5.48T 88K none data03/pg02/postgres 78.5G 5.48T 51.8G /jails/pg02/var/db/postgres data03/pg02/rsyncer 1.02M 5.48T 144K /jails/pg02/usr/home/rsyncer/backups data03/pg03 772G 5.48T 88K none data03/pg03/postgres 573G 5.48T 550G /jails/pg03/var/db/postgres data03/pg03/rsyncer 199G 5.48T 33.2G /jails/pg03/usr/home/rsyncer/backups data03/poudriere 245G 5.48T 96K /usr/local/poudriere data03/poudriere/ccache 16.5M 5.48T 104K /var/ccache data03/poudriere/ccache/ccache.13amd64 96K 5.48T 96K /var/ccache/ccache.13amd64 data03/poudriere/ccache/ccache.amd64 16.3M 5.48T 16.3M /var/ccache/ccache.amd64 data03/poudriere/data 93.1G 5.48T 28.1G /usr/local/poudriere/data data03/poudriere/data/cache 211M 5.48T 174M /usr/local/poudriere/data/cache data03/poudriere/data/cronjob-logs 4.01M 5.48T 1.61M /usr/local/poudriere/data/cronjob-logs data03/poudriere/data/packages 64.6G 5.48T 59.0G /usr/local/poudriere/data/packages data03/poudriere/distfiles 132G 5.48T 132G /usr/ports/distfiles data03/poudriere/jails 2.75G 5.48T 88K /usr/local/poudriere/jails data03/poudriere/jails/142amd64 1.37G 5.48T 1.37G /usr/local/poudriere/jails/142amd64 data03/poudriere/jails/143amd64 1.38G 5.48T 1.38G /usr/local/poudriere/jails/143amd64 data03/poudriere/ports 16.4G 5.48T 3.25G /usr/local/poudriere/ports data03/poudriere/ports/default 2.97G 5.48T 2.77G /usr/local/poudriere/ports/default data03/poudriere/ports/main 1.60G 5.48T 1.60G /usr/local/poudriere/ports/main data03/poudriere/ports/pgeu_system 1.06G 5.48T 1.06G /usr/local/poudriere/ports/pgeu_system data03/poudriere/ports/testing 7.02G 5.48T 2.71G /usr/local/poudriere/ports/testing data03/poudriere/test 96K 5.48T 96K /usr/local/poudriere/test data03/public 23.4G 5.48T 23.4G none data03/repos 9.05G 5.48T 88K none data03/repos/gitea 812M 5.48T 308M /jails/git/var/db/gitea data03/repos/subversion 8.25G 5.48T 7.75G /jails/svn/usr/local/svn/repos data03/reserved 362G 5.84T 96K none root@r730-01:/home/dvl # zpool export data03 root@r730-01:/home/dvl # zpool export data02 cannot unmount '/jails/dev-ingress01/jails/freshports': pool or dataset is busy root@r730-01:/home/dvl # mount | grep /jails/dev-ingress01/jails/freshports data02/freshports/dev-ingress01/jails/freshports on /jails/dev-ingress01/jails/freshports (zfs, local, noatime, nfsv4acls) devfs on /jails/dev-ingress01/jails/freshports/dev (devfs) root@r730-01:/home/dvl # umount /jails/dev-ingress01/jails/freshports/dev root@r730-01:/home/dvl # mount | grep /dev devfs on /dev (devfs) data02/jails/dev-ingress01 on /jails/dev-ingress01 (zfs, local, noatime, nfsv4acls) data02/freshports/dev-ingress01/jails on /jails/dev-ingress01/jails (zfs, local, noatime, nfsv4acls) data02/freshports/dev-ingress01/jails/freshports on /jails/dev-ingress01/jails/freshports (zfs, local, noatime, nfsv4acls) root@r730-01:/home/dvl # zpool export data02 cannot unmount '/jails/bacula-sd-03': pool or dataset is busy root@r730-01:/home/dvl # mount | grep /jails/bacula-sd-03 data02/jails/bacula-sd-03 on /jails/bacula-sd-03 (zfs, local, noatime, nfsv4acls) data01/bacula on /jails/bacula-sd-03/usr/local/bacula (zfs, local, noatime, nfsv4acls) data01/bacula/working on /jails/bacula-sd-03/usr/local/bacula/working (zfs, local, noatime, nfsv4acls) data01/bacula/volumes on /jails/bacula-sd-03/usr/local/bacula/volumes (zfs, local, noatime, nfsv4acls) data01/bacula/volumes/DiffFile-03 on /jails/bacula-sd-03/usr/local/bacula/volumes/DiffFile-03 (zfs, local, noatime, nfsv4acls) data01/bacula/volumes/IncrFile-03 on /jails/bacula-sd-03/usr/local/bacula/volumes/IncrFile-03 (zfs, local, noatime, nfsv4acls) data01/bacula/volumes/FullFile-03 on /jails/bacula-sd-03/usr/local/bacula/volumes/FullFile-03 (zfs, local, noatime, nfsv4acls) root@r730-01:/home/dvl # zpool export data01 root@r730-01:/home/dvl # zpool export data02 root@r730-01:/home/dvl #
Renaming
This renames data02:
root@r730-01:/home/dvl # zpool import data02 data02_old root@r730-01:/home/dvl # zpool export data02_old root@r730-01:/home/dvl #
Now let’s check mount points:
root@r730-01:/home/dvl # zfs get -t filesystem -r mountpoint data02 NAME PROPERTY VALUE SOURCE data02 mountpoint none received data02/freshports mountpoint none inherited from data02 data02/freshports/dev-ingress01 mountpoint none received data02/freshports/dev-ingress01/dvl-src mountpoint /jails/dev-ingress01/usr/home/dvl/src received data02/freshports/dev-ingress01/freshports mountpoint /jails/dev-ingress01/var/db/freshports received data02/freshports/dev-ingress01/freshports/cache mountpoint /jails/dev-ingress01/var/db/freshports/cache inherited from data02/freshports/dev-ingress01/freshports data02/freshports/dev-ingress01/freshports/cache/html mountpoint /jails/dev-ingress01/var/db/freshports/cache/html inherited from data02/freshports/dev-ingress01/freshports data02/freshports/dev-ingress01/freshports/cache/spooling mountpoint /jails/dev-ingress01/var/db/freshports/cache/spooling inherited from data02/freshports/dev-ingress01/freshports data02/freshports/dev-ingress01/freshports/message-queues mountpoint /jails/dev-ingress01/var/db/freshports/message-queues received data02/freshports/dev-ingress01/freshports/message-queues/archive mountpoint /jails/dev-ingress01/var/db/freshports/message-queues/archive inherited from data02/freshports/dev-ingress01/freshports/message-queues data02/freshports/dev-ingress01/ingress mountpoint /jails/dev-ingress01/var/db/ingress received data02/freshports/dev-ingress01/ingress/latest_commits mountpoint /jails/dev-ingress01/var/db/ingress/latest_commits inherited from data02/freshports/dev-ingress01/ingress data02/freshports/dev-ingress01/ingress/message-queues mountpoint /jails/dev-ingress01/var/db/ingress/message-queues inherited from data02/freshports/dev-ingress01/ingress data02/freshports/dev-ingress01/ingress/repos mountpoint /jails/dev-ingress01/var/db/ingress/repos inherited from data02/freshports/dev-ingress01/ingress data02/freshports/dev-ingress01/ingress/repos/doc mountpoint /jails/dev-ingress01/var/db/ingress/repos/doc inherited from data02/freshports/dev-ingress01/ingress data02/freshports/dev-ingress01/ingress/repos/ports mountpoint /jails/dev-ingress01/var/db/ingress/repos/ports inherited from data02/freshports/dev-ingress01/ingress data02/freshports/dev-ingress01/ingress/repos/src mountpoint /jails/dev-ingress01/var/db/ingress/repos/src inherited from data02/freshports/dev-ingress01/ingress data02/freshports/dev-ingress01/jails mountpoint /jails/dev-ingress01/jails received data02/freshports/dev-ingress01/jails/freshports mountpoint /jails/dev-ingress01/jails/freshports inherited from data02/freshports/dev-ingress01/jails data02/freshports/dev-ingress01/jails/freshports/ports mountpoint /jails/dev-ingress01/jails/freshports/usr/ports received data02/freshports/dev-ingress01/modules mountpoint /jails/dev-ingress01/usr/local/lib/perl5/site_perl/FreshPorts received data02/freshports/dev-ingress01/scripts mountpoint /jails/dev-ingress01/usr/local/libexec/freshports received data02/freshports/dev-nginx01 mountpoint none inherited from data02 data02/freshports/dev-nginx01/www mountpoint /jails/dev-nginx01/usr/local/www received data02/freshports/dev-nginx01/www/freshports mountpoint /jails/dev-nginx01/usr/local/www/freshports inherited from data02/freshports/dev-nginx01/www data02/freshports/dev-nginx01/www/freshsource mountpoint /jails/dev-nginx01/usr/local/www/freshsource inherited from data02/freshports/dev-nginx01/www data02/freshports/dvl-ingress01 mountpoint none received data02/freshports/dvl-ingress01/dvl-src mountpoint /jails/dvl-ingress01/usr/home/dvl/src received data02/freshports/dvl-ingress01/freshports mountpoint /jails/dvl-ingress01/var/db/freshports received data02/freshports/dvl-ingress01/freshports/cache mountpoint /jails/dvl-ingress01/var/db/freshports/cache inherited from data02/freshports/dvl-ingress01/freshports data02/freshports/dvl-ingress01/freshports/cache/html mountpoint /jails/dvl-ingress01/var/db/freshports/cache/html inherited from data02/freshports/dvl-ingress01/freshports data02/freshports/dvl-ingress01/freshports/cache/spooling mountpoint /jails/dvl-ingress01/var/db/freshports/cache/spooling inherited from data02/freshports/dvl-ingress01/freshports data02/freshports/dvl-ingress01/freshports/message-queues mountpoint /jails/dvl-ingress01/var/db/freshports/message-queues inherited from data02/freshports/dvl-ingress01/freshports data02/freshports/dvl-ingress01/freshports/message-queues/archive mountpoint /jails/dvl-ingress01/var/db/freshports/message-queues/archive inherited from data02/freshports/dvl-ingress01/freshports data02/freshports/dvl-ingress01/ingress mountpoint /jails/dvl-ingress01/var/db/ingress received data02/freshports/dvl-ingress01/ingress/latest_commits mountpoint /jails/dvl-ingress01/var/db/ingress/latest_commits inherited from data02/freshports/dvl-ingress01/ingress data02/freshports/dvl-ingress01/ingress/message-queues mountpoint /jails/dvl-ingress01/var/db/ingress/message-queues inherited from data02/freshports/dvl-ingress01/ingress data02/freshports/dvl-ingress01/ingress/repos mountpoint /jails/dvl-ingress01/var/db/ingress/repos inherited from data02/freshports/dvl-ingress01/ingress data02/freshports/dvl-ingress01/ingress/repos/doc mountpoint /jails/dvl-ingress01/var/db/ingress/repos/doc inherited from data02/freshports/dvl-ingress01/ingress data02/freshports/dvl-ingress01/ingress/repos/ports mountpoint /jails/dvl-ingress01/var/db/ingress/repos/ports inherited from data02/freshports/dvl-ingress01/ingress data02/freshports/dvl-ingress01/ingress/repos/src mountpoint /jails/dvl-ingress01/var/db/ingress/repos/src inherited from data02/freshports/dvl-ingress01/ingress data02/freshports/dvl-ingress01/jails mountpoint /jails/dvl-ingress01/jails received data02/freshports/dvl-ingress01/jails/freshports mountpoint /jails/dvl-ingress01/jails/freshports inherited from data02/freshports/dvl-ingress01/jails data02/freshports/dvl-ingress01/jails/freshports/ports mountpoint /jails/dvl-ingress01/jails/freshports/usr/ports received data02/freshports/dvl-ingress01/modules mountpoint /jails/dvl-ingress01/usr/local/lib/perl5/site_perl/FreshPorts received data02/freshports/dvl-ingress01/scripts mountpoint /jails/dvl-ingress01/usr/local/libexec/freshports received data02/freshports/dvl-nginx01 mountpoint none inherited from data02 data02/freshports/dvl-nginx01/www mountpoint none inherited from data02 data02/freshports/dvl-nginx01/www/freshports mountpoint /jails/dvl-nginx01/usr/local/www/freshports received data02/freshports/dvl-nginx01/www/freshsource mountpoint /jails/dvl-nginx01/usr/local/www/freshsource received data02/freshports/jailed mountpoint none inherited from data02 data02/freshports/jailed/dev-ingress01 mountpoint none inherited from data02 data02/freshports/jailed/dev-nginx01 mountpoint none inherited from data02 data02/freshports/jailed/dev-nginx01/cache mountpoint /var/db/freshports/cache received data02/freshports/jailed/dev-nginx01/cache/categories mountpoint /var/db/freshports/cache/categories inherited from data02/freshports/jailed/dev-nginx01/cache data02/freshports/jailed/dev-nginx01/cache/commits mountpoint /var/db/freshports/cache/commits inherited from data02/freshports/jailed/dev-nginx01/cache data02/freshports/jailed/dev-nginx01/cache/daily mountpoint /var/db/freshports/cache/daily inherited from data02/freshports/jailed/dev-nginx01/cache data02/freshports/jailed/dev-nginx01/cache/general mountpoint /var/db/freshports/cache/general inherited from data02/freshports/jailed/dev-nginx01/cache data02/freshports/jailed/dev-nginx01/cache/news mountpoint /var/db/freshports/cache/news inherited from data02/freshports/jailed/dev-nginx01/cache data02/freshports/jailed/dev-nginx01/cache/packages mountpoint /var/db/freshports/cache/packages inherited from data02/freshports/jailed/dev-nginx01/cache data02/freshports/jailed/dev-nginx01/cache/pages mountpoint /var/db/freshports/cache/pages inherited from data02/freshports/jailed/dev-nginx01/cache data02/freshports/jailed/dev-nginx01/cache/ports mountpoint /var/db/freshports/cache/ports inherited from data02/freshports/jailed/dev-nginx01/cache data02/freshports/jailed/dev-nginx01/cache/spooling mountpoint /var/db/freshports/cache/spooling inherited from data02/freshports/jailed/dev-nginx01/cache data02/freshports/jailed/dvl-ingress01 mountpoint none received data02/freshports/jailed/dvl-ingress01/distfiles mountpoint none inherited from data02/freshports/jailed/dvl-ingress01 data02/freshports/jailed/dvl-nginx01 mountpoint none inherited from data02 data02/freshports/jailed/dvl-nginx01/cache mountpoint /var/db/freshports/cache received data02/freshports/jailed/dvl-nginx01/cache/categories mountpoint /var/db/freshports/cache/categories inherited from data02/freshports/jailed/dvl-nginx01/cache data02/freshports/jailed/dvl-nginx01/cache/commits mountpoint /var/db/freshports/cache/commits inherited from data02/freshports/jailed/dvl-nginx01/cache data02/freshports/jailed/dvl-nginx01/cache/daily mountpoint /var/db/freshports/cache/daily inherited from data02/freshports/jailed/dvl-nginx01/cache data02/freshports/jailed/dvl-nginx01/cache/general mountpoint /var/db/freshports/cache/general inherited from data02/freshports/jailed/dvl-nginx01/cache data02/freshports/jailed/dvl-nginx01/cache/news mountpoint /var/db/freshports/cache/news inherited from data02/freshports/jailed/dvl-nginx01/cache data02/freshports/jailed/dvl-nginx01/cache/packages mountpoint /var/db/freshports/cache/packages inherited from data02/freshports/jailed/dvl-nginx01/cache data02/freshports/jailed/dvl-nginx01/cache/pages mountpoint /var/db/freshports/cache/pages inherited from data02/freshports/jailed/dvl-nginx01/cache data02/freshports/jailed/dvl-nginx01/cache/ports mountpoint /var/db/freshports/cache/ports inherited from data02/freshports/jailed/dvl-nginx01/cache data02/freshports/jailed/dvl-nginx01/cache/spooling mountpoint /var/db/freshports/cache/spooling inherited from data02/freshports/jailed/dvl-nginx01/cache data02/freshports/jailed/dvl-nginx01/freshports mountpoint none inherited from data02 data02/freshports/jailed/stage-ingress01 mountpoint none inherited from data02 data02/freshports/jailed/stage-ingress01/data mountpoint none inherited from data02 data02/freshports/jailed/stage-nginx01 mountpoint none inherited from data02 data02/freshports/jailed/stage-nginx01/cache mountpoint /var/db/freshports/cache received data02/freshports/jailed/stage-nginx01/cache/categories mountpoint /var/db/freshports/cache/categories inherited from data02/freshports/jailed/stage-nginx01/cache data02/freshports/jailed/stage-nginx01/cache/commits mountpoint /var/db/freshports/cache/commits inherited from data02/freshports/jailed/stage-nginx01/cache data02/freshports/jailed/stage-nginx01/cache/daily mountpoint /var/db/freshports/cache/daily inherited from data02/freshports/jailed/stage-nginx01/cache data02/freshports/jailed/stage-nginx01/cache/general mountpoint /var/db/freshports/cache/general inherited from data02/freshports/jailed/stage-nginx01/cache data02/freshports/jailed/stage-nginx01/cache/news mountpoint /var/db/freshports/cache/news inherited from data02/freshports/jailed/stage-nginx01/cache data02/freshports/jailed/stage-nginx01/cache/packages mountpoint /var/db/freshports/cache/packages inherited from data02/freshports/jailed/stage-nginx01/cache data02/freshports/jailed/stage-nginx01/cache/pages mountpoint /var/db/freshports/cache/pages inherited from data02/freshports/jailed/stage-nginx01/cache data02/freshports/jailed/stage-nginx01/cache/ports mountpoint /var/db/freshports/cache/ports inherited from data02/freshports/jailed/stage-nginx01/cache data02/freshports/jailed/stage-nginx01/cache/spooling mountpoint /var/db/freshports/cache/spooling inherited from data02/freshports/jailed/stage-nginx01/cache data02/freshports/jailed/test-ingress01 mountpoint none inherited from data02 data02/freshports/jailed/test-ingress01/data mountpoint none inherited from data02 data02/freshports/jailed/test-nginx01 mountpoint none inherited from data02 data02/freshports/jailed/test-nginx01/cache mountpoint /var/db/freshports/cache received data02/freshports/jailed/test-nginx01/cache/categories mountpoint /var/db/freshports/cache/categories inherited from data02/freshports/jailed/test-nginx01/cache data02/freshports/jailed/test-nginx01/cache/commits mountpoint /var/db/freshports/cache/commits inherited from data02/freshports/jailed/test-nginx01/cache data02/freshports/jailed/test-nginx01/cache/daily mountpoint /var/db/freshports/cache/daily inherited from data02/freshports/jailed/test-nginx01/cache data02/freshports/jailed/test-nginx01/cache/general mountpoint /var/db/freshports/cache/general inherited from data02/freshports/jailed/test-nginx01/cache data02/freshports/jailed/test-nginx01/cache/news mountpoint /var/db/freshports/cache/news inherited from data02/freshports/jailed/test-nginx01/cache data02/freshports/jailed/test-nginx01/cache/packages mountpoint /var/db/freshports/cache/packages inherited from data02/freshports/jailed/test-nginx01/cache data02/freshports/jailed/test-nginx01/cache/pages mountpoint /var/db/freshports/cache/pages inherited from data02/freshports/jailed/test-nginx01/cache data02/freshports/jailed/test-nginx01/cache/ports mountpoint /var/db/freshports/cache/ports inherited from data02/freshports/jailed/test-nginx01/cache data02/freshports/jailed/test-nginx01/cache/spooling mountpoint /var/db/freshports/cache/spooling inherited from data02/freshports/jailed/test-nginx01/cache data02/freshports/stage-ingress01 mountpoint none inherited from data02 data02/freshports/stage-ingress01/cache mountpoint /jails/stage-ingress01/var/db/freshports/cache received data02/freshports/stage-ingress01/cache/html mountpoint /jails/stage-ingress01/var/db/freshports/cache/html inherited from data02/freshports/stage-ingress01/cache data02/freshports/stage-ingress01/cache/spooling mountpoint /jails/stage-ingress01/var/db/freshports/cache/spooling inherited from data02/freshports/stage-ingress01/cache data02/freshports/stage-ingress01/freshports mountpoint none inherited from data02 data02/freshports/stage-ingress01/freshports/archive mountpoint /jails/stage-ingress01/var/db/freshports/message-queues/archive received data02/freshports/stage-ingress01/freshports/message-queues mountpoint /jails/stage-ingress01/var/db/freshports/message-queues received data02/freshports/stage-ingress01/ingress mountpoint /jails/stage-ingress01/var/db/ingress received data02/freshports/stage-ingress01/ingress/latest_commits mountpoint /jails/stage-ingress01/var/db/ingress/latest_commits inherited from data02/freshports/stage-ingress01/ingress data02/freshports/stage-ingress01/ingress/message-queues mountpoint /jails/stage-ingress01/var/db/ingress/message-queues inherited from data02/freshports/stage-ingress01/ingress data02/freshports/stage-ingress01/ingress/repos mountpoint /jails/stage-ingress01/var/db/ingress/repos inherited from data02/freshports/stage-ingress01/ingress data02/freshports/stage-ingress01/jails mountpoint /jails/stage-ingress01/jails received data02/freshports/stage-ingress01/jails/freshports mountpoint /jails/stage-ingress01/jails/freshports inherited from data02/freshports/stage-ingress01/jails data02/freshports/stage-ingress01/ports mountpoint /jails/stage-ingress01/jails/freshports/usr/ports received data02/freshports/test-ingress01 mountpoint none inherited from data02 data02/freshports/test-ingress01/freshports mountpoint /jails/test-ingress01/var/db/freshports received data02/freshports/test-ingress01/freshports/cache mountpoint /jails/test-ingress01/var/db/freshports/cache inherited from data02/freshports/test-ingress01/freshports data02/freshports/test-ingress01/freshports/cache/html mountpoint /jails/test-ingress01/var/db/freshports/cache/html inherited from data02/freshports/test-ingress01/freshports data02/freshports/test-ingress01/freshports/cache/spooling mountpoint /jails/test-ingress01/var/db/freshports/cache/spooling inherited from data02/freshports/test-ingress01/freshports data02/freshports/test-ingress01/freshports/message-queues mountpoint /jails/test-ingress01/var/db/freshports/message-queues inherited from data02/freshports/test-ingress01/freshports data02/freshports/test-ingress01/freshports/message-queues/archive mountpoint /jails/test-ingress01/var/db/freshports/message-queues/archive inherited from data02/freshports/test-ingress01/freshports data02/freshports/test-ingress01/ingress mountpoint /jails/test-ingress01/var/db/ingress received data02/freshports/test-ingress01/ingress/latest_commits mountpoint /jails/test-ingress01/var/db/ingress/latest_commits inherited from data02/freshports/test-ingress01/ingress data02/freshports/test-ingress01/ingress/message-queues mountpoint /jails/test-ingress01/var/db/ingress/message-queues inherited from data02/freshports/test-ingress01/ingress data02/freshports/test-ingress01/ingress/repos mountpoint /jails/test-ingress01/var/db/ingress/repos inherited from data02/freshports/test-ingress01/ingress data02/freshports/test-ingress01/jails mountpoint /jails/test-ingress01/jails received data02/freshports/test-ingress01/jails/freshports mountpoint /jails/test-ingress01/jails/freshports inherited from data02/freshports/test-ingress01/jails data02/freshports/test-ingress01/jails/freshports/ports mountpoint /jails/test-ingress01/jails/freshports/usr/ports received data02/jails mountpoint /jails received data02/jails/bacula mountpoint /jails/bacula inherited from data02/jails data02/jails/bacula-sd-02 mountpoint /jails/bacula-sd-02 inherited from data02/jails data02/jails/bacula-sd-03 mountpoint /jails/bacula-sd-03 inherited from data02/jails data02/jails/besser mountpoint /jails/besser inherited from data02/jails data02/jails/certs mountpoint /jails/certs inherited from data02/jails data02/jails/certs-rsync mountpoint /jails/certs-rsync inherited from data02/jails data02/jails/cliff2 mountpoint /jails/cliff2 inherited from data02/jails data02/jails/dev-ingress01 mountpoint /jails/dev-ingress01 inherited from data02/jails data02/jails/dev-nginx01 mountpoint /jails/dev-nginx01 inherited from data02/jails data02/jails/dev-pgeu mountpoint /jails/dev-pgeu inherited from data02/jails data02/jails/dns-hidden-master mountpoint /jails/dns-hidden-master inherited from data02/jails data02/jails/dns1 mountpoint /jails/dns1 inherited from data02/jails data02/jails/dvl-ingress01 mountpoint /jails/dvl-ingress01 inherited from data02/jails data02/jails/dvl-nginx01 mountpoint /jails/dvl-nginx01 inherited from data02/jails data02/jails/freebsd143 mountpoint /jails/freebsd143 inherited from data02/jails data02/jails/git mountpoint /jails/git inherited from data02/jails data02/jails/jail_within_jail mountpoint /jails/jail_within_jail inherited from data02/jails data02/jails/mqtt01 mountpoint /jails/mqtt01 inherited from data02/jails data02/jails/mydev mountpoint /jails/mydev inherited from data02/jails data02/jails/mysql01 mountpoint /jails/mysql01 inherited from data02/jails data02/jails/nsnotify mountpoint /jails/nsnotify inherited from data02/jails data02/jails/pg01 mountpoint /jails/pg01 inherited from data02/jails data02/jails/pg02 mountpoint /jails/pg02 inherited from data02/jails data02/jails/pg03 mountpoint /jails/pg03 inherited from data02/jails data02/jails/pkg01 mountpoint /jails/pkg01 inherited from data02/jails data02/jails/samdrucker mountpoint /jails/samdrucker inherited from data02/jails data02/jails/serpico mountpoint /jails/serpico inherited from data02/jails data02/jails/stage-ingress01 mountpoint /jails/stage-ingress01 inherited from data02/jails data02/jails/stage-nginx01 mountpoint /jails/stage-nginx01 inherited from data02/jails data02/jails/svn mountpoint /jails/svn inherited from data02/jails data02/jails/talos mountpoint /jails/talos inherited from data02/jails data02/jails/test-ingress01 mountpoint /jails/test-ingress01 inherited from data02/jails data02/jails/test-nginx01 mountpoint /jails/test-nginx01 inherited from data02/jails data02/jails/unifi01 mountpoint /jails/unifi01 inherited from data02/jails data02/jails/webserver mountpoint /jails/webserver inherited from data02/jails data02/reserved mountpoint none inherited from data02 data02/vm mountpoint /usr/local/vm received data02/vm/freebsd-test mountpoint /usr/local/vm/freebsd-test inherited from data02/vm data02/vm/hass mountpoint /usr/local/vm/hass inherited from data02/vm data02/vm/home-assistant mountpoint /usr/local/vm/home-assistant inherited from data02/vm data02/vm/myguest mountpoint /usr/local/vm/myguest inherited from data02/vm
That looks good.
Next, import these zpools so they are present on the next reboot.
root@r730-01:/home/dvl # zpool import data01 root@r730-01:/home/dvl # zpool import data03
What zpool are not in use?
This lists the zpool which can be imported but are not.
root@r730-01:/home/dvl # zpool import
pool: data02_old
id: 14532602998618854058
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
data02_old ONLINE
mirror-0 ONLINE
gpt/Samsung_990_S7U8NJ0Y716854P ONLINE
gpt/Samsung_990_S7U8NJ0Y716801F ONLINE
pool: data04_old
id: 14613959245391720618
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
data04_old ONLINE
mirror-0 ONLINE
da13 ONLINE
da12 ONLINE
root@r730-01:/home/dvl #
All of those drives are destined to be used by the new big zpool.
Before rebooting
Before I restart, I disable these services. They are done now, after I’ve manually stopped them and before reboot.
root@r730-01:/home/dvl # sysrc jail_enable="NO" jail_enable: YES -> NO root@r730-01:/home/dvl # sysrc vm_enable="NO" vm_enable: YES -> NO
After I reboot the host and know that it looks good, I’ll enable those services, start them manually, and review. If all good, one more restart.
Here is where I tell you: shutdown -r now is not the same as reboot
reboot does not run your rc scripts.
Although I say reboot and restart, that is always accomplished via shutdown -r now
root@r730-01:/home/dvl # shutdown -r now
Shutdown NOW!
shutdown: [pid 52693]
root@r730-01:/home/dvl #
*** FINAL System shutdown message from dvl@r730-01.int.unixathome.org ***
System going down IMMEDIATELY
System shutdown time has arrived
Connection to r730-01.int.unixathome.org closed by remote host.
Connection to r730-01.int.unixathome.org closed.
After the reboot
Here we go, this looks good:
[17:29 r730-01 dvl ~] % uptime 5:29PM up 34 secs, 1 user, load averages: 0.33, 0.09, 0.03 [17:29 r730-01 dvl ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data01 5.81T 5.10T 731G - - 3% 87% 1.00x ONLINE - data02 928G 646G 282G - - 0% 69% 1.00x ONLINE - data03 7.25T 1.29T 5.96T - - 33% 17% 1.00x ONLINE - zroot 107G 30.2G 76.8G - - 15% 28% 1.00x ONLINE - [17:29 r730-01 dvl ~] %
Let’s start bhyve first:
[17:31 r730-01 dvl ~] % sudo sysrc vm_enable="YES" vm_enable: NO -> YES [17:31 r730-01 dvl ~] % sudo service vm start [17:31 r730-01 dvl ~] %
Yes, that looks good. On to the jails.
[17:31 r730-01 dvl ~] % sudo sysrc jail_enable="YES" jail_enable: NO -> YES [17:42 r730-01 dvl ~] % sudo service jail start Starting jails: dns1 cliff2 mysql01 pg01 pg02 pg03 pkg01 dev-ingress01 dev-nginx01 dvl-ingress01 dvl-nginx01 test-ingress01 test-nginx01 stage-ingress01 stage-nginx01 bacula besser certs-rsync certs git svn webserver mqtt01 bacula-sd-02 talos samdrucker bacula-sd-03 mydev jail_within_jail serpico dns-hidden-master nsnotify unifi01. [17:43 r730-01 dvl ~] %
And now I want little while for monitoring to catch up…
It’s now ten minutes later, and the only Nagios alerts are time related:
FILE_AGE WARNING: /var/db/ingress/latest_commits/latest_commits.txt is 6451 seconds old and 3662 bytes
Well, yeah, the host wasn’t up and running to do that…
I declare good-to-go.
Oh wait, I need to do a full proper reboot
sudo shutdown -r now
Yep, that went well. Now it’s time to create that new zpool
