I’m retiring knew and moving the services over to r730-03. I recently added 2 12TB drives to the latter host.
In this post, I’ll start the migration process by identifying what data needs to move, from where, and to where.
In this post:
The data
There are the zpools in question:
[15:19 knew dan ~] % zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT nvd 222G 46.9G 175G - - 45% 21% 1.00x ONLINE - system 90.5T 65.1T 25.4T - - 16% 71% 1.00x ONLINE - tank_fast01 928G 216G 712G - - 5% 23% 1.00x ONLINE - zroot 19.9G 13.6G 6.31G - - 63% 68% 1.00x ONLINE - [15:21 knew dan ~] %
A brief overview:
- nvd – mostly data for Bacula regression testing – the home directory where the compiled code occurs. Regression testing no longer occurs
because the project no longer hosts a cdash instance. This data still be copied over. - system – the bulk of the data, mostly backups. Some of this data will be copied over, but clearly not all of the 65TB can fit into the 12TB space available. More detail will appear below
- tank_fast01 – data for the database cloning operations. Mostly database backups, etc. This will be copied over.
- zroot – typical zroot installation – I’ll bring over the home directories only.
system
Here are all the filesystems of the system zpool.
system 49.6T 18.9T 320K none system/TRASH 219K 18.9T 219K none system/backups-by-snapshot 281G 18.9T 384K /usr/home/ansel/snapshots system/backups-by-snapshot/mailjail 9.37G 18.9T 9.37G /usr/home/ansel/snapshots/mailjail system/backups-by-snapshot/mounted 329K 18.9T 329K /usr/home/ansel/snapshots/mounted system/backups-by-snapshot/syncoid 272G 18.9T 219K /usr/home/ansel/snapshots/syncoid system/backups-by-snapshot/syncoid/homeassistant 124G 18.9T 32.0G /usr/home/ansel/snapshots/syncoid/homeassistant system/backups-by-snapshot/syncoid/homeassistant-r730-01 148G 18.9T 29.5G /usr/home/ansel/snapshots/syncoid/homeassistant-r730-01 system/backups-by-snapshot/testing 329K 18.9T 329K /usr/home/ansel/snapshots/testing system/data 49.1T 18.9T 329K none system/data/bacula 60.6M 18.9T 219K /jails/bacula-sd-01/usr/local/bacula system/data/bacula-restores 201K 18.9T 201K /tmp/bacula-restores system/data/bacula-volumes 44.8T 18.9T 219K /jails/bacula-sd-01/usr/local/bacula/volumes system/data/bacula-volumes/DailyBackups 238G 18.9T 238G /jails/bacula-sd-01/usr/local/bacula/volumes/DailyBackups system/data/bacula-volumes/DiffFile 1.13T 18.9T 876G /jails/bacula-sd-01/usr/local/bacula/volumes/DiffFile system/data/bacula-volumes/FullAutoNoNextPool 7.46T 18.9T 7.30T /jails/bacula-sd-01/usr/local/bacula/volumes/FullAutoNoNextPool system/data/bacula-volumes/FullFile 32.2T 18.9T 31.7T /jails/bacula-sd-01/usr/local/bacula/volumes/FullFile system/data/bacula-volumes/IncrFile 3.43T 18.9T 2.25T /jails/bacula-sd-01/usr/local/bacula/volumes/IncrFile system/data/bacula-volumes/IncrFileNoNextPool 193G 18.9T 193G /jails/bacula-sd-01/usr/local/bacula/volumes/IncrFileNoNextPool system/data/bacula-volumes/MonthlyBackups 108G 18.9T 108G /jails/bacula-sd-01/usr/local/bacula/volumes/MonthlyBackups system/data/bacula-volumes/TwoHourlyBackups 28.7M 18.9T 28.7M /jails/bacula-sd-01/usr/local/bacula/volumes/TwoHourlyBackups system/data/bacula-volumes/WeeklyBackups 66.0G 18.9T 66.0G /jails/bacula-sd-01/usr/local/bacula/volumes/WeeklyBackups system/data/bacula/spooling 329K 18.9T 201K /jails/bacula-sd-01/usr/local/bacula/spooling system/data/bacula/working 59.9M 18.9T 292K /jails/bacula-sd-01/usr/local/bacula/working system/data/downloads 1.82G 18.9T 1.82G /usr/home/dan/downloads system/data/freshports 6.66G 18.9T 219K /freshports system/data/freshports/archives 6.66G 18.9T 219K /freshports/archives system/data/freshports/archives/supernews 6.66G 18.9T 6.66G /freshports/archives/supernews system/data/snapshots 201K 18.9T 201K none system/data/testing-rsync 219K 18.9T 219K /jails/dbclone/usr/home/testrsync/backup-testing system/data/time_capsule 4.23T 18.9T 219K /data/time_capsule system/data/time_capsule/dvl-air01 1.17T 205G 995G /data/time_capsule/dvl-air01 system/data/time_capsule/dvl-dent 625G 575G 625G /data/time_capsule/dvl-dent system/data/time_capsule/dvl-dent-sparse 167G 153G 167G /data/time_capsule/dvl-dent-sparse system/data/time_capsule/dvl-pro02 1.27T 238G 962G /data/time_capsule/dvl-pro02 system/data/time_capsule/dvl-pro03 1.02T 18.9T 691G /data/time_capsule/dvl-pro03 system/jails 204G 18.9T 347K /jails system/jails/ansible 14.7G 18.9T 13.7G /jails/ansible system/jails/bacula-sd-01 10.8G 18.9T 10.5G /jails/bacula-sd-01 system/jails/cliff1 11.4G 18.9T 11.1G /jails/cliff1 system/jails/dbclone 19.5G 18.9T 18.1G /jails/dbclone system/jails/empty 13.7G 18.9T 6.67G /jails/empty system/jails/fruity-ext 11.0G 18.9T 10.7G /jails/fruity-ext system/jails/fruity-int 12.0G 18.9T 11.6G /jails/fruity-int system/jails/mysql57 13.6G 18.9T 13.6G /jails/mysql57 system/jails/mysql80 9.06G 18.9T 9.05G /jails/mysql80 system/jails/pg11 14.1G 18.9T 14.1G /jails/pg11 system/jails/pg12 13.6G 18.9T 13.6G /jails/pg12 system/jails/pg13 22.1G 18.9T 22.1G /jails/pg13 system/jails/pg14 25.4G 18.9T 25.4G /jails/pg14 system/jails/toiler 13.2G 18.9T 11.4G /jails/toiler system/mkjail 5.99G 18.9T 219K /mkjail system/mkjail/13.0-RELEASE 1.91G 18.9T 1.91G /mkjail/13.0-RELEASE system/mkjail/13.1-RELEASE 2.02G 18.9T 2.02G /mkjail/13.1-RELEASE system/mkjail/13.2-RELEASE 2.06G 18.9T 2.06G /mkjail/13.2-RELEASE system/usr 3.57G 18.9T 329K none system/usr/home 3.57G 18.9T 329K /usr/home system/usr/home/dan 3.57G 18.9T 904M /usr/home/dan tank_fast01 216G 683G 23K none tank_fast01/dbclone 4.94G 683G 4.94G /usr/jails/dbclone tank_fast01/dbclone.backups.rsyncer 195G 683G 195G /jails/dbclone/usr/home/rsyncer/backups tank_fast01/dbclone.postgres 989M 683G 989M /jails/dbclone/var/db/postgres tank_fast01/empty 1.87G 683G 24K none tank_fast01/empty/ports 1.87G 683G 1.87G /jails/empty/usr/ports tank_fast01/vm 13.3G 683G 346M /usr/local/vm tank_fast01/vm/mkjail 12.9G 683G 12.9G /usr/local/vm/mkjail
Of note to be migrated:
- system/backups-by-snapshot – this is backups of bhyve on r730-01
- system/data/bacula – Backups by Bacula, perhaps spanning back three years – I don’t need them all and will be copying them over via Bacula copy jobs. The new host will have a new bacula-sd configured and will start receiving all new backups.
- system/data/freshports/archives/supernews – some old FreshPorts files
- system/data/time_capsule – backups via Time Capsule for Apple of laptops
- system/jails – the jails running on this host
- system/usr/home/dan – yes, I’m bringing that over too.
How will the data be copied?
I’ll be using syncoid to copy over the data. I’ve recently used that for other hosts. Based on that post, here are things I did differently.
The failures, don’t do it this way
You can skip this section. It shows what I did wrong.
On the receiving host, I added mountpoint permissions:
[15:53 r730-03 dvl ~] % sudo zfs allow -u syncoid create,mount,receive,mountpoint data01
Wasn’t enough. Not even https://github.com/jimsalterjrs/sanoid/issues/660 with these settings helped:
[16:05 r730-03 dvl ~] % sudo zfs allow -u syncoid compression,create,destroy,hold,mount,mountpoint,receive,refreservation,release,rollback,snapshot data01
Still getting errors, although the data seems to be copying over:
[syncoid@r730-03 ~]$ syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --no-sync-snap --compress=lzo -quiet --no-privilege-elevation syncoid@knew.int.unixathome.org:system/jails data01/jails cannot mount '/data01/jails': failed to create mountpoint: Permission denied CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@knew.int.unixathome.org-1692115762 syncoid@knew.int.unixathome.org ' zfs send '"'"'system/jails'"'"'@'"'"'autosnap_2023-08-09_00:00:01_daily'"'"' | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lzop -dfc | zfs receive -s -F 'data01/jails' failed: 256 at /usr/local/bin/syncoid line 492. cannot mount '/data01/jails/ansible': failed to create mountpoint: Permission denied CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@knew.int.unixathome.org-1692115762 syncoid@knew.int.unixathome.org ' zfs send '"'"'system/jails/ansible'"'"'@'"'"'autosnap_2023-08-09_00:00:29_daily'"'"' | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lzop -dfc | zfs receive -s -F 'data01/jails/ansible' failed: 256 at /usr/local/bin/syncoid line 492. cannot mount '/data01/jails/bacula-sd-01': failed to create mountpoint: Permission denied CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@knew.int.unixathome.org-1692115762 syncoid@knew.int.unixathome.org ' zfs send '"'"'system/jails/bacula-sd-01'"'"'@'"'"'autosnap_2023-08-09_00:00:15_daily'"'"' | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lzop -dfc | zfs receive -s -F 'data01/jails/bacula-sd-01' failed: 256 at /usr/local/bin/syncoid line 492. cannot mount '/data01/jails/cliff1': failed to create mountpoint: Permission denied CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@knew.int.unixathome.org-1692115762 syncoid@knew.int.unixathome.org ' zfs send '"'"'system/jails/cliff1'"'"'@'"'"'autosnap_2023-08-09_00:00:18_daily'"'"' | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lzop -dfc | zfs receive -s -F 'data01/jails/cliff1' failed: 256 at /usr/local/bin/syncoid line 492. cannot mount '/data01/jails/dbclone': failed to create mountpoint: Permission denied CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@knew.int.unixathome.org-1692115762 syncoid@knew.int.unixathome.org ' zfs send '"'"'system/jails/dbclone'"'"'@'"'"'autosnap_2023-08-09_00:00:09_daily'"'"' | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lzop -dfc | zfs receive -s -F 'data01/jails/dbclone' failed: 256 at /usr/local/bin/syncoid line 492.
The next session contains what worked.
The permissions
OK, after trial and error.
These are the permissions on the sending host:
[20:55 knew dan ~] % sudo zfs allow -u syncoid send,snapshot,hold,destroy,mount system/jails
These are the permissions on the receiving host:
[20:29 r730-03 dvl ~] % sudo zfs allow -u syncoid create,mount,receive data01
To avoid those “failed to create mountpoint: Permission denied” errors, I did this on the receiving host:
sudo zfs set mountpoint=none data01
Trying again
Then it worked with this:
[syncoid@r730-03 ~]$ time syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lzo -quiet --no-privilege-elevation syncoid@knew.int.unixathome.org:system/jails/toiler data01/toiler real 2m56.345s user 0m21.674s sys 0m32.735s [syncoid@r730-03 ~]$
NOTE: that’s just one jail. Let’s destroy it and do the whole tree. This was just for testing.
Now let’s try the whole tree.
[syncoid@r730-03 ~]$ time syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lzo -quiet --no-privilege-elevation syncoid@knew.int.unixathome.org:system/jails data01/jails
The next day
Last night and this morning I ran through several iterations of moving jails and data from one host to another. At present, there is only one jail remaining – the big one:
[15:04 knew dan ~] % jls
JID IP Address Hostname Path
2 10.55.0.36 bacula-sd-01.int.unixathome.o /jails/bacula-sd-01
[15:13 knew dan ~] %
On the the new host, we have:
[15:11 r730-03 dvl ~] % jls
JID IP Address Hostname Path
1 10.55.0.116 fruity-int.int.unixathome.org /jails/fruity-int
3 10.55.0.14 cliff1.int.unixathome.org /jails/cliff1
4 10.55.0.21 empty.int.unixathome.org /jails/empty
5 10.55.0.117 fruity-ext.int.unixathome.org /jails/fruity-ext
6 10.55.0.113 ansible.int.unixathome.org /jails/ansible
7 10.55.0.140 dbclone.int.unixathome.org /jails/dbclone
8 10.55.0.13 toiler.int.unixathome.org /jails/toiler
[15:16 r730-03 dvl ~] %
All active and running fine. Nagios is happy. If all the database backups run fine tonight (via the dbclone jail), I’ll be happy tomorrow.
Next up, configuring a new jail to act as a Bacula Storage Daemon (named bacula-sd-04) and configure my Bacula installation to use it. I think that will be a completely separate blog post.
