I’ve started merging these two hosts:
into this one host:
The goal is to then retire those first two hosts. This post covers the initial bulk copy of all the data from two hosts into the one.
In this post:
- FreeBSD 13.1
- Each jail runs on its own zfs filesystem
Not covered by this post
- Creating jails
- Creating snapshost
- Detailed usage of syncoid
- hosts not using zfs
The overall process for each jail is
- zfs snapshot
- zfs send | zfs receive
There is more to it than that, but let’s get into it.
Important notes about copying the data
In this post, I’ll be copying over a lot of data while the jails are running. This is to get the bulk of the data over and in place. Later, when moving each jail individually, I will stop the jail, run another syncoid, then start the new jail. That second sync of the data will catch any changes since the initial copy.
zfs allow
EDIT: 2023-04-12 – One item which did not come over to the new host: the zfs allow permissions. I just ran this on each of my test, dev, and stage hosts for FreshPorts.
sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/categories sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/commits sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/daily sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/general sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/news sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/packages sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/pages sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/ports sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/spooling
… with stage varying as appropriate to the host in question.
syncoid
I’ll be using syncoid to copy the data (including jail filesystems) from the source to the destination. I’m using my using syncoid to backup ZFS snapshots – home assistant blog post as my starting point.
I am using non-root users to do the send | receive bits. I created new users specifically for this task. I’ll remove/disable their access when this transfer is done. I created the same user (syncoid) on both hosts.
Permissions
On the receiving host:
[r730-01 dvl ~] % sudo zfs allow -u syncoid create,mount,receive data01
ON the sending host:
[r720-01 dan ~] % sudo zfs allow -u syncoid send,snapshot tank_fast/poudriere
I created an ssh-key on the receiving host, because the transfers are going to be initiated from there.
[r730-01 dvl ~] % sudo su -l syncoid Want to run the same command again? In many shells (e.g., tcsh, zsh, bash) you can type "!!". [syncoid@r730-01 ~]$ ssh-keygen -t ed25519 Generating public/private ed25519 key pair. Enter file in which to save the key (/home/syncoid/.ssh/id_ed25519): Created directory '/home/syncoid/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/syncoid/.ssh/id_ed25519 Your public key has been saved in /home/syncoid/.ssh/id_ed25519.pub The key fingerprint is: SHA256:ObsnBzlBzIPHsGdbKexI3fbqJZpLLg6UiiadlUYXkl4 syncoid@r730-01.int.unixathome.org The key's randomart image is: +--[ED25519 256]--+ | ..o* | | ..E=B. . | | ...+o*.= | | ..= =.* . | | * . So . | | o * +o . | |o.+ . ooo . | |o ..o.=oo | | ...*=. | +----[SHA256]-----+
Then I allowed the destination syncoid user to ssh in here: this is the source host. We are pulling, not pushing. That echo is putting the ssh public key for the destination syncoid user.
[syncoid@r720-01 ~]$ mkdir ~/.ssh [syncoid@r720-01 ~]$ chmod 700 .ssh [syncoid@r720-01 ~]$ touch ~/.ssh/authorized_keys [syncoid@r720-01 ~]$ chmod 600 ~/.ssh/authorized_keys [syncoid@r720-01 ~]$ echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL9XHzjLnxnP3O9zNMBAFpnhBVvWEKXkJq2/SsG/DjlJ syncoid@r730-01.int.unixathome.org' >> ~/.ssh/authorized_keys [syncoid@r720-01 ~]$
Let’s try moving some data
This command is issued on the receiving host, as the syncoid user. I’m trying to get the remote tank_fast/poudriere filesystem received over as the local data01/poudriere filesystem.
I get errors.
[syncoid@r730-01 ~]$ syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --no-sync-snap --compress=lzo -quiet --no-privilege-elevation syncoid@r720-01.int.unixathome.org:tank_fast/poudriere data01/poudriere
CRITICAL ERROR: Target data01/poudriere exists but has no snapshots matching with tank_fast/poudriere!
Replication to target would require destroying existing
target. Cowardly refusing to destroy your existing target.
NOTE: Target data01/poudriere dataset is < 64MB used - did you mistakenly run
`zfs create data01/poudriere` on the target? ZFS initial
replication must be to a NON EXISTENT DATASET, which will
then be CREATED BY the initial replication process.
WARN: --no-sync-snap is set, and getnewestsnapshot() could not find any snapshots on source for current dataset. Continuing.
CRITICAL: no snapshots exist on source tank_fast/poudriere/ccache.13amd64, and you asked for --no-sync-snap.
WARN: --no-sync-snap is set, and getnewestsnapshot() could not find any snapshots on source for current dataset. Continuing.
CRITICAL: no snapshots exist on source tank_fast/poudriere/ccache.13i386, and you asked for --no-sync-snap.
warning: cannot send 'tank_fast/poudriere/ccache.amd64@autosnap_2021-04-21_00:00:00_daily': permission denied
cannot receive: failed to read from stream
...
CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676731130 syncoid@r720-01.int.unixathome.org ' zfs send '"'"'tank_fast/poudriere/test'"'"'@'"'"'autosnap_2021-04-25_16:40:00_daily'"'"' | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lzop -dfc | zfs receive -s -F 'data01/poudriere/test' failed: 256 at /usr/local/bin/syncoid line 492.
See https://mastodon.social/@dvl@bsd.network/109886779035438338 for detailed example
This time, I remove the –no-sync-snap option.
Progress:
[syncoid@r730-01 ~]$ syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lzo -quiet --no-privilege-elevation syncoid@pkg01.int.unixathome.org:tank_fast/poudriere data01/poudriere WARN: lzop not available on source ssh:-S /tmp/syncoid-syncoid@pkg01.int.unixathome.org-1676742537 syncoid@pkg01.int.unixathome.org- sync will continue without compression.
Now what do I have on the receiving host?
[r730-01 dvl ~] % zfs list -r data01/poudriere NAME USED AVAIL REFER MOUNTPOINT data01/poudriere 258G 3.75T 222K none data01/poudriere/ccache.13amd64 205K 3.75T 205K none data01/poudriere/ccache.13i386 205K 3.75T 205K none data01/poudriere/ccache.amd64 52.8G 3.75T 47.3G none data01/poudriere/ccache.i386 9.27G 3.75T 6.07G none data01/poudriere/data 93.7G 3.75T 29.0G none data01/poudriere/data/cache 198M 3.75T 68.6M none data01/poudriere/data/cronjob-logs 9.68M 3.75T 3.03M none data01/poudriere/data/packages 62.9G 3.75T 54.0G none data01/poudriere/distfiles 67.6G 3.75T 67.5G none data01/poudriere/jails 13.0G 3.75T 239K none data01/poudriere/jails/114R 1.76G 3.75T 1.76G none data01/poudriere/jails/121amd64 2.00G 3.75T 2.00G none data01/poudriere/jails/121i386 1.73G 3.75T 1.73G none data01/poudriere/jails/131amd64 2.04G 3.75T 2.04G none data01/poudriere/jails/131i386 1.72G 3.75T 1.72G none data01/poudriere/jails/13amd64 2.04G 3.75T 2.04G none data01/poudriere/jails/13i386 1.72G 3.75T 1.72G none data01/poudriere/ports 21.9G 3.75T 188K none data01/poudriere/ports/default 5.12G 3.75T 3.08G none data01/poudriere/ports/main 2.55G 3.75T 2.40G none data01/poudriere/ports/testing 14.2G 3.75T 3.06G none data01/poudriere/test 478K 3.75T 205K none
Good.
Now let’s bring over all the jails.
[syncoid@r730-01 ~]$ time syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lzo -quiet --no-privilege-elevation syncoid@r720-01.int.unixathome.org:tank_fast/jails data02/jails cannot destroy snapshots: permission denied cannot destroy snapshots: permission denied WARNING: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:16-GMT00:00; zfs destroy '"'"'tank_fast/jails'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:37-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380. cannot destroy snapshots: permission denied cannot destroy snapshots: permission denied WARNING: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails/keycloak'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:17-GMT00:00; zfs destroy '"'"'tank_fast/jails/keycloak'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:38-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380. cannot destroy snapshots: permission denied cannot destroy snapshots: permission denied WARNING: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails/mqtt01'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:38-GMT00:00; zfs destroy '"'"'tank_fast/jails/mqtt01'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:17-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380. ... WARNING: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails/pg03'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:39-GMT00:00; zfs destroy '"'"'tank_fast/jails/pg03'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:18-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380. cannot destroy snapshots: permission denied cannot destroy snapshots: permission denied WARNING: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails/pkg01'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:18-GMT00:00; zfs destroy '"'"'tank_fast/jails/pkg01'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:39-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380. real 22m3.694s user 5m41.922s sys 6m37.454s
This would probably have been faster if I’d hooked up the 10G card in this host.
In the above, I was moving jails from slocum. Now for the other host, r720-01:
[syncoid@r730-01 ~]$ time syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lz4 -quiet --no-privilege-elevation syncoid@slocum.int.unixathome.org:system/jails data02/jails
CRITICAL ERROR: Target data02/jails exists but has no snapshots matching with system/jails!
Replication to target would require destroying existing
target. Cowardly refusing to destroy your existing target.
Side note
So what’s going on under the hood?
[r730-01 dvl ~] % ps auwwx | grep syncoid 23:52:45 syncoid 54517 12.2 0.0 18632 8392 - Ss 23:49 1:12.03 ssh: /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 [mux] (ssh) syncoid 54552 6.2 0.0 18020 7420 3 S+ 23:49 0:16.35 zfs receive -s -F data02/jails/bacula syncoid 54551 6.2 0.0 26288 10732 3 S+ 23:49 0:34.11 lz4 -dc syncoid 54550 1.2 0.0 39460 22132 3 S+ 23:49 0:14.94 mbuffer -q -s 128k -m 16M syncoid 54136 0.0 0.0 14964 5264 - Ss 23:46 0:00.07 tmux: server (/tmp/tmux-1004/default) (tmux) root 54129 0.0 0.0 18724 8240 2 I 23:46 0:00.01 sudo su -l syncoid root 54130 0.0 0.0 13688 3080 2 I 23:46 0:00.00 su -l syncoid syncoid 54131 0.0 0.0 14524 4256 2 I 23:46 0:00.00 -su (bash) syncoid 54134 0.0 0.0 14964 4456 2 I+ 23:46 0:00.01 tmux: client (/tmp/tmux-1004/default) (tmux) syncoid 54137 0.0 0.0 14524 4272 3 Is 23:46 0:00.01 -bash (bash) syncoid 54513 0.0 0.0 26928 16508 3 I+ 23:49 0:00.13 /usr/local/bin/perl /usr/local/bin/syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lz4 -quiet --no-privilege-elevation syncoid@slocum.int.unixathome.org:system/jails data02/jails syncoid 54548 0.0 0.0 13596 2960 3 I+ 23:49 0:00.00 sh -c ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send '"'"'system/jails/bacula'"'"'@'"'"'mkjail-202203272107'"'"' | lz4 | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lz4 -dc | zfs receive -s -F 'data02/jails/bacula' syncoid 54549 0.0 0.0 18632 7852 3 I+ 23:49 0:00.00 ssh -i /home/syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org zfs send 'system/jails/bacula'@'mkjail-202203272107' | lz4 | mbuffer -q -s 128k -m 16M 2>/dev/null dvl 54801 0.0 0.0 12840 2324 4 S+ 23:52 0:00.00 grep syncoid [r730-01 dvl ~] % 23:52:52
Back to the command
This is the command I issued.
[syncoid@r730-01 ~]$ time syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lz4 -quiet --no-privilege-elevation syncoid@slocum.int.unixathome.org:system/jails data02/jails
CRITICAL ERROR: Target data02/jails exists but has no snapshots matching with system/jails!
Replication to target would require destroying existing
target. Cowardly refusing to destroy your existing target.
cannot mount '/jails/bacula': failed to create mountpoint: Permission denied
CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send '"'"'system/jails/bacula'"'"'@'"'"'mkjail-202203272107'"'"' | lz4 | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lz4 -dc | zfs receive -s -F 'data02/jails/bacula' failed: 256 at /usr/local/bin/syncoid line 492.
cannot mount '/jails/bacula-sd-03': failed to create mountpoint: Permission denied
CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send '"'"'system/jails/bacula-sd-03'"'"'@'"'"'mkjail-202203272107'"'"' | lz4 | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lz4 -dc | zfs receive -s -F 'data02/jails/bacula-sd-03' failed: 256 at /usr/local/bin/syncoid line 492.
cannot mount '/jails/besser': failed to create mountpoint: Permission denied
CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send '"'"'system/jails/besser'"'"'@'"'"'mkjail-202203272107'"'"' | lz4 | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lz4 -dc | zfs receive -s -F 'data02/jails/besser' failed: 256 at /usr/local/bin/syncoid line 492.
Watching that, I tried this:
[r730-01 dvl ~] % sudo zfs allow -u syncoid create,mount,receive,mountpoint data02
But that didn’t help:
cannot mount '/jails/zm': failed to create mountpoint: Permission denied CRITICAL ERROR: ssh -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send '"'"'system/jails/zm'"'"'@'"'"'mkjail-202203272107'"'"' | lz4 | mbuffer -q -s 128k -m 16M 2>/dev/null' | mbuffer -q -s 128k -m 16M 2>/dev/null | lz4 -dc | zfs receive -s -F 'data02/jails/zm' failed: 256 at /usr/local/bin/syncoid line 492. real 55m40.730s user 3m39.145s sys 12m17.804s
However, all the jails got copied over:
[r730-01 dvl ~] % zfs list -r data02/jails NAME USED AVAIL REFER MOUNTPOINT data02/jails 294G 1.39T 144K /jails data02/jails/bacula 19.2G 1.39T 19.2G /jails/bacula data02/jails/bacula-sd-02 7.89G 1.39T 5.34G /jails/bacula-sd-02 data02/jails/bacula-sd-03 5.02G 1.39T 5.02G /jails/bacula-sd-03 data02/jails/besser 16.8G 1.39T 16.8G /jails/besser data02/jails/certs 4.01G 1.39T 4.01G /jails/certs data02/jails/certs-rsync 3.59G 1.39T 3.59G /jails/certs-rsync data02/jails/cliff2 4.14G 1.39T 4.14G /jails/cliff2 data02/jails/dev-ingress01 5.49G 1.39T 5.49G /jails/dev-ingress01 data02/jails/dev-nginx01 4.40G 1.39T 4.40G /jails/dev-nginx01 data02/jails/dev-pgeu 11.0G 1.39T 5.80G /jails/dev-pgeu data02/jails/dns-hidden-master 4.27G 1.39T 4.27G /jails/dns-hidden-master data02/jails/fileserver 6.51G 1.39T 6.51G /jails/fileserver data02/jails/git 6.01G 1.39T 6.01G /jails/git data02/jails/homeassistant.first.success 2.74G 1.39T 2.74G /jails/homeassistant.first.success data02/jails/homeassistant.repeatable 1.52G 1.39T 1.52G /jails/homeassistant.repeatable data02/jails/jail-testing 1.35G 1.39T 1.35G /jails/jail-testing data02/jails/jail_within_jail 594M 1.39T 594M /jails/jail_within_jail data02/jails/keycloak 16.1G 1.39T 13.1G /jails/keycloak data02/jails/mobile-nginx01 4.37G 1.39T 4.37G /jails/mobile-nginx01 data02/jails/mqtt01 8.04G 1.39T 5.38G /jails/mqtt01 data02/jails/mydev 10.6G 1.39T 10.6G /jails/mydev data02/jails/mysql01 4.62G 1.39T 4.62G /jails/mysql01 data02/jails/nsnotify 3.91G 1.39T 3.91G /jails/nsnotify data02/jails/pg01 10.2G 1.39T 10.2G /jails/pg01 data02/jails/pg02 22.3G 1.39T 7.88G /jails/pg02 data02/jails/pg03 22.5G 1.39T 5.76G /jails/pg03 data02/jails/pkg01 19.4G 1.39T 10.9G /jails/pkg01 data02/jails/samdrucker 5.30G 1.39T 5.30G /jails/samdrucker data02/jails/serpico 4.89G 1.39T 4.89G /jails/serpico data02/jails/stage-ingress01 2.34G 1.39T 2.34G /jails/stage-ingress01 data02/jails/stage-nginx01 1.55G 1.39T 1.55G /jails/stage-nginx01 data02/jails/svn 5.51G 1.39T 5.51G /jails/svn data02/jails/talos 3.60G 1.39T 3.60G /jails/talos data02/jails/test-ingress01 4.11G 1.39T 4.11G /jails/test-ingress01 data02/jails/test-nginx01 1.66G 1.39T 1.66G /jails/test-nginx01 data02/jails/unifi01 20.0G 1.39T 20.0G /jails/unifi01 data02/jails/webserver 17.4G 1.39T 17.4G /jails/webserver data02/jails/zm 1.04G 1.39T 1.04G /jails/zm [r730-01 dvl ~] %
I call that a success.
Other posts will deal with the finer parts of copying the jails.
