Transferring a jail from one host to another

I’ve started merging these two hosts:

  1. r710-01
  2. slocum

into this one host:

The goal is to then retire those first two hosts. This post covers the initial bulk copy of all the data from two hosts into the one.

In this post:

  • FreeBSD 13.1
  • Each jail runs on its own zfs filesystem

Not covered by this post

  • Creating jails
  • Creating snapshost
  • Detailed usage of syncoid
  • hosts not using zfs

The overall process for each jail is

  1. zfs snapshot
  2. zfs send | zfs receive

There is more to it than that, but let’s get into it.

Important notes about copying the data

In this post, I’ll be copying over a lot of data while the jails are running. This is to get the bulk of the data over and in place. Later, when moving each jail individually, I will stop the jail, run another syncoid, then start the new jail. That second sync of the data will catch any changes since the initial copy.

zfs allow

EDIT: 2023-04-12 – One item which did not come over to the new host: the zfs allow permissions. I just ran this on each of my test, dev, and stage hosts for FreshPorts.

sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/categories
sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/commits
sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/daily
sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/general
sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/news
sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/packages
sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/pages
sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/ports
sudo zfs allow -u freshports rollback data02/freshports/jailed/stage-nginx01/cache/spooling

… with stage varying as appropriate to the host in question.

syncoid

I’ll be using syncoid to copy the data (including jail filesystems) from the source to the destination. I’m using my using syncoid to backup ZFS snapshots – home assistant blog post as my starting point.

I am using non-root users to do the send | receive bits. I created new users specifically for this task. I’ll remove/disable their access when this transfer is done. I created the same user (syncoid) on both hosts.

Permissions

On the receiving host:

[r730-01 dvl ~] % sudo zfs allow -u syncoid create,mount,receive data01

ON the sending host:

[r720-01 dan ~] % sudo zfs allow -u syncoid send,snapshot tank_fast/poudriere

I created an ssh-key on the receiving host, because the transfers are going to be initiated from there.

[r730-01 dvl ~] % sudo su -l syncoid
Want to run the same command again?
In many shells (e.g., tcsh, zsh, bash) you can type "!!".
[syncoid@r730-01 ~]$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/syncoid/.ssh/id_ed25519): 
Created directory '/home/syncoid/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/syncoid/.ssh/id_ed25519
Your public key has been saved in /home/syncoid/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:ObsnBzlBzIPHsGdbKexI3fbqJZpLLg6UiiadlUYXkl4 syncoid@r730-01.int.unixathome.org
The key's randomart image is:
+--[ED25519 256]--+
|    ..o*         |
|    ..E=B. .     |
|   ...+o*.=      |
|   ..= =.* .     |
|    * . So  .    |
| o *    +o .     |
|o.+ .   ooo .    |
|o    ..o.=oo     |
|     ...*=.      |
+----[SHA256]-----+

Then I allowed the destination syncoid user to ssh in here: this is the source host. We are pulling, not pushing. That echo is putting the ssh public key for the destination syncoid user.

[syncoid@r720-01 ~]$ mkdir ~/.ssh
[syncoid@r720-01 ~]$ chmod 700 .ssh
[syncoid@r720-01 ~]$ touch ~/.ssh/authorized_keys
[syncoid@r720-01 ~]$ chmod 600 ~/.ssh/authorized_keys
[syncoid@r720-01 ~]$ echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL9XHzjLnxnP3O9zNMBAFpnhBVvWEKXkJq2/SsG/DjlJ syncoid@r730-01.int.unixathome.org' >> ~/.ssh/authorized_keys
[syncoid@r720-01 ~]$ 

Let’s try moving some data

This command is issued on the receiving host, as the syncoid user. I’m trying to get the remote tank_fast/poudriere filesystem received over as the local data01/poudriere filesystem.

I get errors.

[syncoid@r730-01 ~]$ syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --no-sync-snap --compress=lzo -quiet --no-privilege-elevation syncoid@r720-01.int.unixathome.org:tank_fast/poudriere data01/poudriere

CRITICAL ERROR: Target data01/poudriere exists but has no snapshots matching with tank_fast/poudriere!
                Replication to target would require destroying existing
                target. Cowardly refusing to destroy your existing target.

          NOTE: Target data01/poudriere dataset is < 64MB used - did you mistakenly run
                `zfs create data01/poudriere` on the target? ZFS initial
                replication must be to a NON EXISTENT DATASET, which will
                then be CREATED BY the initial replication process.

WARN: --no-sync-snap is set, and getnewestsnapshot() could not find any snapshots on source for current dataset. Continuing.
CRITICAL: no snapshots exist on source tank_fast/poudriere/ccache.13amd64, and you asked for --no-sync-snap.
WARN: --no-sync-snap is set, and getnewestsnapshot() could not find any snapshots on source for current dataset. Continuing.
CRITICAL: no snapshots exist on source tank_fast/poudriere/ccache.13i386, and you asked for --no-sync-snap.
warning: cannot send 'tank_fast/poudriere/ccache.amd64@autosnap_2021-04-21_00:00:00_daily': permission denied
cannot receive: failed to read from stream

...

CRITICAL ERROR: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676731130 syncoid@r720-01.int.unixathome.org ' zfs send  '"'"'tank_fast/poudriere/test'"'"'@'"'"'autosnap_2021-04-25_16:40:00_daily'"'"' | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc |  zfs receive  -s -F 'data01/poudriere/test' failed: 256 at /usr/local/bin/syncoid line 492.


See https://mastodon.social/@dvl@bsd.network/109886779035438338 for detailed example

This time, I remove the –no-sync-snap option.

Progress:

[syncoid@r730-01 ~]$ syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lzo -quiet --no-privilege-elevation syncoid@pkg01.int.unixathome.org:tank_fast/poudriere data01/poudriere
WARN: lzop not available on source ssh:-S /tmp/syncoid-syncoid@pkg01.int.unixathome.org-1676742537 syncoid@pkg01.int.unixathome.org- sync will continue without compression.

Now what do I have on the receiving host?

[r730-01 dvl ~] % zfs list -r data01/poudriere
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
data01/poudriere                     258G  3.75T      222K  none
data01/poudriere/ccache.13amd64      205K  3.75T      205K  none
data01/poudriere/ccache.13i386       205K  3.75T      205K  none
data01/poudriere/ccache.amd64       52.8G  3.75T     47.3G  none
data01/poudriere/ccache.i386        9.27G  3.75T     6.07G  none
data01/poudriere/data               93.7G  3.75T     29.0G  none
data01/poudriere/data/cache          198M  3.75T     68.6M  none
data01/poudriere/data/cronjob-logs  9.68M  3.75T     3.03M  none
data01/poudriere/data/packages      62.9G  3.75T     54.0G  none
data01/poudriere/distfiles          67.6G  3.75T     67.5G  none
data01/poudriere/jails              13.0G  3.75T      239K  none
data01/poudriere/jails/114R         1.76G  3.75T     1.76G  none
data01/poudriere/jails/121amd64     2.00G  3.75T     2.00G  none
data01/poudriere/jails/121i386      1.73G  3.75T     1.73G  none
data01/poudriere/jails/131amd64     2.04G  3.75T     2.04G  none
data01/poudriere/jails/131i386      1.72G  3.75T     1.72G  none
data01/poudriere/jails/13amd64      2.04G  3.75T     2.04G  none
data01/poudriere/jails/13i386       1.72G  3.75T     1.72G  none
data01/poudriere/ports              21.9G  3.75T      188K  none
data01/poudriere/ports/default      5.12G  3.75T     3.08G  none
data01/poudriere/ports/main         2.55G  3.75T     2.40G  none
data01/poudriere/ports/testing      14.2G  3.75T     3.06G  none
data01/poudriere/test                478K  3.75T      205K  none

Good.

Now let’s bring over all the jails.

[syncoid@r730-01 ~]$ time syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lzo -quiet --no-privilege-elevation syncoid@r720-01.int.unixathome.org:tank_fast/jails data02/jails
cannot destroy snapshots: permission denied
cannot destroy snapshots: permission denied
WARNING: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:16-GMT00:00;  zfs destroy '"'"'tank_fast/jails'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:37-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380.
cannot destroy snapshots: permission denied
cannot destroy snapshots: permission denied
WARNING: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails/keycloak'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:17-GMT00:00;  zfs destroy '"'"'tank_fast/jails/keycloak'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:38-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380.
cannot destroy snapshots: permission denied
cannot destroy snapshots: permission denied
WARNING: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails/mqtt01'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:38-GMT00:00;  zfs destroy '"'"'tank_fast/jails/mqtt01'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:17-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380.


...


WARNING: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails/pg03'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:39-GMT00:00;  zfs destroy '"'"'tank_fast/jails/pg03'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:18-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380.
cannot destroy snapshots: permission denied
cannot destroy snapshots: permission denied
WARNING: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@r720-01.int.unixathome.org-1676753493 syncoid@r720-01.int.unixathome.org ' zfs destroy '"'"'tank_fast/jails/pkg01'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:18-GMT00:00;  zfs destroy '"'"'tank_fast/jails/pkg01'"'"'@syncoid_r730-01.int.unixathome.org_2023-02-18:20:46:39-GMT00:00' failed: 256 at /usr/local/bin/syncoid line 1380.

real	22m3.694s
user	5m41.922s
sys	6m37.454s

This would probably have been faster if I’d hooked up the 10G card in this host.

In the above, I was moving jails from slocum. Now for the other host, r720-01:

[syncoid@r730-01 ~]$ time syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lz4 -quiet --no-privilege-elevation syncoid@slocum.int.unixathome.org:system/jails data02/jails

CRITICAL ERROR: Target data02/jails exists but has no snapshots matching with system/jails!
                Replication to target would require destroying existing
                target. Cowardly refusing to destroy your existing target.

Side note

So what’s going on under the hood?

[r730-01 dvl ~] % ps auwwx | grep syncoid                                                                                                                                          23:52:45
syncoid 54517   12.2  0.0 18632  8392  -  Ss   23:49       1:12.03 ssh: /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 [mux] (ssh)
syncoid 54552    6.2  0.0 18020  7420  3  S+   23:49       0:16.35 zfs receive -s -F data02/jails/bacula
syncoid 54551    6.2  0.0 26288 10732  3  S+   23:49       0:34.11 lz4 -dc
syncoid 54550    1.2  0.0 39460 22132  3  S+   23:49       0:14.94 mbuffer -q -s 128k -m 16M
syncoid 54136    0.0  0.0 14964  5264  -  Ss   23:46       0:00.07 tmux: server (/tmp/tmux-1004/default) (tmux)
root    54129    0.0  0.0 18724  8240  2  I    23:46       0:00.01 sudo su -l syncoid
root    54130    0.0  0.0 13688  3080  2  I    23:46       0:00.00 su -l syncoid
syncoid 54131    0.0  0.0 14524  4256  2  I    23:46       0:00.00 -su (bash)
syncoid 54134    0.0  0.0 14964  4456  2  I+   23:46       0:00.01 tmux: client (/tmp/tmux-1004/default) (tmux)
syncoid 54137    0.0  0.0 14524  4272  3  Is   23:46       0:00.01 -bash (bash)
syncoid 54513    0.0  0.0 26928 16508  3  I+   23:49       0:00.13 /usr/local/bin/perl /usr/local/bin/syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lz4 -quiet --no-privilege-elevation syncoid@slocum.int.unixathome.org:system/jails data02/jails
syncoid 54548    0.0  0.0 13596  2960  3  I+   23:49       0:00.00 sh -c ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send  '"'"'system/jails/bacula'"'"'@'"'"'mkjail-202203272107'"'"' | lz4  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lz4 -dc |  zfs receive  -s -F 'data02/jails/bacula'
syncoid 54549    0.0  0.0 18632  7852  3  I+   23:49       0:00.00 ssh -i /home/syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org  zfs send  'system/jails/bacula'@'mkjail-202203272107' | lz4  | mbuffer  -q -s 128k -m 16M 2>/dev/null
dvl     54801    0.0  0.0 12840  2324  4  S+   23:52       0:00.00 grep syncoid
[r730-01 dvl ~] %                                                                                                                                                                  23:52:52

Back to the command

This is the command I issued.

[syncoid@r730-01 ~]$ time syncoid --sshkey=~syncoid/.ssh/id_ed25519 -r --compress=lz4 -quiet --no-privilege-elevation syncoid@slocum.int.unixathome.org:system/jails data02/jails

CRITICAL ERROR: Target data02/jails exists but has no snapshots matching with system/jails!
                Replication to target would require destroying existing
                target. Cowardly refusing to destroy your existing target.

cannot mount '/jails/bacula': failed to create mountpoint: Permission denied
CRITICAL ERROR: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send  '"'"'system/jails/bacula'"'"'@'"'"'mkjail-202203272107'"'"' | lz4  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lz4 -dc |  zfs receive  -s -F 'data02/jails/bacula' failed: 256 at /usr/local/bin/syncoid line 492.
cannot mount '/jails/bacula-sd-03': failed to create mountpoint: Permission denied
CRITICAL ERROR: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send  '"'"'system/jails/bacula-sd-03'"'"'@'"'"'mkjail-202203272107'"'"' | lz4  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lz4 -dc |  zfs receive  -s -F 'data02/jails/bacula-sd-03' failed: 256 at /usr/local/bin/syncoid line 492.
cannot mount '/jails/besser': failed to create mountpoint: Permission denied
CRITICAL ERROR: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send  '"'"'system/jails/besser'"'"'@'"'"'mkjail-202203272107'"'"' | lz4  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lz4 -dc |  zfs receive  -s -F 'data02/jails/besser' failed: 256 at /usr/local/bin/syncoid line 492.

Watching that, I tried this:

[r730-01 dvl ~] % sudo zfs allow -u syncoid create,mount,receive,mountpoint data02

But that didn’t help:

cannot mount '/jails/zm': failed to create mountpoint: Permission denied
CRITICAL ERROR: ssh    -i ~syncoid/.ssh/id_ed25519 -S /tmp/syncoid-syncoid@slocum.int.unixathome.org-1676764160 syncoid@slocum.int.unixathome.org ' zfs send  '"'"'system/jails/zm'"'"'@'"'"'mkjail-202203272107'"'"' | lz4  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lz4 -dc |  zfs receive  -s -F 'data02/jails/zm' failed: 256 at /usr/local/bin/syncoid line 492.

real    55m40.730s
user    3m39.145s
sys     12m17.804s

However, all the jails got copied over:

[r730-01 dvl ~] % zfs list -r data02/jails
NAME                                       USED  AVAIL     REFER  MOUNTPOINT
data02/jails                               294G  1.39T      144K  /jails
data02/jails/bacula                       19.2G  1.39T     19.2G  /jails/bacula
data02/jails/bacula-sd-02                 7.89G  1.39T     5.34G  /jails/bacula-sd-02
data02/jails/bacula-sd-03                 5.02G  1.39T     5.02G  /jails/bacula-sd-03
data02/jails/besser                       16.8G  1.39T     16.8G  /jails/besser
data02/jails/certs                        4.01G  1.39T     4.01G  /jails/certs
data02/jails/certs-rsync                  3.59G  1.39T     3.59G  /jails/certs-rsync
data02/jails/cliff2                       4.14G  1.39T     4.14G  /jails/cliff2
data02/jails/dev-ingress01                5.49G  1.39T     5.49G  /jails/dev-ingress01
data02/jails/dev-nginx01                  4.40G  1.39T     4.40G  /jails/dev-nginx01
data02/jails/dev-pgeu                     11.0G  1.39T     5.80G  /jails/dev-pgeu
data02/jails/dns-hidden-master            4.27G  1.39T     4.27G  /jails/dns-hidden-master
data02/jails/fileserver                   6.51G  1.39T     6.51G  /jails/fileserver
data02/jails/git                          6.01G  1.39T     6.01G  /jails/git
data02/jails/homeassistant.first.success  2.74G  1.39T     2.74G  /jails/homeassistant.first.success
data02/jails/homeassistant.repeatable     1.52G  1.39T     1.52G  /jails/homeassistant.repeatable
data02/jails/jail-testing                 1.35G  1.39T     1.35G  /jails/jail-testing
data02/jails/jail_within_jail              594M  1.39T      594M  /jails/jail_within_jail
data02/jails/keycloak                     16.1G  1.39T     13.1G  /jails/keycloak
data02/jails/mobile-nginx01               4.37G  1.39T     4.37G  /jails/mobile-nginx01
data02/jails/mqtt01                       8.04G  1.39T     5.38G  /jails/mqtt01
data02/jails/mydev                        10.6G  1.39T     10.6G  /jails/mydev
data02/jails/mysql01                      4.62G  1.39T     4.62G  /jails/mysql01
data02/jails/nsnotify                     3.91G  1.39T     3.91G  /jails/nsnotify
data02/jails/pg01                         10.2G  1.39T     10.2G  /jails/pg01
data02/jails/pg02                         22.3G  1.39T     7.88G  /jails/pg02
data02/jails/pg03                         22.5G  1.39T     5.76G  /jails/pg03
data02/jails/pkg01                        19.4G  1.39T     10.9G  /jails/pkg01
data02/jails/samdrucker                   5.30G  1.39T     5.30G  /jails/samdrucker
data02/jails/serpico                      4.89G  1.39T     4.89G  /jails/serpico
data02/jails/stage-ingress01              2.34G  1.39T     2.34G  /jails/stage-ingress01
data02/jails/stage-nginx01                1.55G  1.39T     1.55G  /jails/stage-nginx01
data02/jails/svn                          5.51G  1.39T     5.51G  /jails/svn
data02/jails/talos                        3.60G  1.39T     3.60G  /jails/talos
data02/jails/test-ingress01               4.11G  1.39T     4.11G  /jails/test-ingress01
data02/jails/test-nginx01                 1.66G  1.39T     1.66G  /jails/test-nginx01
data02/jails/unifi01                      20.0G  1.39T     20.0G  /jails/unifi01
data02/jails/webserver                    17.4G  1.39T     17.4G  /jails/webserver
data02/jails/zm                           1.04G  1.39T     1.04G  /jails/zm
[r730-01 dvl ~] % 

I call that a success.

Other posts will deal with the finer parts of copying the jails.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Leave a Comment

Scroll to Top