In this post, I am using FreeBSD 9.3 but what I do here should have wide application, especially the errors I encounter.
I believe in backups. Not just rsync, but multiple copies and in multiple places. RAID is not a backup. rsync is not a backup. The goal of this post is to send a copy of a ZFS filesystem to a remote server.
The server in question is already backed up in a number of ways:
- via Bacula on a daily basis
- via an hourly rsync
- via copies of incoming mail sent to a remote mbox
In this post, I will show you how I started sending snapshots of the filesystem to a remove server.
My main reading for this was the ZFS Administration section of the FreeBSD Handbook.
Preparation
The filesystem I will send from the zuul server is system/DansMaildir.
On the receiving server, slocum, the destination will be system/usr/home/dan/mailjail
In this section, I will set permissions so non-root can snapshot, send, and receive. Some of these commands are done on the sender, some on the receiver, server.
As usual, I will show you the mistakes as well as the successes.
On the sender, I used these permissions
[dan@zuul:~] $ sudo zfs allow -u dan send,snapshot,hold system/DansMaildir [dan@zuul:~] $
The permissions I finally needed on the receiver were:
[dan@slocum:~] $ sudo zfs allow -u dan compression,mountpoint,create,mount,receive system/usr/home/dan/mailjail [dan@slocum:~] $
Some of the failures
I started out with this on the receiver:
[dan@slocum:~] $ sudo zfs create system/usr/home/dan/mailjail [dan@slocum:~] $ sudo zfs allow -u dan create,mount,receive system/usr/home/dan/mailjail [dan@slocum:~] $
On the sender, I started off with this:
[dan@zuul:~] $ sudo zfs allow -u dan send,snapshot system/DansMaildir [dan@zuul:~] $
First attempt at sending
Sending can take some time. It can be useful to do this command within screen or tmux.
Here, I start tmux, take a snapshot, and attempt, but fail to send it:
[dan@zuul:~] $ tmux [dan@zuul:~] $ zfs snapshot -r system/DansMaildir@InitialSlocumSend [dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh ' dan@slocum.int.unixathome.org zfs recv -dvu system/usr/home/dan/mailjail' Pseudo-terminal will not be allocated because stdin is not a terminal. cannot hold: permission denied cannot send 'system/DansMaildir': permission denied
It was then that I added the hold permission you see in the previous section.
[dan@zuul:~] $ zfs allow system/DansMaildir
---- Permissions on system/DansMaildir -------------------------------
Local+Descendent permissions:
user dan send,snapshot
[dan@zuul:~] $ sudo zfs allow -u dan send,snapshot,hold system/DansMaildir
[dan@zuul:~] $ zfs allow system/DansMaildir
---- Permissions on system/DansMaildir -------------------------------
Local+Descendent permissions:
user dan hold,send,snapshot
[dan@zuul:~] $
I also read up on that Pseudo-terminal error message and added -T to the ssh command, and I tried again:
[dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh -T -c blowfish dan@slocum.int.unixathome.org 'zfs recv -dvu system/usr/home/dan/mailjail' receiving full stream of system/DansMaildir@backup.2014.11.08 into system/usr/home/dan/mailjail/DansMaildir@backup.2014.11.08 cannot receive compression property on system/usr/home/dan/mailjail: permission denied cannot receive mountpoint property on system/usr/home/dan/mailjail: permission denied received 5.13GB stream in 1227 seconds (4.28MB/sec) receiving incremental stream of system/DansMaildir@backup.2014.11.08.2 into system/usr/home/dan/mailjail/DansMaildir@backup.2014.11.08.2 cannot receive compression property on system/usr/home/dan/mailjail/DansMaildir: permission denied cannot receive mountpoint property on system/usr/home/dan/mailjail/DansMaildir: permission denied received 15.6MB stream in 2 seconds (7.79MB/sec) receiving incremental stream of system/DansMaildir@BeforeMergeMaster9.3 into system/usr/home/dan/mailjail/DansMaildir@BeforeMergeMaster9.3 cannot receive compression property on system/usr/home/dan/mailjail/DansMaildir: permission denied cannot receive mountpoint property on system/usr/home/dan/mailjail/DansMaildir: permission denied received 898MB stream in 201 seconds (4.47MB/sec) receiving incremental stream of system/DansMaildir@InitialSlocumSend into system/usr/home/dan/mailjail/DansMaildir@InitialSlocumSend summary: 6633 MiByte in 25min 34.8sec - average of 4426 KiB/s cannot receive compression property on system/usr/home/dan/mailjail/DansMaildir: permission denied cannot receive mountpoint property on system/usr/home/dan/mailjail/DansMaildir: permission denied summary: 6633 MiByte in 25min 34.9sec - average of 4425 KiB/s received 469MB stream in 104 seconds (4.51MB/sec)
I see compression and mountpoint permission issues.
Let’s fix that on the receiver:
[dan@slocum:~] $ sudo zfs allow -u dan compression,mountpoint,create,mount,receive system/usr/home/dan/mailjail [dan@slocum:~] $
I try again:
[dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh -T -c blowfish dan@slocum.int.unixathome.org 'zfs recv -dvu system/usr/home/dan/mailjail' cannot receive new filesystem stream: destination 'system/usr/home/dan/mailjail/DansMaildir' exists must specify -F to overwrite it
Try again, with -F on receiver:
[dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh -T -c blowfish dan@slocum.int.unixathome.org zfs recv -Fdvu system/usr/home/dan/mailjail' cannot receive new filesystem stream: destination has snapshots (eg. system/usr/home/dan/mailjail/DansMaildir@BeforeMergeMaster9.3)
Oh. I know what happened. I have snapshots created via cronjobs… I removed them from the destination, and tried again.
[dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh -T -c blowfish dan@slocum.int.unixathome.org 'zfs recv -dvu system/usr/home/dan/mailjail' receiving full stream of system/DansMaildir@backup.2014.11.08 into system/usr/home/dan/mailjail/DansMaildir@backup.2014.11.08 received 5.13GB stream in 1132 seconds (4.64MB/sec) receiving incremental stream of system/DansMaildir@backup.2014.11.08.2 into system/usr/home/dan/mailjail/DansMaildir@backup.2014.11.08.2 received 15.6MB stream in 5 seconds (3.12MB/sec) receiving incremental stream of system/DansMaildir@BeforeMergeMaster9.3 into system/usr/home/dan/mailjail/DansMaildir@BeforeMergeMaster9.3 received 898MB stream in 123 seconds (7.30MB/sec) receiving incremental stream of system/DansMaildir@InitialSlocumSend into system/usr/home/dan/mailjail/DansMaildir@InitialSlocumSend summary: 6633 MiByte in 22min 06.2sec - average of 5122 KiB/s received 469MB stream in 67 seconds (7.00MB/sec)
That worked! Success! I have a copy. Good.
Updating the copy
Now I have a copy. But as new mail comes in, and I read, delete, or file it, the copy quickly becomes out dated.
ZFS has the concept of snapshots. We send a snapshot. We received a snapshop.
If we take another snapshot, we can send the differences between the two spanshots.
Here we go, on the sender:
[dan@zuul:~] $ zfs snapshot -r system/DansMaildir@2015.02.15.1345 [dan@zuul:~] $ zfs send -Ri system/DansMaildir@InitialSlocumSend system/DansMaildir@2015.02.15.1345 | ssh -T -c blowfish dan@slocum.int.unixathome.org 'zfs recv -dvu system/usr/home/dan/mailjail' receiving incremental stream of system/DansMaildir@2015.02.15.1345 into system/usr/home/dan/mailjail/DansMaildir@2015.02.15.1345 received 220MB stream in 90 seconds (2.44MB/sec) [dan@zuul:~] $
Doing it on a regular basis
To do this on a regular basis is not covered by this post. I figured this step was hard enough to get to, so let’s leave the automation of this copy to a future post.
Doing it more securely
When I go to automate this, I will restrict the ssh-key to only doing zfs receive.
