Feb 162015
 

In this post, I am using FreeBSD 9.3 but what I do here should have wide application, especially the errors I encounter.

I believe in backups. Not just rsync, but multiple copies and in multiple places. RAID is not a backup. rsync is not a backup. The goal of this post is to send a copy of a ZFS filesystem to a remote server.

The server in question is already backed up in a number of ways:

In this post, I will show you how I started sending snapshots of the filesystem to a remove server.

My main reading for this was the ZFS Administration section of the FreeBSD Handbook.

Preparation

The filesystem I will send from the zuul server is system/DansMaildir.

On the receiving server, slocum, the destination will be system/usr/home/dan/mailjail

In this section, I will set permissions so non-root can snapshot, send, and receive. Some of these commands are done on the sender, some on the receiver, server.

As usual, I will show you the mistakes as well as the successes.

On the sender, I used these permissions

[dan@zuul:~] $ sudo zfs allow -u dan send,snapshot,hold system/DansMaildir
[dan@zuul:~] $ 

The permissions I finally needed on the receiver were:

[dan@slocum:~] $ sudo zfs allow -u dan compression,mountpoint,create,mount,receive system/usr/home/dan/mailjail
[dan@slocum:~] $ 

Some of the failures

I started out with this on the receiver:

[dan@slocum:~] $ sudo zfs create system/usr/home/dan/mailjail    
[dan@slocum:~] $ sudo zfs allow -u dan create,mount,receive system/usr/home/dan/mailjail
[dan@slocum:~] $ 

On the sender, I started off with this:

[dan@zuul:~] $ sudo zfs allow -u dan send,snapshot system/DansMaildir
[dan@zuul:~] $ 

First attempt at sending

Sending can take some time. It can be useful to do this command within screen or tmux.

Here, I start tmux, take a snapshot, and attempt, but fail to send it:

[dan@zuul:~] $ tmux
[dan@zuul:~] $ zfs snapshot -r system/DansMaildir@InitialSlocumSend

[dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh ' dan@slocum.int.unixathome.org zfs recv -dvu system/usr/home/dan/mailjail'
Pseudo-terminal will not be allocated because stdin is not a terminal.
cannot hold: permission denied
cannot send 'system/DansMaildir': permission denied

It was then that I added the hold permission you see in the previous section.

[dan@zuul:~] $ zfs allow system/DansMaildir
---- Permissions on system/DansMaildir -------------------------------
Local+Descendent permissions:
        user dan send,snapshot

[dan@zuul:~] $ sudo zfs allow -u dan send,snapshot,hold system/DansMaildir

[dan@zuul:~] $ zfs allow system/DansMaildir
---- Permissions on system/DansMaildir -------------------------------
Local+Descendent permissions:
        user dan hold,send,snapshot
[dan@zuul:~] $ 

I also read up on that Pseudo-terminal error message and added -T to the ssh command, and I tried again:

[dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh -T -c blowfish dan@slocum.int.unixathome.org 'zfs recv -dvu system/usr/home/dan/mailjail'
receiving full stream of system/DansMaildir@backup.2014.11.08 into system/usr/home/dan/mailjail/DansMaildir@backup.2014.11.08
cannot receive compression property on system/usr/home/dan/mailjail: permission denied
cannot receive mountpoint property on system/usr/home/dan/mailjail: permission denied
received 5.13GB stream in 1227 seconds (4.28MB/sec)
receiving incremental stream of system/DansMaildir@backup.2014.11.08.2 into system/usr/home/dan/mailjail/DansMaildir@backup.2014.11.08.2
cannot receive compression property on system/usr/home/dan/mailjail/DansMaildir: permission denied
cannot receive mountpoint property on system/usr/home/dan/mailjail/DansMaildir: permission denied
received 15.6MB stream in 2 seconds (7.79MB/sec)
receiving incremental stream of system/DansMaildir@BeforeMergeMaster9.3 into system/usr/home/dan/mailjail/DansMaildir@BeforeMergeMaster9.3
cannot receive compression property on system/usr/home/dan/mailjail/DansMaildir: permission denied
cannot receive mountpoint property on system/usr/home/dan/mailjail/DansMaildir: permission denied
received 898MB stream in 201 seconds (4.47MB/sec)
receiving incremental stream of system/DansMaildir@InitialSlocumSend into system/usr/home/dan/mailjail/DansMaildir@InitialSlocumSend
summary: 6633 MiByte in 25min 34.8sec - average of 4426 KiB/s
cannot receive compression property on system/usr/home/dan/mailjail/DansMaildir: permission denied
cannot receive mountpoint property on system/usr/home/dan/mailjail/DansMaildir: permission denied

summary: 6633 MiByte in 25min 34.9sec - average of 4425 KiB/s
received 469MB stream in 104 seconds (4.51MB/sec)

I see compression and mountpoint permission issues.

Let’s fix that on the receiver:

[dan@slocum:~] $ sudo zfs allow -u dan compression,mountpoint,create,mount,receive system/usr/home/dan/mailjail
[dan@slocum:~] $ 

I try again:

[dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh -T -c blowfish dan@slocum.int.unixathome.org 'zfs recv -dvu system/usr/home/dan/mailjail'
cannot receive new filesystem stream: destination 'system/usr/home/dan/mailjail/DansMaildir' exists
must specify -F to overwrite it

Try again, with -F on receiver:

[dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh -T -c blowfish dan@slocum.int.unixathome.org  zfs recv -Fdvu system/usr/home/dan/mailjail'              
cannot receive new filesystem stream: destination has snapshots (eg. system/usr/home/dan/mailjail/DansMaildir@BeforeMergeMaster9.3)

Oh. I know what happened. I have snapshots created via cronjobs… I removed them from the destination, and tried again.

[dan@zuul:~] $ zfs send -R system/DansMaildir@InitialSlocumSend | ssh -T -c blowfish dan@slocum.int.unixathome.org 'zfs recv -dvu system/usr/home/dan/mailjail'               
receiving full stream of system/DansMaildir@backup.2014.11.08 into system/usr/home/dan/mailjail/DansMaildir@backup.2014.11.08
received 5.13GB stream in 1132 seconds (4.64MB/sec)
receiving incremental stream of system/DansMaildir@backup.2014.11.08.2 into system/usr/home/dan/mailjail/DansMaildir@backup.2014.11.08.2
received 15.6MB stream in 5 seconds (3.12MB/sec)
receiving incremental stream of system/DansMaildir@BeforeMergeMaster9.3 into system/usr/home/dan/mailjail/DansMaildir@BeforeMergeMaster9.3
received 898MB stream in 123 seconds (7.30MB/sec)
receiving incremental stream of system/DansMaildir@InitialSlocumSend into system/usr/home/dan/mailjail/DansMaildir@InitialSlocumSend
summary: 6633 MiByte in 22min 06.2sec - average of 5122 KiB/s
received 469MB stream in 67 seconds (7.00MB/sec)

That worked! Success! I have a copy. Good.

Updating the copy

Now I have a copy. But as new mail comes in, and I read, delete, or file it, the copy quickly becomes out dated.

ZFS has the concept of snapshots. We send a snapshot. We received a snapshop.

If we take another snapshot, we can send the differences between the two spanshots.

Here we go, on the sender:

[dan@zuul:~] $ zfs snapshot -r system/DansMaildir@2015.02.15.1345

[dan@zuul:~] $ zfs send -Ri system/DansMaildir@InitialSlocumSend  system/DansMaildir@2015.02.15.1345 | ssh -T -c blowfish dan@slocum.int.unixathome.org 'zfs recv -dvu system/usr/home/dan/mailjail'                                                                                                 
receiving incremental stream of system/DansMaildir@2015.02.15.1345 into system/usr/home/dan/mailjail/DansMaildir@2015.02.15.1345
received 220MB stream in 90 seconds (2.44MB/sec)
[dan@zuul:~] $

Doing it on a regular basis

To do this on a regular basis is not covered by this post. I figured this step was hard enough to get to, so let’s leave the automation of this copy to a future post.

Doing it more securely

When I go to automate this, I will restrict the ssh-key to only doing zfs receive.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive