My first big Tarsnap backup

NOTE: I wrote this post nearly two years ago, in May 2015. It has been sitting ignored and unloved in my Drafts. I’ve just published it today.

I’m big on backups and I use Bacula. I have about 18 TB on about 350 tapes and about 10 TB of backups on disk.

I want more.

I last used Tarsnap back in July 2010 (I know that because I found the old Tarsnap registration confirmation email). I did not pursue Tarsnap because I had too much data to backup. Right now, my monthly backups are about 400GB I think and I keep stuff for three years. I did the math on storing 10 TB of data and decided not to proceed.

Now I’m taking a different approach: backup the vital data to Tarsnap. What vital data? My svn repository which contains all my websites (bsdcan.org, pgcon.org, freshports.org langille.org, freebsddiary.org, etc). I have multiple copies of that data in multiple places.

How big is that repo?

[dan@svn:/usr/local/svn/repos] $ sudo du -ch -d 0 .
 15G	.
 15G	total
[dan@svn:/usr/local/svn/repos] $

I did the math. $0.25 / GB for storage means it costs me $4 a month for that backup. But the actual cost will be less because of deduplication and compression.

Getting started

First, I compiled sysutils/tarsnap via my poudriere instance.

On my svn server (running in a FreeBSD jail), I installed: pkg install tarsnap

Create your tarsnap account

Fund your tarsnap account

After you’ve logged into Tarsnap, you’ll need to add some funds to your account. You can pay with a credit card, Bitcoin, or Paypal

Create your tarsnap key

Funding is important. You cannot backup/restore without funds in your account.

However, your key is crucial. If you lose your key, your backups are gone. They cannot be restored.

Make backups of your key.

I printed mine out and also saved it in several locations.

This is the command I used to created my key. I did this as the root user.

tarsnap-keygen --keyfile /root/tarsnap.key --user dan@example.org --machine svn.int.example.org

NOTE: I recommend creating your tarsnap keys in a central location, then copying them to the service you need.

You will be prompted for your Tarsnap password during this process.

Save your key

Before you start your first backup, save the key you just created to a few safe locations.

My first Tarsnap backup

As root, I issued this command, and then left it to run overnight. I also started a tmux session. You could also use screen. This allows me to disconnect and leave the command running.

# tarsnap -c -f svn.int.2015.05.15 /usr/local/svn/repos 
Directory /usr/local/tarsnap-cache created for "--cachedir /usr/local/tarsnap-cache"
tarsnap: Removing leading '/' from member names

The next morning, I logged in, reattached my tumx session, and found:

# tarsnap -c -f svn.int.2015.05.15 /usr/local/svn/repos 
Directory /usr/local/tarsnap-cache created for "--cachedir /usr/local/tarsnap-cache"
tarsnap: Removing leading '/' from member names
                                       Total size  Compressed size
All archives                           7756892305       7664432848
  (unique data)                        7696029985       7603278799
This archive                           7756892305       7664432848
New data                               7696029985       7603278799

That 16GB of data compressed down to 7.6GB. That’s more than 50%. My $4 just went to $2 a month. I can afford this.

Let’s do another backup

My other vital data is my Bacula configuration. Let’s back that up.

This data is on another server, so I created another key, same way as before, on the Bacula server:

# tarsnap-keygen --keyfile /root/tarsnap.key --user dan@example.org --machine bacula.int.example.org

Then I ran the backup:

# tarsnap -c -f bacula.int.BaculaConfiguration.2015.05.16 /usr/local/etc/bacula
Directory /usr/local/tarsnap-cache created for "--cachedir /usr/local/tarsnap-cache"
tarsnap: Removing leading '/' from member names
                                       Total size  Compressed size
All archives                               121888            26105
  (unique data)                            113696            24107
This archive                               121888            26105
New data                                   113696            24107

That was fast. It’s only 469KB on disk, and compressed, it’s just 24KB.

How much did that cost?

There will be rounding errors in my math. Ignore them.

It is now the next day. Tarsnap has updated their records (which happens each day at about 0000 UTC). Here is what I see in my accounts for the repository backup:

The backup cost me $1.90. Storing the backup costs me $0.06 (or roughly $22 a year).

Looking at the other server, from which I backed up the Bacula configuration, I find:

That backup cost me: $0.000008 That figure is not an error. This means I could backup that data every day for a year and it would cost me less than one cent.

How much to store it? Daily? $0.0000002

Which means I could store it for 100 years, and it would cost me less than one cent.

Excluding some stuff

For my next server, I decided to exclude some files. This is really getting fancy. I also added the –dry-run and -v options to specify that nothing would be backed up, but instead, show me a list of what would have been backed up. This is especially useful when setting up a new backup.

# tarsnap --dry-run -v -c -f backups.2015.05.17 --exclude /home/dvl/backups/database-backup/postgresql/archive /home/dvl/backups/database-backup/postgresql
tarsnap: Removing leading '/' from member names
a home/dvl/backups/database-backup/postgresql
a home/dvl/backups/database-backup/postgresql/pgcon_phorum.dump
a home/dvl/backups/database-backup/postgresql/pgcon.dump
a home/dvl/backups/database-backup/postgresql/globals.sql
a home/dvl/backups/database-backup/postgresql/bsdcan.dump
a home/dvl/backups/database-backup/postgresql/bsdcan_phorum.dump
                                       Total size  Compressed size
All archives                               724637           526354
  (unique data)                            724637           526354
This archive                               724637           526354
New data                                   724637           526354

Yes, that looks good. Let’s do it for real, and time it:

# time tarsnap -v -c -f backups.2015.05.17 --exclude /home/dvl/backups/database-backup/postgresql/archive /home/dan/backups/database-backup/postgresql
tarsnap: Removing leading '/' from member names
a home/dvl/backups/database-backup/postgresql
a home/dvl/backups/database-backup/postgresql/pgcon_phorum.dump
a home/dvl/backups/database-backup/postgresql/pgcon.dump
a home/dvl/backups/database-backup/postgresql/globals.sql
a home/dvl/backups/database-backup/postgresql/bsdcan.dump
a home/dvl/backups/database-backup/postgresql/bsdcan_phorum.dump
                                       Total size  Compressed size
All archives                               724627           526344
  (unique data)                            724627           526344
This archive                               724627           526344
New data                                   724627           526344

real	0m5.796s
user	0m0.104s
sys	0m0.009s