Bacula on FreeBSD with ZFS

Leave a Comment / Advocacy, Backups, Bacula, FreeBSD, PostgreSQL, ZFS / By

Full disclosure: I’m biased. I like the combination of PostgreSQL, FreeBSD, ZFS, and Bacula. In fact, I like it so much, I wrote the PostgreSQL backend for Bacula. Since then, I have been using Bacula for over 10 years. I have backed up to HDD, DDS, DLT, and SDLT. Bacula has proven to be reliable and it just works. Clients are available for a wide range of operating systems and Bacula doesn’t care what file system you are using.

I am also the FreeBSD port maintainer for Bacula.

I first wrote about Bacula for O’Reilly (that post was https://www.onlamp.com/pub/a/onlamp/2004/01/09/bacula.html but it has been removed). I later reposted that article to my diary. I am updating that article for this issue of FreeBSD Journal.

There actually isn’t much useful information in this post about ZFS, but after writing this article, I found there was so much to cover, and omitting ZFS in details made sense.

About this article

In this article you will learn:

  1. How to install Bacula
  2. Get your first backup running

I will be using:

  1. Bacula 7.0.5
  2. PostgreSQL 9.3.5
  3. FreeBSD 9.3
  4. ZFS

You don’t have to be using the same choices to get something out of this article. You will learn Bacula concepts that will help you through commonly misunderstood issues. I promise you’ll get something out of it.

Unless otherwise mentioned, all configuration files for Bacula will be in /usr/local/etc/bacula/.

Bacula directives ignore spaces. Thus, WorkingDirectory and Working Directory are identical. Use whichever you prefer.

Some values are in “quotes”, some are not. Use whichever you prefer, but remember that if the value contains a space, “you need to put it in quotes”.

Words which are Bacula components or concepts will be in Bold with the first letter in Capital Letters. This will help you different a File directive from a file on disk, or a Volume from a volume.

Programs or commands will appear like bconsole. Files will appear like /etc/rc.conf.

What I will cover

  1. Short introduction of Bacula
  2. Clients
  3. Director
  4. Storage
  5. Backups
  6. Restores

Short introduction of Bacula

This description is short and generalized. Just because I don’t mention something, doesn’t mean Bacula can’t do it.

Bacula is a backup solution based on a networked client-server model. It can backup to disk or tape. The community is supportive and thriving. It has great commercial support if you want that. Control of backups is centralized around a Director. When scaling out, you can have as many Directors as you want. Each Director has Clients, which might be shared with other Directors.

The Director contacts the Client, and tells it: backup this FileSet to this Storage. The Client then does as it is asked and backs up the files. Later, that backup job might be copied or migrated to another Storage, which might be local or remote. For example, you might be backing up to disk and you want that job copied to tape. Or copied to a remote location.

Whatever you do with your backups, the Catalog keeps track of what was backed up, when, from what Client, and where it is located now. The Catalog is also used for retention purposes and for pruning and purging.

The main components

Bacula is not one program; it is several. The networked programs talk to each other via TCP/IP. The main components of Bacula are:

  1. Director – daemon (bacula-dir) which controls jobs.
  2. bconsole – command line interface console.
  3. Storage – daemon (bacula-sd) which receives backups from Clients and stores them to a storage device (e.g. tape, disk).
  4. Client – daemon (bacula-fd) which reads files and sends them to Storage.
  5. Catalog – a database which keeps track of what jobs have been run and where they are stored. I recommend PostgreSQL for this. MySQL can also be used, but you’ll get better performance from PostgreSQL. I do not recommend using SQLite for this purpose.

The rest of this section can be skipped. It is not vital to understanding how to get Bacula running for your first time. All of these components need to be able to talk to each other. Keep this in mind with respect to your firewalls. Especially when you read the following and see the order of the communications.

  • bacula-dir contacts bacula-fd to tell it what to backup up
  • bacula-fd contacts bacula-sd to send it backups
  • bacula-sd contacts bacula-dir with the status of completed backups
  • bacula-dir contacts the database server with the status
  • bconsole contacts bacula-dir
  • You can query status of bacula-dir, bacula-sd, and bacula-fd via bconsole. When you do this, bacula-dir contacts bacula-sd, and then bacula-fd to ask their status and relay it back to you
  • if copying jobs from one storage to another, the bacula-sd needs to talk to the other bacula-sd
  • if you use the SD Calls Client directive, bacula-sd needs to be able to contact bacula-fd

My backups

When I first started with Bacula, I backed up directly to tape. Now, with HDD being larger and cheaper, I backup to disk. Some people think that’s enough. I don’t. One power supply failure could wipe all those HDD. ZFS is the best filesystem there is, but it does not protect against catastrophic tragedies. I believe in multiple copies of your backups. At least one here, and one there. This is why I backup to tape, and then move those tapes to another location.

I have plans to send backups to a remote bacula-sd as well. I have not implemented that yet, but I will. Soon.

Installation

Installing Bacula is rather easy. If you want to use PostgreSQL, and I recommend you do, you can install from the pre-built packages. If not, install from the ports tree.

Here is what I installed:

# pkg install bacula-server
# pkg install postgresql93-server

I am using PostgreSQL 9.3 as that is the current default in the FreeBSD ports/packages system. Other versions will be fine.

Database setup

This is a light weight introduction to configuring PostgreSQL for use with Bacula. The official documentation is much more complete.

In these steps, we first create a new PostgreSQL database cluster and then start the server:

# service postgresql initdb

The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/local/bin/postgres -D /usr/local/pgsql/data
or
    /usr/local/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start

# service postgresql start
LOG:  could not create IPv6 socket: Protocol not supported
LOG:  ending log output to stderr
HINT:  Future log output will go to log destination "syslog".

Next, we will create the database, the Catalog, for Bacula to use. Some output has been trimmed from this example because it is repetitive and not very exciting to read.

# su pgsql
# cd /usr/local/share/bacula/
$ createuser -d bacula
./create_bacula_database 
Creating postgresql database
CREATE DATABASE
ALTER DATABASE
Creation of bacula database succeeded.
Database encoding OK
[pgsql@baculatest /usr/local/share/bacula]$ psql -l
                          List of databases
   Name    | Owner | Encoding  | Collate | Ctype | Access privileges 
-----------+-------+-----------+---------+-------+-------------------
 bacula    | pgsql | SQL_ASCII | C       | C     | 
 postgres  | pgsql | UTF8      | C       | C     | 
 template0 | pgsql | UTF8      | C       | C     | =c/pgsql         +
           |       |           |         |       | pgsql=CTc/pgsql
 template1 | pgsql | UTF8      | C       | C     | =c/pgsql         +
           |       |           |         |       | pgsql=CTc/pgsql
(4 rows)


./make_bacula_tables
Making postgresql tables
CREATE TABLE
ALTER TABLE
CREATE INDEX
CREATE TABLE
ALTER TABLE

[some output trimmed]

INSERT 0 1
INSERT 0 1
INSERT 0 1

[some output trimmed]

INSERT 0 1
Creation of Bacula PostgreSQL tables succeeded.
[pgsql@baculatest /usr/local/share/bacula]$ ./grant_bacula_privileges
Granting postgresql privileges
psql::2: ERROR:  role "bacula" already exists
GRANT
GRANT
GRANT

[some output trimmed]

GRANT
GRANT
GRANT
Privileges for user bacula granted on database bacula.

This is what you should see now:

$ psql -l
                          List of databases
   Name    | Owner | Encoding  | Collate | Ctype | Access privileges 
-----------+-------+-----------+---------+-------+-------------------
 bacula    | pgsql | SQL_ASCII | C       | C     | 
 postgres  | pgsql | UTF8      | C       | C     | 
 template0 | pgsql | UTF8      | C       | C     | =c/pgsql         +
           |       |           |         |       | pgsql=CTc/pgsql
 template1 | pgsql | UTF8      | C       | C     | =c/pgsql         +
           |       |           |         |       | pgsql=CTc/pgsql
(4 rows)

Configuration

All of this configuration will be done on a single IP address. For starting out, I recommend this approach. This will let you get things running first, then move them to another server. One step at a time.

The IP address I am using is 10.55.0.76 and that is the only value you should have to change in your configuration if everything is installed on one system.

The are four main configuration files with Bacula. The default location is /usr/local/etc/bacula/:

  1. bacula-dir.conf
  2. bacula-fd.conf
  3. bacula-sd.conf
  4. bconsole.conf

These file names are self-explanatory. In the next few sections, I will give you a minimal configuration which will get you started. I guarantee it.

In the following examples, we will start slowly and quickly gain speed. The first bacula-dir.conf file you see will be added to as we go along. I will provide links for all configuration files at the end of this post.

bacula-dir.conf

bacula-dir.conf is, by far, the largest configuration file you will create for Bacula. Fortunately, you can break it up into logical areas and keep them in as many, or as few, files as you like. Amongst other things, the items specified by bacula-dir.conf include:

  • identity of this particular bacula-dir
  • identify the Client it will backup
  • list the Job which will run on that Client
  • specify the Catalog which will document those backups
  • list the FileSet (list of files) for that Job
  • specify the Schedule for that Job
  • identify the Storage to be used

There are other directives (e.g. such as Pools) but we will postpone that discussion until later.

I like to align my configuration files like this. I find it easier to read. It is not required.

Director {
  Name             = MyBaculaDirector
  Password         = "the bacula-dir password"

  QueryFile        = "/usr/local/share/bacula/query.sql"

  PidDirectory     = "/var/run"
  WorkingDirectory = "/usr/local/bacula/working"

  Messages         = MyMessages
}

Catalog {
  Name      = MyCatalog
  dbname    = bacula
  dbaddress = 10.55.0.76
  user      = bacula
  password  = ""
}

Messages {
  Name            = MyMessages
  mailcommand     = "/usr/local/sbin/bsmtp -h localhost -f \"\(Bacula\) %r\" -s \"Bacula: %t %e of %c %l\" %r"
  operatorcommand = "/usr/local/sbin/bsmtp -h localhost -f \"\(Bacula\) %r\" -s \"Bacula: Intervention needed for %j\" %r"

  operator = root@localhost = mount
  mail     = root@localhost = all
  console  = all
  append   = "/var/log/bacula.log" = all, !skipped, !restored
  catalog  = all, !skipped, !saved
}

You can use this configuration as shown. I recommend keeping that password for your first setup. Change it later, after you have everything working.

That Name and Password as the credentials which this Director will accept when you login via bconsole. Those values will appear in two places:

  1. in bacula-dir.conf, so bacula-dir knows what credentials to accept
  2. in bconsole.conf, so bconsole knows what credentials to send

This ‘appear in two places’ is an important concept to learn early on. It is a design used by Bacula in several places. I will point these out when they appear in later examples.

As your bacula-dir.conf file grows, you can split out your bacula-dir.conf configuration into separate files. Each can be included with a command such as this:

@/usr/local/etc/bacula/client-myclient.conf

I will make use of this feature later on.

Lines 13-19 identify the database to be used for the Catalog. It does not specify what type of database server (e.g. PostgreSQL); that option is set at compile time.

Lines 21-31 specify how messages should get to you. This configuration assumes you have an SMTP server on your system, but you will not need it to follow this example. It is recommended if you start to depend upon your backups.

bacula-fd.conf

Here is a minimal bacula-fd.conf:

Director {
  Name     = MyBaculaDirector
  Password = "the bacula-fd password"
}

FileDaemon {
  Name             = MyFirstBaculaClient
  WorkingDirectory = /var/db/bacula
  Pid Directory    = /var/run
}

Messages {
  Name     = Standard
  director = MyBaculaDirector = all, !skipped, !restored
}

As with bacula-dir.conf, the Name and Password represent credentials. However, in this case, the Name and Password identify one Director which can contact this Director. Combined, these two values are the credentials required to communicate with this bacula-fd. It will be stored in two places:

  1. this bacula-fd.conf, so this bacula-fd knows what credentials to accept
  2. in bacula-dir.conf, so the bacula-dir knows what credentials to present when contacting this bacula-fd

I will show item #2 later.

An advanced setup can have multiple Directors. In which case, a given bacula-fd.conf can have multiple Director directives, but we won’t go there in this simple example.

bacula-sd.conf

This is our minimal bacula-sd.conf:

Storage {
  Name             = MyBaculaStorage
  WorkingDirectory = "/usr/local/bacula/working"
  Pid Directory    = "/var/run"
}

Director {
  Name     = MyBaculaDirector
  Password = "the bacula-sd password"
}

Device {
  Name           = MyFirstStorageDevice
  Media Type     = MyMediaType
  Archive Device = /usr/local/bacula/volumes
  LabelMedia     = yes
  Random Access  = yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen     = no
}

Messages {
  Name     = Standard
  director = MyBaculaDirector = all
}

I think you see the pattern now. That Name and Password identifies a Director which can contact us. That set of credentials will be stored:

  1. in bacula-sd.conf so bacula-sd knows what credentials must be presented
  2. in bacula-dir.conf so bacula-dir knows what credentials to present to this bacula-sd

As with bacula-fd.conf, we’ll get back to item #2 later.

bconsole.conf

This minimal bconsole.conf will let you interact with bacula-dir:

Director {
  Name     = MyBaculaDirector
  Password = "the bacula-dir password"

  Address  = 10.55.0.76
}

This Name and Password are the same that appear in bacula-dir.conf. These are the previously mentioned credentials required to talk to bacula-dir. This file will be read by bconsole when it starts up.

That’s just the start

In the previous sections, you have seen the bare basics. In the following sections, you’ll see how it is all tied together. I will show you how the Director knows about Clients, Jobs, and FileSets. So far, we have just been defining the foundation. Now we start building our backup system.

Client

In this section, we will add a Client to the Bacula configuration. This could be added directly to bacula-dir.conf, but I prefer to put everything specifically related to a Client in a separate file.

I am putting the following into client-myclient.conf: 

Client {
  Name           = MyFirstBaculaClient
  Password       = "the bacula-fd password"
  Address        = 10.55.0.76
  Catalog        = MyCatalog
}

To pull this configuration into bacula-dir.conf, I added the following to bacula-dir.conf:

@/usr/local/etc/bacula/client-myclient.conf

This configuration will tell bacula-dir to include the file, and within that file will be the contact details for our bacula-fd.

Remember how the credentials are stored in two places? One for the sender, one for the receiver? Here is an example to demonstrate that:

# grep "the bacula-fd password" *
bacula-fd.conf:  Password = "the bacula-fd password"
client-myclient.conf:  Password       = "the bacula-fd password"

The first instance tells bacula-fd what password to accept. The second tells bacula-dir what credentials to use when contacting that bacula-fd.

Please note: Whenever a Director contacts a Client, the credentials it must use are the Director‘s Name (not the Client‘s Name) and the Password specified in the Client resource in bacula-dir.conf. To be clear, in our example, the correct values that the Director must pass to the Client are:

  1. MyBaculaDirector
  2. the bacula-fd password

Thus, the Name in a Client resource names the client and does not a role in the credentials. This is very important.

FileSet

Now that we have told Bacula about our Client, we will specify the files we want to backup: a FileSet.

I put the following into filesets.conf.

FileSet {
  Name    = "MyFirstFileSet"
  Include {
    Options {
      signature=MD5
    }

    File = /usr/local/etc/bacula/
  }

  Exclude {
    File = *~
  }
}

In this FileSet, we are backing up one directory. Do not be confused by that name, File. I’ve chosen to backup /usr/local/etc/bacula/ because I know you’ll have that.

Each file will have its MD5 recorded. This is optional, but highly recommended. It is used when running a Verify Job to compare what you have on disk to what you backed up.

We will be excluding any files ending with a ~ (tilde).

Note:

  • one FileSet per job
  • a given FileSet can be used by more than one Job

To tell bacula-dir about this FileSet, I added this line to bacula-dir.conf:

@/usr/local/etc/bacula/filesets.conf

The next step: create a Backup Job.

Job

A Job identifies what is to backed up, the Client it is located on, and the bacula-sd which will store this backup. This is what I added to client-myclient.conf:

Job {
  Name     = "MyFirstJob"
  Client   = MyFirstBaculaClient
  Type     = "Backup"
  FileSet  = "MyFirstFileSet"
  Storage  = MyFirstStorage
  Schedule = MyFirstSchedule
  Messages = MyMessages

  Pool        = FullFile # required parameter for all Jobs, despite what appears in the next few lines

  Full         Backup Pool = FullFile
  Differential Backup Pool = DiffFile
  Incremental  Backup Pool = IncrFile
}

Now we have told Bacula nearly everything: what to backup, from where, and where to store it. Now we have to tell Bacula when to run this Job.

Schedule

A Schedule is very flexible. We will start with simple and classic. The following tells us when the Job will be run, and at what Level (Incremental, Differential, Full). I put this into schedules.conf.

Schedule {
  Name = MyFirstSchedule
  Run = Level=Full         1st     sun at 8:15
  Run = Level=Differential 2nd-5th sun at 8:15
  Run = Level=Incremental  mon-sat     at 8:15
}

I added this line to bacula-dir.conf to include the above:

@/usr/local/etc/bacula/schedules.conf

My server is on UTC time, so these will run at about 3:15 AM EST (I live just outside Philadelphia).

  1. On the first Sunday of every month, this will run a Full backup.
  2. On all other Sunday, this will run a Differential backup.
  3. On every day, we will run an Incremental backup.

The best description for Level is in the Job Resource documentation.

Storage

In that Job, we told Bacula where to send the backups: MyFirstStorage. We have already defined this Storage in bacula-sd.conf, but we have not told bacula-dir.conf about it. This may sound odd. Why do we have to specify it twice? Bacula is built upon components. In this example, everything is running on the same computer. However, they can easily run on different computers. Therefore, the bacula-dir.conf must identify the bacula-sd it can use. bacula-sd.conf might be on another computer.

Here is how I told bacula-dir about the bacula-sd it should use. I added this to bacula-dir.conf:

Storage {
  Name       = MyFirstStorage
  Address    = 10.55.0.76
  Password   = "the bacula-sd password"
  Device     = MyFirstStorageDevice
  Media Type = MyMediaType
}

Here is that concept of Name and Password again. But this time, it’s something slighting different. in this case In this case, we are specifying the credentials bacula-dir should use when contacting a bacula-sd at that Address.

The values for Device and Media Type must match those specified in bacula-sd.conf

I want to repeat something I mentioned when I added the Client resource to bacula-dir.conf. The Name of this Storage resource does not form part of any credentials. When our Director contains the bacula-sd at 10.55.0.73, it will use these credentials:

  1. MyBaculaDirector
  2. the bacula-sd password

Pools & Volumes

So far, I have glossed over Pools. It is easier to introduce Pools by describing tapes. Imagine you have a box of tapes for Full backups. Every tape in that box will be kept for 12 months. You have another box for Incremental backups. Each of those will be kept for 15 days. Another box has Differential backups, each of which is kept for 3 months.

Each of those boxes is a Pool. Each of those tapes is a Volume.

Do not confuse a Bacula Volume with a filesystem or another concept you may have of a volume. A Bacula Volume contains a backup (or part of a backup), or it might be empty.

A Pool definition is used to describe the initial and/or common characteristics of a Volume. Thus, a Pool can be thought of as a collection of Volumes which share common characteristics, such as Retention.

These are the Pool definitions I added to pools.conf:

Pool {
  Name             = FullFile
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 3 years
  Storage          = MyFirstStorage

  Maximum Volume Bytes = 5G
  Maximum Volumes      = 5

  LabelFormat = "FullAuto-"
}

Pool {
  Name             = DiffFile
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 6 weeks
  Storage          = MyFirstStorage

  Maximum Volume Bytes = 5G
  Maximum Volumes      = 5

  LabelFormat = "DiffAuto-"
}

Pool {
  Name             = IncrFile
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 3 weeks
  Storage          = MyFirstStorage

  Maximum Volume Bytes = 5G
  Maximum Volumes      = 5

  LabelFormat = "IncrAuto-"
}

This specifies three pools, one for each Level of backup, and each with different values for Retention.

Like the other separate files, this one is included via this entry in bacula-dir.conf:

@/usr/local/etc/bacula/client-myclient.conf

If you want to skip ahead to running jobs, you can ignore the net few sections.

Pool limits

You will notice that each Pool I created has two maximum settings. Bacula will keep using a Volume forever unless you tell it otherwise. I have chosen to restrict my Pools by specifying Maximum Volume Bytes. When a Volume reaches this size, Bacula will stop using it and create another. This automatic creation can occur because of these directives:

  1. the Pool has a LabelFormat
  2. the Device in bacula-sd.conf has LabelMedia set to yes

I have also specified a maximum of 5 Volumes in each Pool. This limits the size of each Pool to 25G. Clearly, these are just values for getting started. Set your values accordingly once you get things running.

There are many such options in the Pool documentation.

Retention

When you first think of Retention, most people consider how long the backup is kept. With respect to Bacula, this is not strictly the correct definition. Bacula Retention refers to the Catalog and specifies how long meta-data will be retained in the database. The Catalog records what File was backed up by what Job and the Volume is it located upon.

Bacula uses three Retention periods:

  1. File Retention
  2. Job Retention
  3. Volume Retention

File Retention and Job Retention are specified in each Client resource of bacula-dir.conf. I have not specified any for my Client, and they therefore default to 60 and 180 days, respectively.

Volume Retention is specified in the Pool definition. Of vital importance is the starting point for Volume Retention. The Volume Retention period starts when a Volume is no longer appendable. The count down for Volume Retention begins only when the Volume Status has been changed to something other than Append (e.g. Full, Used, Purged, …).

Meta-data is removed from the Catalog by Pruning. You will remember that each of my Pools has AutoPrune = yes.

Keep aware that once File Retention has passed for a particular backup (and the Catalog has been pruned), you will be unable to select the files you wish to restore from that backup. You will need to restore the entire Job, and once Job Retention (for that backup) has passed (and the Catalog has been pruned), you will be unable to restore that backup without restoring to bextract or bscan. If you have to do that, it will not be the highlight of your week.

Disk is space is cheap. Time is not. Keep your File and Job Retention set high, to make your restore easier. I think my Catalog database is about 10GB when dumped and compressed. I recommend not trying to skimp on disk space by reducing Job or File Retention unnecessarily.

I set File and Job Retention to large values. Then I let Volume Retention determine how long the meta-data is retained. Thus, I can copy the same backup to different Pools and let that Pool Volume Retention decide how long that backup will be retained.

Restoring pruned data

If Pruning has occurred, you can use bscan. I hope you never have to use it under emergency conditions. I view bscan as a tool of last resort when everything else has been lost.

Recycling

By design, Bacula avoids overwriting backups. It will do everything it can to avoid this. You must impose restrictions which indicate that Bacula is allowed to overwrite. If you do not do this, Bacula will continue appending to existing Volumes until it runs out of space. NOTE: disk space monitoring is outside scope for this post; use Nagios.

Once you impose restrictions, Recycling can occur and Bacula can then overwrite a Volume. The details of those restrictions are defined in the documentation. I will not go into Recycling here, but the examples given here are very restrictive.

Running a job

Let’s run our first job.

Oh… Umm, we have some last minute things to do.

Last minute things

The following directory is used by both bacula-dir and bacula-sd:

mkdir -p /usr/local/bacula/working
chown bacula:bacula /usr/local/bacula/working

The following log is used by bacula-dir:

touch /var/log/bacula.log
chown bacula:bacula /var/log/bacula.log

The following allows incoming database connections. You may wish to tighten this down later, and perhaps use a password. I added this entry to the end of /usr/local/pgsql/data/pg_hba.conf:

host    bacula         bacula         10.55.0.76/32           trust

This is where bacula-sd is configured to store the backups. This is where your Volumes will be stored This is set in bacula-sd.conf.

mkdir -p /usr/local/bacula/volumes
chown bacula:bacula /usr/local/bacula/volumes

Starting the daemons

With these commands, I started the Bacula daemons:

# service bacula-fd start
# service bacula-sd start
# service bacula-dir start

They can be started in any order.

Running bconsole

bconsole is your friend. You can run backups and restores with bconsole. Here is what you should see when you start it:

# bconsole
Connecting to Director 10.55.0.76:9101
1000 OK: 1 MyBaculaDirector Version: 7.0.5 (28 July 2014)
Enter a period to cancel a command.
*

If you don’t see that, you might want to verify that bacula-dir is running.

Status

In this section, we will run status on the Director, Client, and Storage.

*status director
MyBaculaDirector Version: 7.0.5 (28 July 2014) amd64-portbld-freebsd9.1 freebsd 9.1-RELEASE-p19
Daemon started 10-Jan-15 20:14. Jobs: run=0, running=0 mode=0,0
 Heap: heap=0 smbytes=253,680 max_bytes=254,453 bufs=168 max_bufs=176

Scheduled Jobs:
Level          Type     Pri  Scheduled          Job Name           Volume
===================================================================================
Differential   Backup    10  11-Jan-15 08:15    MyFirstJob         *unknown*
====

Running Jobs:
Console connected at 10-Jan-15 20:20
No Jobs running.
====
No Terminated Jobs.
====
*

You can abbreviate commands. I could have entered st di and received the same output.

*st client
Automatically selected Client: MyFirstBaculaClient
Connecting to Client MyFirstBaculaClient at 10.55.0.76:9102

MyFirstBaculaClient Version: 7.0.5 (28 July 2014)  amd64-portbld-freebsd9.1 freebsd 9.1-RELEASE-p19
Daemon started 10-Jan-15 20:14. Jobs: run=0 running=0.
 Heap: heap=0 smbytes=185,618 max_bytes=185,765 bufs=50 max_bufs=51
 Sizes: boffset_t=8 size_t=8 debug=0 trace=0 mode=0,0 bwlimit=0kB/s

Running Jobs:
Director connected at: 10-Jan-15 20:22
No Jobs running.
====

Terminated Jobs:
====
*

Because we have only one Client, bacula-dir selected that Client automatically and showed us the status. When executing the command, bacula-dir connects to bacula-fd and asks it for the status. Seeing this type of output confirms that bacula-dir had the correct credentials for that client. This situation is vital to running a job.

Let’s check the storage.

*st stor
Automatically selected Storage: MyFirstStorage
Connecting to Storage daemon MyFirstStorage at 10.55.0.76:9103

MyBaculaStorage Version: 7.0.5 (28 July 2014) amd64-portbld-freebsd9.1 freebsd 9.1-RELEASE-p19
Daemon started 10-Jan-15 20:26. Jobs: run=0, running=0.
 Heap: heap=0 smbytes=187,799 max_bytes=312,674 bufs=56 max_bufs=57
 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 mode=0,0

Running Jobs:
No Jobs running.
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
====

Device status:

Device "MyFirstStorage" (/usr/local/bacula/volumes) is not open.
==
====

Used Volume status:
====

====

*

Do not be concerned with the ‘Device “MyFirstStorage” (/usr/local/bacula/volumes) is not open‘ message. This is completely normal when a job is not underway.

Your first job

Let’s run a Job.

*run
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
A job name must be specified.
Automatically selected Job: MyFirstJob
Run Backup job
JobName:  MyFirstJob
Level:    Incremental
Client:   MyFirstBaculaClient
FileSet:  MyFirstFileSet
Pool:     FullFile (From Job resource)
Storage:  MyFirstStorage (From Pool resource)
When:     2015-01-10 20:35:08
Priority: 10
OK to run? (yes/mod/no): yes
Job queued. JobId=4
You have messages.
*m
10-Jan 20:35 MyBaculaDirector JobId 4: No prior Full backup Job record found.
10-Jan 20:35 MyBaculaDirector JobId 4: No prior or suitable Full backup found in catalog. Doing FULL backup.
*m
10-Jan 20:35 MyBaculaDirector JobId 4: Start Backup JobId 4, Job=MyFirstJob.2015-01-10_20.35.10_03
10-Jan 20:35 MyBaculaDirector JobId 4: Created new Volume="FullAuto-0001", Pool="FullFile", MediaType="MyMediaType" in catalog.
10-Jan 20:35 MyBaculaDirector JobId 4: Using Device "MyFirstStorageDevice" to write.
10-Jan 20:35 MyBaculaStorage JobId 4: Labeled new Volume "FullAuto-0001" on file device "MyFirstStorageDevice" (/usr/local/bacula/volumes).
10-Jan 20:35 MyBaculaStorage JobId 4: Wrote label to prelabeled Volume "FullAuto-0001" on file device "MyFirstStorageDevice" (/usr/local/bacula/volumes)
*m
You have no messages.
*m
You have no messages.
*m
You have no messages.
*m
10-Jan 20:35 MyBaculaStorage JobId 4: Elapsed time=00:00:10, Transfer rate=4.517 K Bytes/second
*m
10-Jan 20:35 MyBaculaDirector JobId 4: Bacula MyBaculaDirector 7.0.5 (28Jul14):
  Build OS:               amd64-portbld-freebsd9.1 freebsd 9.1-RELEASE-p19
  JobId:                  4
  Job:                    MyFirstJob.2015-01-10_20.35.10_03
  Backup Level:           Full (upgraded from Incremental)
  Client:                 "MyFirstBaculaClient" 7.0.5 (28Jul14) amd64-portbld-freebsd9.1,freebsd,9.1-RELEASE-p19
  FileSet:                "MyFirstFileSet" 2015-01-10 20:27:51
  Pool:                   "FullFile" (From Job FullPool override)
  Catalog:                "MyCatalog" (From Client resource)
  Storage:                "MyFirstStorage" (From Pool resource)
  Scheduled time:         10-Jan-2015 20:35:08
  Start time:             10-Jan-2015 20:35:13
  End time:               10-Jan-2015 20:35:23
  Elapsed time:           10 secs
  Priority:               10
  FD Files Written:       20
  SD Files Written:       20
  FD Bytes Written:       42,576 (42.57 KB)
  SD Bytes Written:       45,179 (45.17 KB)
  Rate:                   4.3 KB/s
  Software Compression:   None
  VSS:                    no
  Encryption:             no
  Accurate:               no
  Volume name(s):         FullAuto-0001
  Volume Session Id:      2
  Volume Session Time:    1420921921
  Last Volume Bytes:      46,495 (46.49 KB)
  Non-fatal FD errors:    0
  SD Errors:              0
  FD termination status:  OK
  SD termination status:  OK
  Termination:            Backup OK

10-Jan 20:35 MyBaculaDirector JobId 4: Begin pruning Jobs older than 6 months .
10-Jan 20:35 MyBaculaDirector JobId 4: No Jobs found to prune.
10-Jan 20:35 MyBaculaDirector JobId 4: Begin pruning Files.
10-Jan 20:35 MyBaculaDirector JobId 4: No Files found to prune.
10-Jan 20:35 MyBaculaDirector JobId 4: End auto prune.

*

There. Done. Your first backup!

However, see JobId 4? Yes, it took me four attempts to get the first job running. I had stuff to fix up and adjust in the configuration. You should have no such problems if you copy the configuration files, adjust the IP addresses, and run through all the last minute steps.

Here are some notes to help you understand the output. The numbers refer to the line numbers in the above output.

18 – m is short for messages, and will display any pending messages from the Director

20 – The job is promoted to Full. This happens when no Jobs exist upon which an Incremental or Differential can be based.

23 – A new Volume was created in the FullFile Pool, with the specified label format. Because we are backing up to disk, the file name will match the Volume name.

25 – The label is also stored inside the Volume.

51-52 – 20 files were saved. You might wonder, why 20 files? That’s way more than in my configuration. Let’s look into that:

[root@baculatest:/usr/local/etc/bacula] # find .
.
./INSTALLED
./INSTALLED/bacula-dir.conf
./INSTALLED/bacula-dir.conf.sample
./INSTALLED/bacula-sd.conf.sample
./INSTALLED/bacula-barcodes.sample
./INSTALLED/bacula-fd.conf
./INSTALLED/bacula-sd.conf
./INSTALLED/bacula-fd.conf.sample
./INSTALLED/bacula-barcodes
./INSTALLED/bconsole.conf
./INSTALLED/bconsole.conf.sample
./bacula-dir.conf~
./bacula-dir.conf
./filesets.conf
./pools.conf
./bacula-fd.conf
./client-myclient.conf
./schedules.conf
./bacula-sd.conf
./bconsole.conf

I moved the files installed by default into a subdirectory, INSTALLED. How many files are there?

[root@baculatest:/usr/local/etc/bacula] # find . | wc -l
      21

21 files? Only 20 backed up? Bacula ignored bacula-dir.conf~ as specified in the FileSet.

Restore

You need only one Restore Job. Restores are done manually, although you could script that. All Job parameters can be altered from within bconsole. This means you do not have to specify all possible combinations you might need.

*restore
Using Catalog "MyCatalog"
No Restore Job Resource found in bacula-dir.conf.
You must create at least one before running this command.
*

Oh. Let’s add that Restore Job to bacula-dir.conf. 

Job {
  Name     = "RestoreFiles"
  Type     = Restore
  Client   = MyFirstBaculaClient
  FileSet  = MyFirstFileSet
  Storage  = MyFirstStorage
  Schedule = MyFirstSchedule
  Messages = MyMessages
  Pool     = FullFile

  Where    = /tmp/bacula-restores
}

Even though this Job refers to MyFirstBaculaClient, I consciously did not add this Job to client-myclient.conf because it can be used for any Client.

Notice that the Restore Job contains a Where directive. This specifies a prefix to the directory name of all files being restored. This permits files to be restored in a different location from which they were backed up. If Where is not specified, the files are restored to their original location. Where can also be specified / overridden from within bconsole (see the example to follow).

I like to always specify a Where directive as a safety/sanity precaution.

Now. Let’s run that Restore.

*restore
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"

First you select one or more JobIds that contain files
to be restored. You will be presented several methods
of specifying the JobIds. Then you will be allowed to
select which files from those JobIds are to be restored.

To select the JobIds, you have the following choices:
     1: List last 20 Jobs run
     2: List Jobs where a given File is saved
     3: Enter list of comma separated JobIds to select
     4: Enter SQL list command
     5: Select the most recent backup for a client
     6: Select backup for a client before a specified time
     7: Enter a list of files to restore
     8: Enter a list of files to restore before a specified time
     9: Find the JobIds of the most recent backup for a client
    10: Find the JobIds for a backup for a client before a specified time
    11: Enter a list of directories to restore for found JobIds
    12: Select full restore to a specified Job date
    13: Cancel
Select item:  (1-13):

Of the options listed, I recommend #5 and #6. The others are great, but those are the two I use the most.

We have just one backup, so I will select #5:

Select item:  (1-13): 5
Automatically selected Client: MyFirstBaculaClient
Automatically selected FileSet: MyFirstFileSet
+-------+-------+----------+----------+---------------------+---------------+
| jobid | level | jobfiles | jobbytes | starttime           | volumename    |
+-------+-------+----------+----------+---------------------+---------------+
|     4 | F     |       20 |   42,576 | 2015-01-10 20:35:13 | FullAuto-0001 |
+-------+-------+----------+----------+---------------------+---------------+
You have selected the following JobId: 4

Building directory tree for JobId(s) 4 ...  
18 files inserted into the tree.

You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.

cwd is: /
$

Bacula has automatically selected both the Client and FileSet, since there is only one.

18 files? I thought we backed up 20 files? Yes we did, and two of those ‘files’ were directories.

At this point, we are in a special Bacula ‘shell’, where we can move around in the directories and select the files we want to restore. We could have specified restore all to automatically mark all files for restore, or we can issue the restore all command now.

I will mark all the *.conf files in /usr/local/etc/bacula

$ ls
usr/
$ cd usr/local/etc
cwd is: /usr/local/etc/
$ ls
bacula/
$ cd bacula
cwd is: /usr/local/etc/bacula/
$ ls
INSTALLED/
bacula-dir.conf
bacula-fd.conf
bacula-sd.conf
bconsole.conf
client-myclient.conf
filesets.conf
pools.conf
schedules.conf
$ mark *.conf
8 files marked.
$ done
Bootstrap records written to /usr/local/bacula/working/MyBaculaDirector.restore.2.bsr

The Job will require the following (*=>InChanger):
   Volume(s)                 Storage(s)                SD Device(s)
===========================================================================
   
    FullAuto-0001             MyFirstStorage            MyFirstStorageDevice     

Volumes marked with "*" are in the Autochanger.


8 files selected to be restored.

Run Restore job
JobName:         RestoreFiles
Bootstrap:       /usr/local/bacula/working/MyBaculaDirector.restore.2.bsr
Where:           /tmp/bacula-restores
Replace:         always
FileSet:         MyFirstFileSet
Backup Client:   MyFirstBaculaClient
Restore Client:  MyFirstBaculaClient
Storage:         MyFirstStorage
When:            2015-01-10 21:17:48
Catalog:         MyCatalog
Priority:        10
OK to run? (yes/mod/no):

You can now override any Job directive with the mod command. I will just say yes and let the files go to /tmp/bacula-restores.

OK to run? (yes/mod/no): yes
Job queued. JobId=5
*m
10-Jan 21:20 MyBaculaDirector JobId 5: Start Restore Job RestoreFiles.2015-01-10_21.20.07_04
10-Jan 21:20 MyBaculaDirector JobId 5: Using Device "MyFirstStorageDevice" to read.
*m
10-Jan 21:20 MyBaculaStorage JobId 5: Ready to read from volume "FullAuto-0001" on file device "MyFirstStorageDevice" (/usr/local/bacula/volumes).
10-Jan 21:20 MyBaculaStorage JobId 5: Forward spacing Volume "FullAuto-0001" to file:block 0:216.
10-Jan 21:20 MyBaculaStorage JobId 5: Elapsed time=00:00:01, Transfer rate=5.177 K Bytes/second
10-Jan 21:20 MyBaculaDirector JobId 5: Bacula MyBaculaDirector 7.0.5 (28Jul14):
  Build OS:               amd64-portbld-freebsd9.1 freebsd 9.1-RELEASE-p19
  JobId:                  5
  Job:                    RestoreFiles.2015-01-10_21.20.07_04
  Restore Client:         MyFirstBaculaClient
  Start time:             10-Jan-2015 21:20:09
  End time:               10-Jan-2015 21:20:09
  Files Expected:         8
  Files Restored:         8
  Bytes Restored:         4,168
  Rate:                   0.0 KB/s
  FD Errors:              0
  FD termination status:  OK
  SD termination status:  OK
  Termination:            Restore OK

10-Jan 21:20 MyBaculaDirector JobId 5: Begin pruning Jobs older than 6 months .
10-Jan 21:20 MyBaculaDirector JobId 5: No Jobs found to prune.
10-Jan 21:20 MyBaculaDirector JobId 5: Begin pruning Files.
10-Jan 21:20 MyBaculaDirector JobId 5: No Files found to prune.
10-Jan 21:20 MyBaculaDirector JobId 5: End auto prune.

*

Done. Boom. Just like that.

Let’s go look at them

# cd /tmp/bacula-restores/usr/local/etc/bacula/
# ls
bacula-dir.conf      bacula-fd.conf       bacula-sd.conf       bconsole.conf        
client-myclient.conf filesets.conf        pools.conf           schedules.conf

Let’s do a diff here

[root@baculatest:/tmp/bacula-restores/usr/local/etc/bacula] # diff . /usr/local/etc/bacula/
Only in /usr/local/etc/bacula/: INSTALLED
diff ./bacula-dir.conf /usr/local/etc/bacula/bacula-dir.conf
53a54,66
> Job {
>   Name     = "RestoreFiles"
>   Type     = Restore
>   Client   = MyFirstBaculaClient
>   FileSet  = MyFirstFileSet
>   Storage  = MyFirstStorage
>   Schedule = MyFirstSchedule
>   Messages = MyMessages
>   Pool     = FullFile
> 
>   Where    = /tmp/bacula-restores
> }
> 
Only in /usr/local/etc/bacula/: bacula-dir.conf~

That bothered me for a second when I first read it, but yes, that’s exactly correct. The Restore job was added after the initial backup was run.

But, wait! There’s more! Lots, lots, more!

You’ve just gone through a lot of stuff. There is a lot to learn about Bacula. There are many configuration options and possible choices. I like that. There is no one-way to do things. You can pick and choose.

What I’m keen to try very soon is bacula-sd to bacula-sd Job Copy. I want to store some of my backups at a remote location, without transporting the tapes there.

The rest of this section is various things I thought of while writing this article. I thought they should be included and deserved their own respective sections.

The Retention periods I use

I set my File and Job Retention values very high, higher than my Volume Retention. A Volume cannot be recycled until all the Jobs/Files it contains are pruned from the database (Catalog). By keeping File and Job Retention set to 3 years, I ensure that my Volumes will be untouched for at least 3 years or the Volume Retention period, whichever is less.

Volumes might still have your backups

Even though File, Job, and Volume Retentions have passed, and the Volume Status has changed to Purged, the backup data remains intact on that Volume. It is not until the Volume is Recycled and new data is written to the Volume that your backup data is lost.

This design consideration is part of Bacula’s absolute hesitation at overwriting your data. If you absolutely need the backup from a Purged Volume, you can get it back using the Bacula utility programs. I recommend trying bls and bextract.

You should also look into using Bootstrap Files with your Jobs because of their use with bextract.

Backups of Catalog and Configuration

In addition to backing up my Catalog dump with Bacula, I copy that dump, all Bacula configuration files, and all Bootstrap Files to multiple locations. I do this to avoid ever having to use bscan to reconstruct my Catalog from my Volumes.

Problems?

bacula-dir is usually the hardest daemon to get configured and running. Keep an eye on /var/log/bacula.log and /var/log/messages. If all else fails, try starting bacula-dir from the command line, directly:

/usr/local/sbin/bacula-dir -u bacula -g bacula -v -c /usr/local/etc/bacula/bacula-dir.conf -f -d10

Thanks for reading

The Bacula configuration files I used for this article can be downloaded from https://www.langille.org/examples/bacula7/

Bacula has provided me with a great platform for backing up my systems. It gets used every day. I backup to ZFS filesystems which have compression enabled and automatic snapshots. It’s fantastic. Every day, those backups are copied to tape. Some tapes get stored at a remote location, just in case. Hopefully I will never recover from a real disaster, but I know Bacula has saved me many time when I’ve deleted / overwitten something.

I hope it serves you well.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Leave a Comment

Scroll to Top