Dec 262017
 

In this post:

  1. Bacula 7.4
  2. FreeBSD 11.1
  3. py-iocage 0.9.9.2

I recently wrote about Converting a Dell R710 into a ZFS-based FreeBSD 11.1 tape library bacula-sd and then about getting the drives ready. That system, henceforth referred to as the r710 is now ready for the Bacula setup.

Last night I created a jail and installed the software I’ll need. Most of this was done via Ansible. I also configured Bacula both on the server and in the jail. Tonight, I’ll be configuring the host so the jail can see and use the tape library. By default, such devices are not available from within a FreeBSD jail.

Most of what I’m doing here is covered by a previous post, Using Bacula in a jail.

The players

All certificates have been generated on my private Certificate Authority (CA). I did this using ssl-admin.

bacula-dir runs on the bacula.int.unixathome.org host (which is a jail on a FreeBSD 11.1 server, slocum).

The new bacula-sd will be running in a jail (bacula-sd-02.int.unixathome.org) on another FreeBSD 11.1 server (r710-01.int.unixathome.org).

Creating the jails will not be covered.

Adjusting the host – and doing it wrong

NOTE: I made some errors in this section, so read on if you wish, or skip to the next section for what actually worked.

In this section, I’ll be modifying /etc/devfs.rules on the host system. As described in the original article, I’ll be adding the following to /etc/devfs.rules:

[devfsrules_jail_unhide_tapes=5]
[devfsrules_jail_unhide_tapes=5]
add path sa0    unhide
add path pass0  unhide
add path pass7  unhide mode 0660
add path ch0    unhide
add path nsa0   unhide

add path sa1    unhide
add path pass8  unhide
add path pass9  unhide mode 0660
add path ch1    unhide
add path nsa1   unhide

[devfsrules_jail_bacula=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path zfs unhide
add include $devfsrules_jail_unhide_tapes

You will notice that my previous post included sa0.0 and nsa0.0 but the above does not. I am not sure why, but those devices (symlinks) do not show up on FreeBSD 11.1 as they did on FreeBSD 10.3.

$ sudo camcontrol devlist
<IBM ULT3580-HH4 C7Q1>             at scbus1 target 4 lun 0 (pass7,sa2)
<IBM 3573-TL B.60>                 at scbus1 target 4 lun 1 (pass8,ch0)
<IBM ULT3580-HH4 C7Q1>             at scbus1 target 5 lun 0 (pass9,sa3)
<IBM 3573-TL B.60>                 at scbus1 target 5 lun 1 (pass10,ch2)

The above devices are based on this output (partially redacted to remove items not relevant to the tape library):

This host is using iocage; the original post talks about ezjail.

Here are the jails on this host:

[dan@r710-01:~] $ jls
   JID  IP Address      Hostname                      Path
     1  10.55.0.33      bacula-sd-02.int.unixathome.o /iocage/jails/bacula-sd-02.int.unixathome.org/root
     2  10.55.0.32      pg02.int.unixathome.org       /iocage/jails/pg02.int.unixathome.org/root
[dan@r710-01:~] $ 

We want access in the bacula-sd-02 jail. From the iocage man page, the command I want to issue is:

[dan@r710-01:~] $ sudo iocage set devfs_ruleset=6 bacula-sd-02
Property: devfs_ruleset has been updated to 6
[dan@r710-01:~] $ 

Now we restart devfs and the jail:

[dan@r710-01:~] $ sudo service devfs restart
[dan@r710-01:~] $ sudo iocage restart bacula-sd-02
* Stopping bacula-sd-02_int_unixathome_org
  + Running prestop OK
  + Stopping services OK
  + Removing jail process OK
  + Running poststop OK
* Starting bacula-sd-02_int_unixathome_org
  + Started OK
  + Starting services OK
[dan@r710-01:~] $ 

Logging into the jail I found:

[dan@bacula-sd-02:~] $ ls -l /dev
total 1
crw-------  1 root  operator  0x6b Dec 15 13:56 ch0
dr-xr-xr-x  2 root  wheel      512 Dec 15 13:56 fd
lrwxr-xr-x  1 root  wheel       14 Dec 15 20:15 log -> ../var/run/log
crw-rw-rw-  1 root  wheel     0x1b Dec 15 20:16 null
crw-------  1 root  operator  0x76 Dec 15 13:56 pass10
crw-rw----  1 root  operator  0x77 Dec 15 13:56 pass11
dr-xr-xr-x  2 root  wheel      512 Dec 15 20:15 pts
crw-r--r--  1 root  wheel      0x7 Dec 15 13:56 random
lrwxr-xr-x  1 root  wheel        4 Dec 15 20:15 stderr -> fd/2
lrwxr-xr-x  1 root  wheel        4 Dec 15 20:15 stdin -> fd/0
lrwxr-xr-x  1 root  wheel        4 Dec 15 20:15 stdout -> fd/1
lrwxr-xr-x  1 root  wheel        6 Dec 15 20:15 urandom -> random
crw-rw-rw-  1 root  wheel     0x1c Dec 15 13:56 zero
crw-rw-rw-  1 root  operator  0x48 Dec 15 13:56 zfs
[dan@bacula-sd-02:~] $ 

I see pass10 and pass11 but nothing related to sa devices. Fail.

Let’s go back to the host.

[dan@r710-01:~] $ ls -l /dev/pass7 /dev/nsa2 /dev/sa2 /dev/pass8 /dev/ch0 /dev/pass9 /dev/nsa3 /dev/sa3
crw-------  1 root  operator  0x6b Dec 15 13:56 /dev/ch0
crw-rw----  1 root  operator  0x65 Dec 15 13:56 /dev/nsa2
crw-rw----  1 root  operator  0x69 Dec 15 13:56 /dev/nsa3
crw-------  1 root  operator  0x73 Dec 15 13:56 /dev/pass7
crw-------  1 root  operator  0x74 Dec 15 13:56 /dev/pass8
crw-------  1 root  operator  0x75 Dec 15 13:56 /dev/pass9
crw-rw----  1 root  operator  0x64 Dec 15 13:56 /dev/sa2
crw-rw----  1 root  operator  0x68 Dec 15 13:56 /dev/sa3
[dan@r710-01:~] $ 

Umm, this is FreeBSD 11 and we don’t see any symlinks between sa devices and nsa devices. Ahh, let’s fix the rules and restart again.

Nope. Still didn’t work.

It took me about 20 minutes to figure out I had incorrectly added the rules to /etc/devfs.conf instead of /etc/devfs.rules.

For the record, here are the rules I’m using now:

[devfsrules_jail_unhide_tapes=5]
add path pass7  unhide
add path sa2    unhide
add path nsa2   unhide
add path pass8  unhide mode 0660
add path ch0    unhide
add path pass9  unhide
add path sa3    unhide
add path nsa3   unhide
add path pass10 unhide mode 0660
add path ch2    unhide

[devfsrules_jail_bacula=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path zfs unhide
add include $devfsrules_jail_unhide_tapes

This is what I see in the jail:

[dan@bacula-sd-02:~] $ ls -l /dev
total 1
crw-------  1 root  operator  0x6b Dec 15 13:56 ch0
crw-------  1 root  operator  0x6c Dec 15 13:56 ch2
dr-xr-xr-x  2 root  wheel      512 Dec 15 13:56 fd
lrwxr-xr-x  1 root  wheel       14 Dec 15 21:05 log -> ../var/run/log
crw-rw----  1 root  operator  0x65 Dec 15 13:56 nsa2
crw-rw----  1 root  operator  0x69 Dec 15 13:56 nsa3
crw-rw-rw-  1 root  wheel     0x1b Dec 15 21:06 null
crw-rw----  1 root  operator  0x76 Dec 15 13:56 pass10
crw-------  1 root  operator  0x73 Dec 15 13:56 pass7
crw-rw----  1 root  operator  0x74 Dec 15 13:56 pass8
crw-------  1 root  operator  0x75 Dec 15 13:56 pass9
dr-xr-xr-x  2 root  wheel      512 Dec 15 21:05 pts
crw-r--r--  1 root  wheel      0x7 Dec 15 13:56 random
crw-rw----  1 root  operator  0x64 Dec 15 13:56 sa2
crw-rw----  1 root  operator  0x68 Dec 15 13:56 sa3
lrwxr-xr-x  1 root  wheel        4 Dec 15 21:05 stderr -> fd/2
lrwxr-xr-x  1 root  wheel        4 Dec 15 21:05 stdin -> fd/0
lrwxr-xr-x  1 root  wheel        4 Dec 15 21:05 stdout -> fd/1
lrwxr-xr-x  1 root  wheel        6 Dec 15 21:05 urandom -> random
crw-rw-rw-  1 root  wheel     0x1c Dec 15 13:56 zero
crw-rw-rw-  1 root  operator  0x48 Dec 15 13:56 zfs
[dan@bacula-sd-02:~] $ 

Now I could see it, but I could not use it. I kept getting these errors:

$ sudo mtx -f /dev/pass7 status
cannot open SCSI device '/dev/pass7' - Operation not permitted

I was asked about kern.securelevel. It was -1, or so I thought.

I couldn’t figure it out. I posted to Twitter and Mastodon. I was getting nowhere.

Saturday I posted to the FreeBSD SCSI mailing list. When that went nowhere, I tried the FreeBSD Current mailing list.

Warner Losh gave me some commands to try out on the host, while running the command in the jail. We tracked it down to … kern.securelevel.

It seems I checked kern.securelevel on bacula-sd-01, but the server in question was bacula-sd-02.

Doh.

What?

Yeah, I was asked twice about kern.securelevel. Both times I reported the original failure. If it helps, when I was asked the second time, the server was offline because it was mid-way through moving it in the rack.

Adjusting the host – and doing it correctly

Here is what I used in /etc/devfs.rules on the jail host so that the jail could view the tape library:

[dan@r710-01:~] $ cat /etc/devfs.rules
[devfsrules_jail_unhide_tapes=5]
add path sa0    unhide
add path pass8  unhide
add path pass9  unhide mode 0660
add path ch0    unhide
add path nsa0   unhide

add path sa1    unhide
add path pass6  unhide
add path pass7  unhide mode 0660
add path ch1    unhide
add path nsa1   unhide

[devfsrules_jail_bacula=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path zfs unhide
add include $devfsrules_jail_unhide_tapes
[dan@r710-01:~] $ 

To get my jail to invoke those rules, I issued this command on the jail host and I also restarted (not shown below) that jail.

[dan@r710-01:~] $ sudo iocage set devfs_ruleset=6 bacula-sd-02
Property: devfs_ruleset has been updated to 6

By thee way, the entries in /etc/devfs.rules are based on these entries are from /boot/device.hints so I can have sa0 and sa1, instead of sa2 and sa3.

# if you get odd results, try swapping the SAS cables

# sa0 <IBM ULT3580-HH4 C7Q1>             at scbus1 target 5 lun 0 (pass8,sa0)
hint.sa.0.at="scbus1"
hint.sa.0.target="5"
hint.sa.0.unit="0"

# ch0 <IBM 3573-TL B.60>                 at scbus1 target 5 lun 1 (pass9,ch0)
hint.ch.0.at="scbus1"
hint.ch.0.target="5"
hint.ch.0.unit="1"

# sa1 <IBM ULT3580-HH4 C7Q1>             at scbus1 target 4 lun 0 (pass6,sa1)
hint.sa.1.at="scbus1"
hint.sa.1.target="4"
hint.sa.1.unit="0"

# ch1 <IBM 3573-TL B.60>                 at scbus1 target 4 lun 1 (pass7,ch1)
hint.ch.1.at="scbus1"
hint.ch.1.target="4"
hint.ch.1.unit="1"

The above entries closely match what you see in this output with respect to scbus, target, and lun:

[dan@r710-01:~] $ sudo camcontrol devlist | egrep 'sa|ch'
<IBM ULT3580-HH4 C7Q1>             at scbus1 target 4 lun 0 (pass6,sa1)
<IBM 3573-TL B.60>                 at scbus1 target 4 lun 1 (pass7,ch1)
<IBM ULT3580-HH4 C7Q1>             at scbus1 target 5 lun 0 (pass8,sa0)
<IBM 3573-TL B.60>                 at scbus1 target 5 lun 1 (pass9,ch0)

You should carefully compare the above information (from the jail host) with what you see within the jail. Here is what I saw:

[dan@bacula-sd-02:~] $ ls -l /dev
total 1
crw-------  1 root  operator  0x6c Dec 21 17:28 ch0
crw-------  1 root  operator  0x6b Dec 21 17:28 ch1
dr-xr-xr-x  2 root  wheel      512 Dec 21 17:28 fd
lrwxr-xr-x  1 root  wheel       14 Dec 22 13:51 log -> ../var/run/log
crw-rw----  1 root  operator  0x69 Dec 22 07:46 nsa0
crw-rw----  1 root  operator  0x65 Dec 22 09:55 nsa1
crw-rw-rw-  1 root  wheel     0x1b Dec 22 13:51 null
crw-------  1 root  operator  0x73 Dec 21 17:28 pass6
crw-rw----  1 root  operator  0x74 Dec 21 17:28 pass7
crw-------  1 root  operator  0x75 Dec 21 17:28 pass8
crw-rw----  1 root  operator  0x76 Dec 21 17:28 pass9
dr-xr-xr-x  2 root  wheel      512 Dec 22 13:51 pts
crw-r--r--  1 root  wheel      0x7 Dec 21 17:28 random
crw-rw----  1 root  operator  0x68 Dec 21 17:28 sa0
crw-rw----  1 root  operator  0x64 Dec 21 17:28 sa1
lrwxr-xr-x  1 root  wheel        4 Dec 22 13:51 stderr -> fd/2
lrwxr-xr-x  1 root  wheel        4 Dec 22 13:51 stdin -> fd/0
lrwxr-xr-x  1 root  wheel        4 Dec 22 13:51 stdout -> fd/1
lrwxr-xr-x  1 root  wheel        6 Dec 22 13:51 urandom -> random
crw-rw-rw-  1 root  wheel     0x1c Dec 21 17:28 zero
crw-rw-rw-  1 root  operator  0x48 Dec 21 17:28 zfs
[dan@bacula-sd-02:~] $ 

I have highlighting the devices relevant to my installation. Note that you must have the nsa devices despite the fact that they not not appear in the camcontrol devlist output.

Drive zero != sa0?

If you find that drive zero is not sa0, this can be fixed by changing the entries in /boot/device.hints: swap sa0 and sa1 around (along with their respective pass devices). That’s what I had to do.

For example, if you attempt to load a tape into drive 1, but it shows up in drive 0, you can fix that with /boot/device.hints.

Also, swapping the SAS cables might work if you are not using /boot/device.hints

Bacula configuration

bacula-dir.conf

Here is the declaration for bacula-sd-01 as found in bacula-dir.conf:

# Definition of bacula-sd-02 with attached TL4000 2x LTO-4 drives
Storage {
  Name        = "bacula-sd-02"
  Address     = bacula-sd-02.int.unixathome.org
  SDPort      = 9103
  Password    = "[redacted password, same as redacted in bacula-sd.conf shown below]"

  AutoChanger = Yes

  Device      = bacula-sd-02
  Media Type  = LTO4

  Maximum Concurrent Jobs = 10

  TLS Enable      = yes
  TLS Require     = no
  TLS CA Certificate File = /usr/local/bacula/certificates/ca.crt

  TLS Certificate = /usr/local/bacula/certificates/client.crt
  TLS Key         = /usr/local/bacula/certificates/client.key
}

bacula-sd.conf

Here is the big stuff from bacula-sd.conf on bacula-sd-02:

Storage {
  Name                      = "bacula-sd-02"
  WorkingDirectory          = "/usr/local/bacula/working"
  Pid Directory             = "/var/run"
  Maximum Concurrent Jobs   = 20

  TLS Enable      = no
  TLS Require     = no
#  TLS Verify Peer = yes

  TLS CA Certificate File = /usr/local/etc/ssl/BSDCabalHeadquarters.crt

  TLS Certificate = /usr/local/etc/ssl/bacula-sd-02.int.unixathome.org.crt
  TLS Key         = /usr/local/etc/ssl/bacula-sd-02.int.unixathome.org.nopassword.key
}

Director {
  Name     = "bacula-dir"
  Password = "[redacted password from bacula-dir.conf]"

  TLS Enable      = yes
  TLS Require     = yes
  TLS Verify Peer = yes

  # Allow only the Director to connect
  TLS Allowed CN          = "bacula.unixathome.org"
  TLS CA Certificate File = /usr/local/etc/ssl/BSDCabalHeadquarters.crt

  TLS Certificate = /usr/local/etc/ssl/bacula-sd-02.int.unixathome.org.crt
  TLS Key         = /usr/local/etc/ssl/bacula-sd-02.int.unixathome.org.nopassword.key
}

Messages {
  Name     = Standard
  director = bacula-dir = all
}

Autochanger {
  Name            = "bacula-sd-02"
  Device          = LTO_0, LTO_1
  Description     = "Dell PowerVault TL4000 Tape Library"
  Changer Device  = /dev/pass7
  Changer Command = "/usr/local/sbin/mtx-changer %c %o %S %a %d"
}

Device {
  Name                    = "LTO_0"
  Description             = "LTO first drive"
  Media Type              = LTO4
  Archive Device          = /dev/nsa0
        
  Autochanger             = yes
  Drive Index             = 0
            
  Offline On Unmount      = no
  Hardware End of Medium  = no
  BSF at EOM              = yes
  Fast Forward Space File = yes
  TWO EOF                 = yes

  AutomaticMount          = yes
  AlwaysOpen              = yes
  RemovableMedia          = yes
  RandomAccess            = no

  Spool Directory         = /usr/local/bacula/spooling
  Maximum Spool Size      = 57445187584
  Maximum Job Spool Size  = 57445187584
}

Device {
  Name                    = "LTO_1"
  Description             = "LTO second drive"
  Media Type              = LTO4
  Archive Device          = /dev/nsa1
        
  Autochanger             = yes
  Drive Index             = 1
            
  Offline On Unmount      = no
  Hardware End of Medium  = no
  BSF at EOM              = yes
  Backward Space Record   = no
  Fast Forward Space File = yes
  TWO EOF                 = yes

  AutomaticMount          = yes
  AlwaysOpen              = yes
  RemovableMedia          = yes
  RandomAccess            = no

  Spool Directory         = /usr/local/bacula/spooling
  Maximum Spool Size      = 57445187584
  Maximum Job Spool Size  = 57445187584
}

You’ll see bacula.unixathome.org specified in the TLS Allowed CN clause above. The CN (bacula.unixathome.org) differs from the hostname (bacula.int.unixathome.org) upon which bacula-dir runs. Don’t let that distract you. Generally though, the hostname and the CN (Common Name) within the certificate match.

Upgrade later

After the above work was completed, all bacula-dir and bacula-sd were upgraded from 7.4.7 to 9.0.6 (NOTE: you must keep all bacula-sd and bacula-dir on the same version. Always. bacula-fd can be the same version [as bacula-dir/bacula-sd] or older; it can never be newer).

The above configuration was not affected by this upgrade.

Hope all this helps. Best of luck.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive