In this post:
- Bacula 7.4
- FreeBSD 11.1
- py-iocage 0.9.9.2
I recently wrote about Converting a Dell R710 into a ZFS-based FreeBSD 11.1 tape library bacula-sd and then about getting the drives ready. That system, henceforth referred to as the r710 is now ready for the Bacula setup.
Last night I created a jail and installed the software I’ll need. Most of this was done via Ansible. I also configured Bacula both on the server and in the jail. Tonight, I’ll be configuring the host so the jail can see and use the tape library. By default, such devices are not available from within a FreeBSD jail.
Most of what I’m doing here is covered by a previous post, Using Bacula in a jail.
The players
All certificates have been generated on my private Certificate Authority (CA). I did this using ssl-admin.
bacula-dir runs on the bacula.int.unixathome.org host (which is a jail on a FreeBSD 11.1 server, slocum).
The new bacula-sd will be running in a jail (bacula-sd-02.int.unixathome.org) on another FreeBSD 11.1 server (r710-01.int.unixathome.org).
Creating the jails will not be covered.
Adjusting the host – and doing it wrong
NOTE: I made some errors in this section, so read on if you wish, or skip to the next section for what actually worked.
In this section, I’ll be modifying /etc/devfs.rules on the host system. As described in the original article, I’ll be adding the following to /etc/devfs.rules:
[devfsrules_jail_unhide_tapes=5] [devfsrules_jail_unhide_tapes=5] add path sa0 unhide add path pass0 unhide add path pass7 unhide mode 0660 add path ch0 unhide add path nsa0 unhide add path sa1 unhide add path pass8 unhide add path pass9 unhide mode 0660 add path ch1 unhide add path nsa1 unhide [devfsrules_jail_bacula=6] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path zfs unhide add include $devfsrules_jail_unhide_tapes
You will notice that my previous post included sa0.0 and nsa0.0 but the above does not. I am not sure why, but those devices (symlinks) do not show up on FreeBSD 11.1 as they did on FreeBSD 10.3.
$ sudo camcontrol devlist <IBM ULT3580-HH4 C7Q1> at scbus1 target 4 lun 0 (pass7,sa2) <IBM 3573-TL B.60> at scbus1 target 4 lun 1 (pass8,ch0) <IBM ULT3580-HH4 C7Q1> at scbus1 target 5 lun 0 (pass9,sa3) <IBM 3573-TL B.60> at scbus1 target 5 lun 1 (pass10,ch2)
The above devices are based on this output (partially redacted to remove items not relevant to the tape library):
This host is using iocage; the original post talks about ezjail.
Here are the jails on this host:
[dan@r710-01:~] $ jls
JID IP Address Hostname Path
1 10.55.0.33 bacula-sd-02.int.unixathome.o /iocage/jails/bacula-sd-02.int.unixathome.org/root
2 10.55.0.32 pg02.int.unixathome.org /iocage/jails/pg02.int.unixathome.org/root
[dan@r710-01:~] $
We want access in the bacula-sd-02 jail. From the iocage man page, the command I want to issue is:
[dan@r710-01:~] $ sudo iocage set devfs_ruleset=6 bacula-sd-02 Property: devfs_ruleset has been updated to 6 [dan@r710-01:~] $
Now we restart devfs and the jail:
[dan@r710-01:~] $ sudo service devfs restart [dan@r710-01:~] $ sudo iocage restart bacula-sd-02 * Stopping bacula-sd-02_int_unixathome_org + Running prestop OK + Stopping services OK + Removing jail process OK + Running poststop OK * Starting bacula-sd-02_int_unixathome_org + Started OK + Starting services OK [dan@r710-01:~] $
Logging into the jail I found:
[dan@bacula-sd-02:~] $ ls -l /dev total 1 crw------- 1 root operator 0x6b Dec 15 13:56 ch0 dr-xr-xr-x 2 root wheel 512 Dec 15 13:56 fd lrwxr-xr-x 1 root wheel 14 Dec 15 20:15 log -> ../var/run/log crw-rw-rw- 1 root wheel 0x1b Dec 15 20:16 null crw------- 1 root operator 0x76 Dec 15 13:56 pass10 crw-rw---- 1 root operator 0x77 Dec 15 13:56 pass11 dr-xr-xr-x 2 root wheel 512 Dec 15 20:15 pts crw-r--r-- 1 root wheel 0x7 Dec 15 13:56 random lrwxr-xr-x 1 root wheel 4 Dec 15 20:15 stderr -> fd/2 lrwxr-xr-x 1 root wheel 4 Dec 15 20:15 stdin -> fd/0 lrwxr-xr-x 1 root wheel 4 Dec 15 20:15 stdout -> fd/1 lrwxr-xr-x 1 root wheel 6 Dec 15 20:15 urandom -> random crw-rw-rw- 1 root wheel 0x1c Dec 15 13:56 zero crw-rw-rw- 1 root operator 0x48 Dec 15 13:56 zfs [dan@bacula-sd-02:~] $
I see pass10 and pass11 but nothing related to sa devices. Fail.
Let’s go back to the host.
[dan@r710-01:~] $ ls -l /dev/pass7 /dev/nsa2 /dev/sa2 /dev/pass8 /dev/ch0 /dev/pass9 /dev/nsa3 /dev/sa3 crw------- 1 root operator 0x6b Dec 15 13:56 /dev/ch0 crw-rw---- 1 root operator 0x65 Dec 15 13:56 /dev/nsa2 crw-rw---- 1 root operator 0x69 Dec 15 13:56 /dev/nsa3 crw------- 1 root operator 0x73 Dec 15 13:56 /dev/pass7 crw------- 1 root operator 0x74 Dec 15 13:56 /dev/pass8 crw------- 1 root operator 0x75 Dec 15 13:56 /dev/pass9 crw-rw---- 1 root operator 0x64 Dec 15 13:56 /dev/sa2 crw-rw---- 1 root operator 0x68 Dec 15 13:56 /dev/sa3 [dan@r710-01:~] $
Umm, this is FreeBSD 11 and we don’t see any symlinks between sa devices and nsa devices. Ahh, let’s fix the rules and restart again.
Nope. Still didn’t work.
It took me about 20 minutes to figure out I had incorrectly added the rules to /etc/devfs.conf instead of /etc/devfs.rules.
For the record, here are the rules I’m using now:
[devfsrules_jail_unhide_tapes=5] add path pass7 unhide add path sa2 unhide add path nsa2 unhide add path pass8 unhide mode 0660 add path ch0 unhide add path pass9 unhide add path sa3 unhide add path nsa3 unhide add path pass10 unhide mode 0660 add path ch2 unhide [devfsrules_jail_bacula=6] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path zfs unhide add include $devfsrules_jail_unhide_tapes
This is what I see in the jail:
[dan@bacula-sd-02:~] $ ls -l /dev total 1 crw------- 1 root operator 0x6b Dec 15 13:56 ch0 crw------- 1 root operator 0x6c Dec 15 13:56 ch2 dr-xr-xr-x 2 root wheel 512 Dec 15 13:56 fd lrwxr-xr-x 1 root wheel 14 Dec 15 21:05 log -> ../var/run/log crw-rw---- 1 root operator 0x65 Dec 15 13:56 nsa2 crw-rw---- 1 root operator 0x69 Dec 15 13:56 nsa3 crw-rw-rw- 1 root wheel 0x1b Dec 15 21:06 null crw-rw---- 1 root operator 0x76 Dec 15 13:56 pass10 crw------- 1 root operator 0x73 Dec 15 13:56 pass7 crw-rw---- 1 root operator 0x74 Dec 15 13:56 pass8 crw------- 1 root operator 0x75 Dec 15 13:56 pass9 dr-xr-xr-x 2 root wheel 512 Dec 15 21:05 pts crw-r--r-- 1 root wheel 0x7 Dec 15 13:56 random crw-rw---- 1 root operator 0x64 Dec 15 13:56 sa2 crw-rw---- 1 root operator 0x68 Dec 15 13:56 sa3 lrwxr-xr-x 1 root wheel 4 Dec 15 21:05 stderr -> fd/2 lrwxr-xr-x 1 root wheel 4 Dec 15 21:05 stdin -> fd/0 lrwxr-xr-x 1 root wheel 4 Dec 15 21:05 stdout -> fd/1 lrwxr-xr-x 1 root wheel 6 Dec 15 21:05 urandom -> random crw-rw-rw- 1 root wheel 0x1c Dec 15 13:56 zero crw-rw-rw- 1 root operator 0x48 Dec 15 13:56 zfs [dan@bacula-sd-02:~] $
Now I could see it, but I could not use it. I kept getting these errors:
$ sudo mtx -f /dev/pass7 status cannot open SCSI device '/dev/pass7' - Operation not permitted
I was asked about kern.securelevel. It was -1, or so I thought.
I couldn’t figure it out. I posted to Twitter and Mastodon. I was getting nowhere.
Saturday I posted to the FreeBSD SCSI mailing list. When that went nowhere, I tried the FreeBSD Current mailing list.
Warner Losh gave me some commands to try out on the host, while running the command in the jail. We tracked it down to … kern.securelevel.
It seems I checked kern.securelevel on bacula-sd-01, but the server in question was bacula-sd-02.
Doh.
What?
Yeah, I was asked twice about kern.securelevel. Both times I reported the original failure. If it helps, when I was asked the second time, the server was offline because it was mid-way through moving it in the rack.
Adjusting the host – and doing it correctly
Here is what I used in /etc/devfs.rules on the jail host so that the jail could view the tape library:
[dan@r710-01:~] $ cat /etc/devfs.rules [devfsrules_jail_unhide_tapes=5] add path sa0 unhide add path pass8 unhide add path pass9 unhide mode 0660 add path ch0 unhide add path nsa0 unhide add path sa1 unhide add path pass6 unhide add path pass7 unhide mode 0660 add path ch1 unhide add path nsa1 unhide [devfsrules_jail_bacula=6] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path zfs unhide add include $devfsrules_jail_unhide_tapes [dan@r710-01:~] $
To get my jail to invoke those rules, I issued this command on the jail host and I also restarted (not shown below) that jail.
[dan@r710-01:~] $ sudo iocage set devfs_ruleset=6 bacula-sd-02 Property: devfs_ruleset has been updated to 6
By thee way, the entries in /etc/devfs.rules are based on these entries are from /boot/device.hints so I can have sa0 and sa1, instead of sa2 and sa3.
# if you get odd results, try swapping the SAS cables # sa0 <IBM ULT3580-HH4 C7Q1> at scbus1 target 5 lun 0 (pass8,sa0) hint.sa.0.at="scbus1" hint.sa.0.target="5" hint.sa.0.unit="0" # ch0 <IBM 3573-TL B.60> at scbus1 target 5 lun 1 (pass9,ch0) hint.ch.0.at="scbus1" hint.ch.0.target="5" hint.ch.0.unit="1" # sa1 <IBM ULT3580-HH4 C7Q1> at scbus1 target 4 lun 0 (pass6,sa1) hint.sa.1.at="scbus1" hint.sa.1.target="4" hint.sa.1.unit="0" # ch1 <IBM 3573-TL B.60> at scbus1 target 4 lun 1 (pass7,ch1) hint.ch.1.at="scbus1" hint.ch.1.target="4" hint.ch.1.unit="1"
The above entries closely match what you see in this output with respect to scbus, target, and lun:
[dan@r710-01:~] $ sudo camcontrol devlist | egrep 'sa|ch' <IBM ULT3580-HH4 C7Q1> at scbus1 target 4 lun 0 (pass6,sa1) <IBM 3573-TL B.60> at scbus1 target 4 lun 1 (pass7,ch1) <IBM ULT3580-HH4 C7Q1> at scbus1 target 5 lun 0 (pass8,sa0) <IBM 3573-TL B.60> at scbus1 target 5 lun 1 (pass9,ch0)
You should carefully compare the above information (from the jail host) with what you see within the jail. Here is what I saw:
[dan@bacula-sd-02:~] $ ls -l /dev total 1 crw------- 1 root operator 0x6c Dec 21 17:28 ch0 crw------- 1 root operator 0x6b Dec 21 17:28 ch1 dr-xr-xr-x 2 root wheel 512 Dec 21 17:28 fd lrwxr-xr-x 1 root wheel 14 Dec 22 13:51 log -> ../var/run/log crw-rw---- 1 root operator 0x69 Dec 22 07:46 nsa0 crw-rw---- 1 root operator 0x65 Dec 22 09:55 nsa1 crw-rw-rw- 1 root wheel 0x1b Dec 22 13:51 null crw------- 1 root operator 0x73 Dec 21 17:28 pass6 crw-rw---- 1 root operator 0x74 Dec 21 17:28 pass7 crw------- 1 root operator 0x75 Dec 21 17:28 pass8 crw-rw---- 1 root operator 0x76 Dec 21 17:28 pass9 dr-xr-xr-x 2 root wheel 512 Dec 22 13:51 pts crw-r--r-- 1 root wheel 0x7 Dec 21 17:28 random crw-rw---- 1 root operator 0x68 Dec 21 17:28 sa0 crw-rw---- 1 root operator 0x64 Dec 21 17:28 sa1 lrwxr-xr-x 1 root wheel 4 Dec 22 13:51 stderr -> fd/2 lrwxr-xr-x 1 root wheel 4 Dec 22 13:51 stdin -> fd/0 lrwxr-xr-x 1 root wheel 4 Dec 22 13:51 stdout -> fd/1 lrwxr-xr-x 1 root wheel 6 Dec 22 13:51 urandom -> random crw-rw-rw- 1 root wheel 0x1c Dec 21 17:28 zero crw-rw-rw- 1 root operator 0x48 Dec 21 17:28 zfs [dan@bacula-sd-02:~] $
I have highlighting the devices relevant to my installation. Note that you must have the nsa devices despite the fact that they not not appear in the camcontrol devlist output.
Drive zero != sa0?
If you find that drive zero is not sa0, this can be fixed by changing the entries in /boot/device.hints: swap sa0 and sa1 around (along with their respective pass devices). That’s what I had to do.
For example, if you attempt to load a tape into drive 1, but it shows up in drive 0, you can fix that with /boot/device.hints.
Also, swapping the SAS cables might work if you are not using /boot/device.hints
Bacula configuration
bacula-dir.conf
Here is the declaration for bacula-sd-01 as found in bacula-dir.conf:
# Definition of bacula-sd-02 with attached TL4000 2x LTO-4 drives
Storage {
Name = "bacula-sd-02"
Address = bacula-sd-02.int.unixathome.org
SDPort = 9103
Password = "[redacted password, same as redacted in bacula-sd.conf shown below]"
AutoChanger = Yes
Device = bacula-sd-02
Media Type = LTO4
Maximum Concurrent Jobs = 10
TLS Enable = yes
TLS Require = no
TLS CA Certificate File = /usr/local/bacula/certificates/ca.crt
TLS Certificate = /usr/local/bacula/certificates/client.crt
TLS Key = /usr/local/bacula/certificates/client.key
}
bacula-sd.conf
Here is the big stuff from bacula-sd.conf on bacula-sd-02:
Storage {
Name = "bacula-sd-02"
WorkingDirectory = "/usr/local/bacula/working"
Pid Directory = "/var/run"
Maximum Concurrent Jobs = 20
TLS Enable = no
TLS Require = no
# TLS Verify Peer = yes
TLS CA Certificate File = /usr/local/etc/ssl/BSDCabalHeadquarters.crt
TLS Certificate = /usr/local/etc/ssl/bacula-sd-02.int.unixathome.org.crt
TLS Key = /usr/local/etc/ssl/bacula-sd-02.int.unixathome.org.nopassword.key
}
Director {
Name = "bacula-dir"
Password = "[redacted password from bacula-dir.conf]"
TLS Enable = yes
TLS Require = yes
TLS Verify Peer = yes
# Allow only the Director to connect
TLS Allowed CN = "bacula.unixathome.org"
TLS CA Certificate File = /usr/local/etc/ssl/BSDCabalHeadquarters.crt
TLS Certificate = /usr/local/etc/ssl/bacula-sd-02.int.unixathome.org.crt
TLS Key = /usr/local/etc/ssl/bacula-sd-02.int.unixathome.org.nopassword.key
}
Messages {
Name = Standard
director = bacula-dir = all
}
Autochanger {
Name = "bacula-sd-02"
Device = LTO_0, LTO_1
Description = "Dell PowerVault TL4000 Tape Library"
Changer Device = /dev/pass7
Changer Command = "/usr/local/sbin/mtx-changer %c %o %S %a %d"
}
Device {
Name = "LTO_0"
Description = "LTO first drive"
Media Type = LTO4
Archive Device = /dev/nsa0
Autochanger = yes
Drive Index = 0
Offline On Unmount = no
Hardware End of Medium = no
BSF at EOM = yes
Fast Forward Space File = yes
TWO EOF = yes
AutomaticMount = yes
AlwaysOpen = yes
RemovableMedia = yes
RandomAccess = no
Spool Directory = /usr/local/bacula/spooling
Maximum Spool Size = 57445187584
Maximum Job Spool Size = 57445187584
}
Device {
Name = "LTO_1"
Description = "LTO second drive"
Media Type = LTO4
Archive Device = /dev/nsa1
Autochanger = yes
Drive Index = 1
Offline On Unmount = no
Hardware End of Medium = no
BSF at EOM = yes
Backward Space Record = no
Fast Forward Space File = yes
TWO EOF = yes
AutomaticMount = yes
AlwaysOpen = yes
RemovableMedia = yes
RandomAccess = no
Spool Directory = /usr/local/bacula/spooling
Maximum Spool Size = 57445187584
Maximum Job Spool Size = 57445187584
}
You’ll see bacula.unixathome.org specified in the TLS Allowed CN clause above. The CN (bacula.unixathome.org) differs from the hostname (bacula.int.unixathome.org) upon which bacula-dir runs. Don’t let that distract you. Generally though, the hostname and the CN (Common Name) within the certificate match.
Upgrade later
After the above work was completed, all bacula-dir and bacula-sd were upgraded from 7.4.7 to 9.0.6 (NOTE: you must keep all bacula-sd and bacula-dir on the same version. Always. bacula-fd can be the same version [as bacula-dir/bacula-sd] or older; it can never be newer).
The above configuration was not affected by this upgrade.
Hope all this helps. Best of luck.
