February 8th, 2010
Bacula is a backup system with three different retention periods:
I set all three to the same value.
It is important to note that these values affect only the Catalog. It determines how long a given record is held in the database. For example, if you set File Retention to be 45 days, it means that there are no records in the database for any files backed up in any jobs more than 45 days ago.
The result of this means you can’t restore a given file after the File retention passed. If the Job retention has not passed, you can still restore the whole job.
Volumes are eligible for recycling once ALL retention periods have passed.
Symptoms of File Retention < Job Retention look like this (given that you are trying to restore after File Retention has passed, but before Job Retention has passed):
Hi there. Using 2.4.4 under Debian with PostgreSQL backend and file volumes. I want to restore all files from a client, but not all files are found, however the last full backup has not been purged from the database, and saved 224000 files for 50 GB, and only 18000 files can be marked with the console in restore mode. What could be wrong ?
The problem is there are no file Retention records for the FULL backup, only for the differential and incremental backups.
Be aware.
Posted in Bacula, General, Open Source | No Comments »
February 6th, 2010
I have about 14,000 data points collected from backups. I started in Oct 2007, but have not done anything with the data. I’d like your help.
I have put the data into a PostgreSQL database and dumped it. The dump is here: http://langille.org/tmp/dlt_stats.sql.gz
The table looks like:
$ psql dan
psql (8.4.2)
Type "help" for help.
dan=# \d dlt_stats
Table "public.dlt_stats"
Column | Type | Modifiers
-------------+-----------------------------+--------------------------------------------------------
id | integer | not null default nextval('dlt_stats_id_seq'::regclass)
backup_date | timestamp without time zone | not null
dlt_id | text | not null
errors | integer | not null
dan=#
The dlt_id is the label on the tape. The errors field presents the number of correctable errors per GB for that backup job. I run a script after each Bacula backup job which grabs the data from the SCSI device. I then toss that data into a flat file, which I have since imported into PostgreSQL
What I’d like is graphs on a webpage. Use whatever tools you want. I’d like to be able to select a label, and see the results. I think a line graph is best for this, but you do what you prefer.
Bonus points for pretty, simple, and elegant. Your results will be published unless otherwise requested by you. I’ll use the results of the graphs to decide what tapes to toss, what tapes to use for full backups etc.
Thank you.
Posted in Open Source, PostgreSQL | 2 Comments »
February 5th, 2010
I’m thinking about creating a storage box. One with a lot of disk space. It will be used for backups and stuff. I will continue to backup to tape as well.
I’m considering this case, which has 8xSATA hot-swap bays:
If I was buying a RAID card, I’d buy this one:
But I may go with ZFS instead of hardware RAID.
Posted in General | 5 Comments »
January 23rd, 2010
Both BSDCan and PGCon use Pentabarf for accepting and reviewing proposals for talks. An issue raised in 2009 highlighted the ability to see the speaker name when review the list of submissions. It is a bit of very useful information.
This year, I’ve found out that we can see that vital data. Provided the speaker role has been set to confirmed. I think Pentabarf design assumes that you have created a talk, and then try to find speakers. The default value of speaker role is ‘offer’. It is not until the role is set to ‘confirmed’ that the speaker name starts to turn up in lists. I can see the logic behind this choice. However, it does not suit the way we work.
Fortunately, with SQL, there is a practical solution.
UPDATE event_person
SET event_role_state = ‘confirmed’
WHERE event_id IN (SELECT event_id FROM event WHERE conference_id = XXXX
AND event_role = ’speaker’;
where XXXX is the conference you wish to adjust.
I recommend wrapping the above in a transaction and reviewing your chance with the following query before you commit:
SELECT EP.*
FROM event_person EP, event E
WHERE EP.event_id = E.event_id
AND E.conference_id=XXXX
AND event_role = ’speaker’;
Hope that helps.
Posted in BSDCan, Bacula, Conferences, FreeBSD, FreshPorts, Mountain Bikes, Open Source, PGCon, PostgreSQL | No Comments »
January 22nd, 2010
Seen this?
/libexec/ld-elf.so.1: Shared object "libcroco-0.6.so.3" not found, required by " msgfmt"
Rebuild gettext.
portupgrade -f gettext
And try again
Port details, not relevant:
# less /var/db/ports/postgresql73/options
# This file is auto-generated by 'make config'.
# No user-servicable parts inside!
# Options for postgresql-server-7.3.21_1
_OPTIONS_READ=postgresql-server-7.3.21_1
WITH_NLS=true
WITHOUT_PAM=true
WITHOUT_MIT_KRB5=true
WITHOUT_HEIMDAL_KRB5=true
WITHOUT_OPTIMIZED_CFLAGS=true
WITHOUT_LIBC_R=true
WITHOUT_TESTS=true
WITHOUT_DEBUG=true
The building errors:
[root@pg73 /usr/ports/databases/postgresql73-server]# ldd `which msgfmt `
/usr/local/bin/msgfmt:
libgettextsrc-0.17.so => /usr/local/lib/libgettextsrc-0.17.so (0x800655000)
libgettextlib-0.17.so => /usr/local/lib/libgettextlib-0.17.so (0x80078f000)
libcroco-0.6.so.3 => not found (0x0)
libxml2.so.5 => not found (0x0)
libz.so.4 => /lib/libz.so.4 (0x800979000)
libm.so.5 => /lib/libm.so.5 (0x800a8d000)
libglib-2.0.so.0 => not found (0x0)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x800bac000)
libpcre.so.0 => not found (0x0)
libncurses.so.7 => /lib/libncurses.so.7 (0x800cb5000)
libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x800e01000)
libc.so.7 => /lib/libc.so.7 (0x800ffa000)
libcroco-0.6.so.3 => not found (0x0)
libxml2.so.5 => not found (0x0)
libglib-2.0.so.0 => not found (0x0)
libpcre.so.0 => not found (0x0)
[root@pg73 /usr/ports/databases/postgresql73-server]#
Posted in General, Open Source, PostgreSQL | No Comments »
January 17th, 2010
NOTE: this problem may not be a problem. After rebooting the system while having the tape library powered up, the problem went away. This may or may not be coincident.
Note to self: mtx-changer needs more customization than I thought.
Today showed great progress in completing the integration of my tape library into my Bacula system. I have been writing the testing and configuration of supporting tools. I had moved onto a new article outlining the integration phase (still under construction). That’s when I found a pretty big block in the mtx-changer script. It is this function:
#
# The purpose of this function to wait a maximum
# time for the drive. It will
# return as soon as the drive is ready, or after
# waiting a maximum of 300 seconds.
# Note, this is very system dependent, so if you are
# not running on Linux, you will probably need to
# re-write it, or at least change the grep target.
# We've attempted to get the appropriate OS grep targets
# in the code at the top of this script.
#
wait_for_drive() {
i=0
while [ $i -le 300 ]; do # Wait max 300 seconds
if mt -f $1 status 2>&1 | grep "${ready}" >/dev/null 2>&1; then
break
fi
debug "Device $1 - not ready, retrying..."
sleep 1
i=`expr $i + 1`
done
}
The problem is, when bacula-sd is running, you cannot get the status of the tape drive. At least with my system. It is *always* busy. This, this function always times out.
For example:
$ sudo mtx -f /dev/nsa1 status
cannot open SCSI device '/dev/nsa1' - Device busy
My solution, yet to be tested:
wait_for_drive() {
debug "waiting for Device $1"
sleep 25
}
We’ll see.
Posted in Bacula, Open Source | No Comments »
January 10th, 2010
These are the places we’re looking at to hire a mountain bike for the upcoming trip:
Posted in General | No Comments »
January 5th, 2010
Pretty sure I will be in New Zealand for Easter. Probably Wellington. Going to rent the bike there, rather than take mine with me.
I’ll be asking the boss for the time off tomorrow.
Posted in General | No Comments »
December 30th, 2009
I noticed a difference in the regression testing output between two systems today.
cmake-2.6.4
Stopping the Bacula File daemon
Stopping the Bacula Storage daemon
Stopping the Bacula Director daemon
===== pretest OK 00:00:39 =====
1/110 Testing disk:acl-xattr-test Passed
2/110 Testing disk:action-on-purge-test Passed
3/110 Testing disk:accurate-test Passed
4/110 Testing disk:allowcompress-test Passed
5/110 Testing disk:auto-label-test Passed
6/110 Testing disk:backup-bacula-test Passed
7/110 Testing disk:backup-to-null Passed
8/110 Testing disk:base-job-test Passed
9/110 Testing disk:bextract-test Passed
10/110 Testing disk:big-fileset-test Passed
11/110 Testing disk:big-vol-test Passed
12/110 Testing disk:bscan-test Passed
13/110 Testing disk:bsr-opt-test Passed
14/110 Testing disk:compressed-test Passed
15/110 Testing disk:compress-encrypt-test Passed
16/110 Testing disk:concurrent-jobs-test Passed
17/110 Testing disk:copy-jobspan-test
cmake-2.8.0
Stopping the Bacula File daemon
Stopping the Bacula Storage daemon
Stopping the Bacula Director daemon
===== pretest OK 16:05:36 =====
Start 1: disk:acl-xattr-test
1/81 Test #1: disk:acl-xattr-test .............. Passed 21.00 sec
Start 2: disk:action-on-purge-test
2/81 Test #2: disk:action-on-purge-test ........ Passed 34.00 sec
Start 3: disk:accurate-test
3/81 Test #3: disk:accurate-test ............... Passed 155.00 sec
Start 4: disk:allowcompress-test
4/81 Test #4: disk:allowcompress-test .......... Passed 37.00 sec
Start 5: disk:auto-label-test
5/81 Test #5: disk:auto-label-test ............. Passed 25.00 sec
Start 6: disk:backup-bacula-test
6/81 Test #6: disk:backup-bacula-test .......... Passed 35.00 sec
Start 7: disk:backup-to-null
7/81 Test #7: disk:backup-to-null .............. Passed 26.00 sec
Start 8: disk:base-job-test
8/81 Test #8: disk:base-job-test ............... Passed 35.00 sec
Start 9: disk:bextract-test
9/81 Test #9: disk:bextract-test ............... Passed 13.00 sec
Start 10: disk:big-fileset-test
10/81 Test #10: disk:big-fileset-test ............ Passed 19.00 sec
Start 11: disk:big-vol-test
11/81 Test #11: disk:big-vol-test ................ Passed 31.00 sec
Start 12: disk:bscan-test
12/81 Test #12: disk:bscan-test .................. Passed 25.00 sec
Start 13: disk:bsr-opt-test
13/81 Test #13: disk:bsr-opt-test ................ Passed 21.00 sec
Start 14: disk:compressed-test
14/81 Test #14: disk:compressed-test ............. Passed 23.00 sec
Start 15: disk:compress-encrypt-test
15/81 Test #15: disk:compress-encrypt-test ....... Passed 25.00 sec
Start 16: disk:concurrent-jobs-test
16/81 Test #16: disk:concurrent-jobs-test ........ Passed 32.00 sec
Start 17: disk:copy-jobspan-test
17/81 Test #17: disk:copy-jobspan-test ........... Passed 29.00 sec
Start 18: disk:copy-job-test
18/81 Test #18: disk:copy-job-test ............... Passed 28.00 sec
Start 19: disk:copy-uncopied-test
I must say: I like the change.
Posted in General | No Comments »
December 29th, 2009
I lived in New Zealand from 1985-2001. I haven’t been back since. It’s time.
The plan: March-April 2010, two weeks. Not nearly enough time for a proper holiday or visit, but we’ll hit the hottest trails. This should be a blast.
We plan to do 8-10 rides in that time. We’ll see. I hope to document as much as I can. I just ordered the mountain biking book to help with some of the planning.
We’ll rent a car, bike, and go from there.
Posted in General | No Comments »