Deleting old Bacula Volumes

From time to time, my Bacula – space and time website shows some old jobs that need to be copied to tape. Stuff from 100 days or more. There’s no reason to copy stuff that old to tape. It’s already been copied to tape. So why does it appear?

Last night I swapped some tapes in the tape library. This morning they would have been used and recycled. I’ll bet that those tapes had copies of the backups which are now old and want to be copied to tape. That is, the original copy on tape has just been recycled. But the original is still on disk.

So why are they still on disk? Because I’m not recycling my disk Volumes quickly enough. At present, I have 95 volumes of 5GB incremental Volumes. Let’s clean that up.

Prune the old Volumes

My Incremental pool has a retention period of 21 days. So let’s look at all Volumes which were last written to before January 2013.

bacula=# select 'prune yes volume=' || volumename from media where lastwritten < '2013-02-01' and poolid = 24 order by lastwritten;
  volumename   |     lastwritten     | volretention
---------------+---------------------+--------------
 IncrAuto-2400 | 2012-05-23 09:21:53 |      1814400
 IncrAuto-1703 | 2012-05-24 06:03:31 |      1814400
 IncrAuto-1508 | 2012-05-24 09:07:37 |      1814400
 IncrAuto-1360 | 2012-05-24 10:38:15 |      1814400
 IncrAuto-1362 | 2012-05-25 08:53:58 |      1814400
 IncrAuto-1363 | 2012-05-25 10:24:05 |      1814400
 IncrAuto-1461 | 2012-05-26 08:34:10 |      1814400
...
...
 prune yes volume=IncrAuto-2341
 prune yes volume=IncrAuto-1936
(65 rows)

bacula=#

To be fair, I should have also checked volstatus to make sure the volumes where FULL. But prune won’t do anything wrong. It’ll just prune what’s ready to be pruned, based on retention.

I copy/pasted that output into bconsole and let it run. Here is some of that output:

* prune yes volume=IncrAuto-1515
The current Volume retention period is: 21 days
There are no more Jobs associated with Volume "IncrAuto-1515". Marking it purged.
* prune yes volume=IncrAuto-1516
The current Volume retention period is: 21 days
There are no more Jobs associated with Volume "IncrAuto-1516". Marking it purged.
* prune yes volume=IncrAuto-1517
The current Volume retention period is: 21 days
There are no more Jobs associated with Volume "IncrAuto-1517". Marking it purged.

Deleting the Volumes from the Catalog

Now that the Volumes are pruned, let’s remove the Volumes from the Catalog. We start by modifying the query from above.

bacula=# select 'delete yes volume=' || volumename from media where lastwritten < '2013-02-01' and poolid = 24 order by lastwritten;
            ?column?
---------------------------------
 delete yes volume=IncrAuto-2400
 delete yes volume=IncrAuto-1703
 delete yes volume=IncrAuto-1508
 delete yes volume=IncrAuto-1360
 delete yes volume=IncrAuto-1362
 delete yes volume=IncrAuto-1363
 delete yes volume=IncrAuto-1461
...

But hold on there, this isn’t perfect. What if some of the Volumes still had Jobs associate with them? Well, they’ll be deleted too using this query. So to be better, we should have checked voljobs = 0 on that query.

Copy pasting that output into bconsole, we get (in part):

* delete yes volume=IncrAuto-1461

This command will delete volume IncrAuto-1461
and all Jobs saved on that volume from the Catalog
* delete yes volume=IncrAuto-2402

This command will delete volume IncrAuto-2402
and all Jobs saved on that volume from the Catalog
* delete yes volume=IncrAuto-1364

This command will delete volume IncrAuto-1364
and all Jobs saved on that volume from the Catalog
* delete yes volume=IncrAuto-1366

This command will delete volume IncrAuto-1366
and all Jobs saved on that volume from the Catalog
* delete yes volume=IncrAuto-1515

Removing the Volumes from disk

It it important to know the difference between the boncole delete command and the unix rm command.

• The bconsole command affects the Catalog (database). It does not touch the Volume on disk.
• The unix command will remove the Volume from the filesystem.

It is important to do these commands in the right order.

In the previous section, we deleted the Volumes from the Catalog.

In this section, we remove Volumes (i.e. disk files) from the filesystem.

We start by modifying the query from above.

bacula=# select ' rm ' || volumename from media where lastwritten < '2013-02-01' and poolid = 24 order by lastwritten;
     ?column?
-------------------
  rm IncrAuto-1703
  rm IncrAuto-1508
  rm IncrAuto-1360
  rm IncrAuto-1362
...

Copy the above commands. Then, while in the proper directory which contains the Volumes in question, paste the commands:

[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1360
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1362
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1363
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1461
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-2402
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1364
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1366
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1515
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1516
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1517
[root@kraken /storage/compressed/bacula/volumes]#   rm IncrAuto-1367
...

Update the Pool definition

Next, I updated the Pool definition, and set the Maximum Volumes parameter to be lower than before. In this case, it had been 95. I just deleted 65 Volumes. Checking the database, I had 30 Volumes in this pool, so that matches up:

bacula=# select count(*) from media where poolid= 24;                                                                           count
-------
    30
(1 row)

bacula=#

I changed Maximum Volumes to 40.

In bconsole, I issued the reload command so bacula-dir would reload the settings from baucla-dir.con.

Then I issued the update command to refresh the Pool definition from the settings.

*reload
*update
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
Update choice:
     1: Volume parameters
     2: Pool from resource
     3: Slots from autochanger
     4: Long term statistics
Choose catalog item to update (1-4): 2
The defined Pool resources are:
     1: Default
     2: FullFile
     3: DiffFile
     4: IncrFile
     5: IncrFileNoNextPool
     6: FullFileNoNextPool
     7: Fulls
     8: Differentials
     9: Incrementals
    10: FullBackupsFile
    11: FilePool
    12: Scratch
    13: MegaFilePool
Select Pool resource (1-13): 4
+--------+----------+---------+---------+---------+------------+-----------------+--------------+----------------+------------+-------------+---------------+-----------+---------+----------+-------------+---------+---------------+---------------+-----------+------------+--------------------+-------------------+---------------+---------------+
| poolid | name     | numvols | maxvols | useonce | usecatalog | acceptanyvolume | volretention | voluseduration | maxvoljobs | maxvolfiles | maxvolbytes   | autoprune | recycle | pooltype | labelformat | enabled | scratchpoolid | recyclepoolid | labeltype | nextpoolid | migrationhighbytes | migrationlowbytes | migrationtime | actiononpurge |
+--------+----------+---------+---------+---------+------------+-----------------+--------------+----------------+------------+-------------+---------------+-----------+---------+----------+-------------+---------+---------------+---------------+-----------+------------+--------------------+-------------------+---------------+---------------+
|     24 | IncrFile |      30 |      40 |       0 |          1 |               0 |    1,814,400 |              0 |          0 |           0 | 5,368,709,120 |         1 |       1 | Backup   | IncrAuto-   |       1 |             0 |             0 |         0 |            |                    |                   |               |             0 |
+--------+----------+---------+---------+---------+------------+-----------------+--------------+----------------+------------+-------------+---------------+-----------+---------+----------+-------------+---------+---------------+---------------+-----------+------------+--------------------+-------------------+---------------+---------------+
Pool DB record updated from resource.
*

All done

Now I just sit and wait, and see how much that affects my backups. Over time, I should have to do this less and less often. Perhaps I’ll have to bump MMaximum Volumes up depending on how large my incremental backups become.

I should also adjust my copy query to ignore backups that are more than 21 days old. I’ll keep that in the web site reporting, to give me an indication of outdated disk copies. But in the actual copy-to-tape SQL, I don’t want to take them.