Volumes missing from bacula and never getting recycled

Nicholas

I'm having trouble with volumes being "forgotten" by bacula, while the volume files are still present on disk, and never getting properly recycled.

We originally noticed the problem when a backup drive ran out of space, and we realized that there are a number of volumes on the disk that no longer appear in the Volumes list in Bacularis, and as far as I can tell are no longer present in the catalog at all (I've even tried directly querying the catalog with sql and these volumes simply are not present in the Media table).

It seems that new volumes are being created whenever it needs a volume, and the old ones are being "forgotten" by bacula instead of getting recycled.

I've checked a number of volumes in my pools and they all have a recycle count of 0 - except for the pool that holds the director's backups, where some of the older volumes have been recycled several times, but the newer ones have not. I'm not sure what changed that made these stop being recycled.

We do have MaximumVolumeJobs set to 1, but if I understand correctly this should simply make the volumes get marked as Used after having one job written to them, and they should get recycled and another job written to them after the JobRetention period is over.

Also we have JobRetention set to 180 days, while the VolumeRetention is only 70 days. Could this have something to do with it? The oldest volumes that have not been "forgotten" by bacula are about 3 months old, which is odd because this doesn't really line up with any of the retention periods. It seems the jobs in the job history also cut off at the same time as the volumes, so apparently job records are also being deleted at the same time as the volumes get "forgotten".

I'm not sure how to troubleshoot this. Any advice would be appreciated.

For context here is an edited version of some of my config:

Client {
  Name = "MyClient_fd"
  Address = "..."
  FdPort = 9102
  Password = "..."
  Catalog = "MyCatalog"
  FileRetention = 5184000
  JobRetention = 15552000
  AutoPrune = yes
}
Storage {
  Name = "MyClient_onsite1"
  SdPort = 9103
  Address = "..."
  Password = "..."
  Device = "Backup1"
  MediaType = "File"
  Autochanger = "MyClient_onsite1"
  MaximumConcurrentJobs = 10
}
Pool {
  Name = "MyClient_onsite1_pool"
  PoolType = "Backup"
  LabelFormat = "Vol-"
  MaximumVolumeJobs = 1
  VolumeRetention = 6048000
  Storage = "MyClient_onsite1"
  AutoPrune = yes
  Recycle = yes
}

Let me know if there's other parts of the config that would be helpful for me to share as well.

gani

Hello @Nicholas,

Welcome to the Bacularis user group.

In the volume recycling I think the most important is to understand the retention matter.

Retention time

The retention for files (FileRetention) defines how long the file records will be stored in the database. The file records are used for various purposes from which the most common is ability to do restore with selecting single files. If the file records for a job are pruned from the database after expiring the retention time, then restore is still possible, of course but it will be full job restore all files without selecting single files or directores.

The retention for jobs (JobRetention) defines how long the job records will be stored in the database. Please note that when the job retention is expired and the job records are pruned in the database, then ALL file records that belong to this job are pruned as well.

The retention for volumes (VolumeRetention) defines how long the volume records will be stored in the database. Please note that when the volume retention is expired and the volume records are pruned in the database, then ALL job records that are stored on this volume and ALL file records that belong to these jobs are pruned as well.

The above means that the reasonable is to set retentions with the following relation:

File Retention <= Job Retention <= Volume Retention

In your case, you set Job Retention to 180 days and the Volume Retention to 70 days that means the Job Retention will never be used because if volume will be recycled, then the job records (and file records) are automatically pruned as well.

Volume recycling

The volume recycling takes place for volumes marked with volume status Full or Used after expiring the volume retention time for volumes. The start time for counting volume retention is the time of last writting volume that changed volume status from Append to Full/Used.

[Append] => [Full/Used] => [Purged] => [Recycled] => [Append] ... and so on.

Please note that volume recycling does not remove volumes nor volume records from the database, it just reuse them from the beginning. You wrote that you have volumes on disk but not in the database. It can mean that they have been removed from the catalog for example by using delete volume action in the Bacularis or delete volume=XYZ Bconsole command. Can you confirm it?

To use these volumes back in Bacula, you may need to use bscan tool to scan volumes for recreating the catalog records. If you no longer need these volumes, then better will be to just remove them since they have no longer "connection" with Bacula.

Volume limit

You wrote that volumes are not reused. I would propose to do two things:

add volume limit to the pool (Maximum Volumes directive) directive to keep fixed number of volumes. In case automatic labeling Bacula prefers to create new volumes than reuse existing and setting the volume limit changes this behaviour.
make sure if after setting the retention time in Pool resource, you ran update pool and update volumes from pool commands to apply the changes in Pool for existing volumes. It can be done in Bacularis on the Pool details page or directly in Bconsole.

Please let us know if it helped.

Good luck!

Best regards,
Marcin Haba (gani)

Nicholas

Thanks for the detailed response!

gani It can mean that they have been removed from the catalog for example by using delete volume action in the Bacularis or delete volume=XYZ Bconsole command. Can you confirm it?

As far as I know delete was not used either via Bacularis or Bconsole, but I can't necessarily say for sure. One thing that could indicate that perhaps these volumes were all removed from the database at some point in the past is that since I've been monitoring the problem it doesn't seem that any more volumes have disappeared from the database. Is there any other type of operation that could have made it remove a number of volumes from the database?

I don't think these old volumes contain data the needs recovered, so using bscan shouldn't be neccessary.

gani In your case, you set Job Retention to 180 days and the Volume Retention to 70 days that means the Job Retention will never be used because if volume will be recycled, then the job records (and file records) are automatically pruned as well.

Yes, I must have configured this incorrectly. I'll see about re-configuring it. Technically it should be working as it is though? Just that it will delete the Job records at 70 days along with the volume instead of waiting for the 180 day mark?

gani add volume limit to the pool (Maximum Volumes directive) directive to keep fixed number of volumes.

What happens if I set Maximum Volumes to too low of a number? We are using MaximumVolumeJobs = 1, so we do need quite a number of volumes.

gani make sure if after setting the retention time in Pool resource, you ran update pool and update volumes from pool commands to apply the changes in Pool for existing volumes.

Aha! it looks like this might be part of the issue. The older volumes that should be getting recycled currently have a much higher retention period (that isn't over yet) than what recent volumes have and what is currently set in the config. The retention period must have been changed in the config without updating the existing pools.

I've updated the volumes in a couple pools and now the When expire date is in the past for the older volumes in the pool. Will see if these volumes start getting recycled now.

gani

Hello @Nicholas,

Nicholas Is there any other type of operation that could have made it remove a number of volumes from the database?

No, there isn't any Bacula action other than manual action with deleting volumes. Bacula does not remove volumes automatically. The general Bacula approach is to keep data on volumes as long as possible, even if it is already expired. Because of that the delete command deletes volumes in the database only while the backup data still stays on the volumes until user removes them manually.

Nicholas Yes, I must have configured this incorrectly. I'll see about re-configuring it. Technically it should be working as it is though? Just that it will delete the Job records at 70 days along with the volume instead of waiting for the 180 day mark?

Yes, exactly like that. In this setting file and job records will be removed on the volume recycling after 70 days.

Nicholas What happens if I set Maximum Volumes to too low of a number? We are using MaximumVolumeJobs = 1, so we do need quite a number of volumes.

If the Maximum Volumes limit in Pool is reached then Bacula tries to find a volume in this Pool that it could use. Bacula will check if there exist volumes that could be recycled and if yes, then Bacula will recycle them and reuse automaticaly. If there is no volume that could be reused, then Bacula will report that no volume available and that it needs new volume to continue. If this happens, it is a good sign for you, because it can mean that you should do one of the two things:

Adjust the recycling directives (limits, retention time...etc.) if it does not work as you expected
Increase the Maximum Volumes limit if it recycling work as you expected but it requires more volumes

Please do not worry a bit bigger number of volumes. They are just files on disk, so it is not a problem. In my environment currently I have 1920 disk volumes in total that come from few virtual disk autochangers.

Nicholas I've updated the volumes in a couple pools and now the When expire date is in the past for the older volumes in the pool. Will see if these volumes start getting recycled now.

Great 🙂

Best regards,
Marcin Haba (gani)