S3 migrated files and restore jobs failed

rdelmas

Hello from France,

Hope all is well for you.

I'm wondering about S3 migration jobs and restore jobs...

When a job is migrated, a local copy is made before the migration (/path/cache/client/S3_file/part.1 etc.) - local copy which is not necessarily kept on server after the migration (S3/path/S3_file/part.1 etc.).

However, file restoration via the GUI and the restore job does not work : the job seems to look for the file not directly on the S3 but rather in the local part used for cache :

director-sd JobId 158243: Warning: acquire.c:234 Read open Cloud device "S3_DEVICE_client" (/opt/data/cache/client) Volume "S3_DATA_client_date" failed: ERR=cloud_dev.c:1471 Could not open(/opt/data/cache/client/S3_DATA_client_date/part.1,OPEN_READ_ONLY,0640): ERR=No such file or directory

Here are the informations in bacula-sd :

Cloud {
  Name = "S3_cloud"
  Driver = "S3"
  HostName = "s3_hostname"
  BucketName = "bucket_name"
  AccessKey = "XXX"
  SecretKey = "XXX"
  Protocol = "HTTPS"
  UriStyle = "Path"
  TruncateCache = "No"
  Upload = "EachPart"
}

Device {
  Name = "S3_DEVICE_client"
  Description = "Device for client logical volume before migration"
  MediaType = "S3_MEDIA_client"
  DeviceType = "Cloud"
  ArchiveDevice = "/opt/data/cache/client"
  RemovableMedia = no
  RandomAccess = yes
  AutomaticMount = yes
  LabelMedia = yes
  AlwaysOpen = no
  MaximumVolumeSize = 5000000000
  MaximumFileSize = 1000000000
  MaximumConcurrentJobs = 5
  MaximumPartSize = 5000000000
  Cloud = "S3_cloud"
}

The migration job is using the S3 device/media and a MigrationTime to the S3_pool associated to the client (NextPool S3)... Here is my bacula-dir.conf file :

Storage {
  Name = "S3_STORAGE_client"
  Description = "S3 logical bucket storage for client"
  Address = "bacula_server"
  Password = "XXX"
  Device = "S3_DEVICE_client"
  MediaType = "S3_MEDIA_client"
  MaximumConcurrentJobs = 5
  TlsEnable = yes
  TlsPskEnable = no
  TlsRequire = yes
  TlsCaCertificateFile = "/etc/ssl/bacula/bacula_ca.crt"
  TlsCertificate = "/etc/ssl/bacula/bacula_server.crt"
  TlsKey = "/etc/ssl/bacula/bacula_server.key"
}
Pool {
  Name = "Full_client"
  Description = "Pool for client full backups"
  PoolType = "Backup"
  LabelFormat = "Full_${Job}_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}_${Hour:p/2/0/r}:${Minute:p/2/0/r}:${Second:p/2/0/r}"
  LabelType = "Bacula"
  UseVolumeOnce = yes
  PurgeOldestVolume = no
  ActionOnPurge = "Truncate"
  MaximumVolumeBytes = 5000000000
  VolumeRetention = 950400
  MigrationTime = 604800
  NextPool = "S3_client"
  Storage = "STORAGE_client"
  AutoPrune = yes
  Recycle = no
  Catalog = "MyCatalog"
  FileRetention = 950400
  JobRetention = 950400
}
Pool {
  Name = "S3_client"
  Description = "Pool used to migrate full oldest client volumes on S3"
  PoolType = "Backup"
  LabelFormat = "S3_${Job}_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}_${Hour:p/2/0/r}:${Minute:p/2/0/r}:${Second:p/2/0/r}"
  LabelType = "Bacula"
  UseVolumeOnce = yes
  PurgeOldestVolume = no
  ActionOnPurge = "Truncate"
  MaximumVolumeBytes = 5000000000
  VolumeRetention = 15724800
  Storage = "S3_STORAGE_client"
  AutoPrune = yes
  Recycle = no
  Catalog = "MyCatalog"
  FileRetention = 15724800
  JobRetention = 15724800
}
Job {
  Name = "MIGRATION_client"
  Description = "Job to migrate oldest full jobs of client on S3"
  Type = "Migrate"
  Messages = "Standard"
  Storage = "STORAGE_client"
  Pool = "Full_client"
  NextPool = "S3_client"
  Client = "client"
  Fileset = "MIGRATION_fileset"
  Schedule = "SCHEDULE_migration"
  PruneJobs = yes
  PruneFiles = yes
  PruneVolumes = yes
  PurgeMigrationJob = yes
  Enabled = yes
  MaximumConcurrentJobs = 5
  Priority = 10
  SelectionType = "PoolTime"
  AllowDuplicateJobs = no
}
Job {
  Name = "RESTORE_S3_client"
  Type = "Restore"
  Messages = "Standard"
  Storage = "S3_STORAGE_client"
  Pool = "S3_client"
  Client = "client"
  Fileset = "RESTORE_fileset"
  Where = "/tmp/restore"
  MaxWaitTime = 1200
  Enabled = yes
  MaximumConcurrentJobs = 5
  Priority = 10
}

So, after migration files on cache server and old jobs datas/metadatas are purged and delete from disk/Catalog, to keep only the migrated job information on S3/database.

I tried with several restore jobs with the same error... Do I have to manually recreate the remote file on S3 locally to use the restoration job ? Or, do i just need to keep a tree in the cache folder if I want to be able to restore the files?

Best regards from here,
Romain

gani

Hello Romain,

Greetings from Poland.

I have not using S3 driver for a long time. This driver is deprecated and recommeneded driver by Bacula for S3 storage is Amazon driver that is modern and it uses AWS CLI. That old S3 driver to cloud support uses old library that is no longer maintained.

For your problem with migration and S3 driver, I think the migration to the cloud storage is fully valid. The problem here, as you correctly noticed, is that the Bacula device is not able to find the local cache volume.

Normally Bacula does not do restore directly from the cloud, but it downloads required volume parts and does this restore from these parts. In described here case, the storage daemon is not able to open volume because part.1 is missing. This part.1 is specially important because inside this part is stored the volume label data, that is required to mount volume.

I have two questions for now:

Could you tell us how you removes this local cache volume parts?
Are you seeing on the remote cloud storage this part.1 volume part file?

At the end I encourage to use this new Amazon driver for S3 storage. It works really well.

Best regards,
Marcin Haba (gani)

rdelmas

Hello Marcin !

I hope you're well !

Thanks for your message.

I'm using a bash script, launch after migrations jobs, which compares files on S3 and files on cache directory ; then it deletes local files when all parts are on S3.
As the script only deletes local files if both files are present on the S3, the part.1 file is present in S3 storage.

I tested recreating the tree structure of volumes/files from S3 to cache directory in local server : I've recovered all the volumes and files in the S3, and I've mkdir -p volumes and touch part files. Obviously, files were empty. In this case, restore job from S3 to servers worked well... So, i conclude that it's enough to keep the architecture (and therefore not delete the files but empty them via a truncate -s).

I will tried the Amazon driver as soon as possible and test a new way of testing S3 restoration !

With all my thanks,
Kind regards,

Romain

gani

Hello Romain,

That is great you were able to do valid restore with data from S3 storage. This truncating volume parts way looks solid.

That algorithm of checking parts sent to cloud and removing them locally is also integrated in the Cloud resource in TruncateCache directive, for example:

TruncateCache = AtEndOfJob

In this case the cloud driver removes it automatically at the end of the job.

For truncating local cache parts manually by bconsole cloud command (menu option Truncate a Volume Cache) worth to mention is that this truncate always keeps part.1 (that contains label) and removes all other parts for volume. It can be to be able mount the local cache volume by SD even if local cache is cleared. I am writing about it because in case your original problem maybe the workaround is to keep locally part.1 only and remove others that are uploaded. They are my thoughts :-)

Good luck with these cloud drivers.

Best regards,
Marcin Haba (gani)

rdelmas

Hi Marcin !

I will try the truncatecache directive and the new driver this afternoon in test mode.

I'll tell you more but it seems a very good option.

Romain

rdelmas

Hi,

I tried truncatecahe directive and it's work as you said.

On S3 :

(venv_cli_aws) :~$ aws s3 ls BUCKET/S3_DATA_CLIENT_2025-04-24_15:23:25/
2025-04-24 15:23:25        334 part.1
2025-04-24 15:24:03  281784082 part.2

On server :

:~$ sudo ls -lah /opt/data/cache/CLIENT/S3_DATA_CLIENT_2025-04-24_15:23:25
total 8.0K
-rw-r-----  1 bacula tape  334 Apr 24 15:23 part.1

gani

Hi @rdelmas,

Excellent 🙂

Thanks for your feedback here.

Best regards,
Marcin Haba (gani)