Backup jobs fail, but only full jobs.

sistemas

Hello Bacularis team!

This is a weird issue that we are having.

Running Bacularis on Ubuntu 22.04, bacula 13.04, with bacula-sd-cloud-driver-13.0.4.so bacula-sd-cloud-s3-driver-13.0.4.so.

Incremental backup jobs for clients are finished without errors, but full jobs not so much. Some finished, but almost failed.

The error is : Error: lib/bsock.c:276 Socket has errors=1 on call to Storage daemon and S3_put_object ERR=AbortedByCallback CURL Effective URL:

Fatal error: filed/backup.c:1008 Network send error to SD. ERR=Input/output error
pc-fd JobId 1205: Error: lib/bsock.c:397 Wrote 43264 bytes to Storage daemon:10.10.10.1:9103, but only 16384 accepted.

What we already tried:

MaximumPartSize tried to resize down from 5GB to 2 or 1 GB. Helped a bit, but some jobs still failed.

Added bacula soft nofile 16384 && bacula hard nofile 32768 in the /etc/security/limits.conf

Added [Service] LimitNOFILE=32768 in the /lib/systemd/system/bacula-sd.service

Checked the network stack, firewall rules, all ports and permissions.

What are we missing here?

gani

Hello @sistemas,

Welcome to the Bacularis user group.

The Bacula S3 could driver is deprecated and no longer developed. This driver is replaced by Amazon driver available from Bacula 15.0. If in the future you will upgrade Bacula to 15.0, I would advise to switch from S3 to Amazon driver. The new driver uses the official Amazon awscli to work with the S3 clouds. If you use Amazon cloud, then the only change to do is just switch in the Cloud resource the Driver directive to Amazon.

For this error that you experienced, I don't know what can be the exact reason. Generally every time, when I see Input/output error first I check dmesg log.

Best regards,
Marcin Haba (gani)

sistemas

Hello @gani, thank you for the quick response.

After reading what you wrote, i realize that indded we are using an older version of bacula 13.03 and a deprecated driver.
Any advice regarding updating the bacula to a newer version? The server is virtualized and i can snapshot it before starting the upgrade process.

Thanks in advance!

gani

Hello @sistemas,

For the upgrade process, it depends on how Bacula in your environment is installed. For all installation methods it is always advised to read the release notes for the new version:

https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/blob/Release-15.0.3/bacula/ReleaseNotes?ref_type=tags

Bacula 15.0 introduces a new catalog format, so if you installed Bacula manually, you will need to run update_DBTYPE_tables script to update the tables format. This script is provided by Bacula.

In Bacula version 15.0 for PostgreSQL database is a new dependency. This is a pg_trgm extension that needs to be installed.

The Bacula deb packages (Debian, Ubuntu...) use dbconfig-common framework that helps to manage database during installs and upgrades. This way you don't need to worry about running Bacula upgrade scripts or about installing dependent PostgreSQL extensions.

Before starting it is good to do the Bacula database backup.

Good luck!

Best regards,
Marcin Haba (gani)