Plugin Idea: Docker DB Backups Without Local Client

michaeldolzani

Hi everyone,
I wanted to suggest an idea: would it be possible to create a new plugin for backing up Docker databases that doesn’t require the DB client to be installed on the host machines?
Currently, to back up Docker databases, I need to have the MySQL or Postgres client installed locally, which can be inconvenient.
A plugin that can perform backups directly from the containers without relying on a local client would be really useful.

Thanks!

gani

Hello @michaeldolzani,

Thank you for describing your idea.

With MySQL and PostgreSQL, do you use the dump backup method? If yes, could you tell me what dump method are you currently using? Is it the dump all databases with writing each database in a separate dump? I mean these:

MySQL plugin - dump all databases separately

https://bacularis.app/doc/plugins/database/mysql-backup-plugin.html#dump-all-databases-separately

PostgreSQL plugin - dump all databases separately

https://bacularis.app/doc/plugins/database/postgresql-backup-plugin.html#dump-all-databases-separately

Or you are using other methods?

Thanks in advance for the confirmation.

Best regards,
Marcin Haba (gani)

michaeldolzani

Hi @gani ,

Yes, I am currently using the MySQL and PostgreSQL plugins with the “dump all databases separately” method, as described in the documentation links you shared. Each database is written to its own dump file.

I also wanted to ask: if the backup job is set as incremental, would that cause any issues with this approach, or does it need to be full backups only?

Thanks for your help!

Best regards,

gani

Hi @michaeldolzani,

Yes, this dump all databases separately method uses the local DB client to determine all database list. Technically this action to list databases could be done directly on the file daemon host, but for Bacularis plugins based on Bacula Bpipe plugin this is not possible to do it on the FD side. This is not because of doing backup but because of doing restore later.

If your database list in Docker container does not change too often or if it isn't dynamic at all, then if you want you can try to use the dump only selected databases method. In this case, the database list is pre-defined in --databases="dbX,dbY,dbZ" parameter and it should not require local DB client. The same is with dump all databases in one dump method.

MySQL - dump only selected databases

https://bacularis.app/doc/plugins/database/mysql-backup-plugin.html#dump-only-selected-databases

PostgreSQL - dump only selected databases

https://bacularis.app/doc/plugins/database/postgresql-backup-plugin.html#dump-only-selected-databases

If you try it, please shere your feedback with us how it went.

MySQL backup and incremental

I noted to check if there is any chance to use dump all databases separately method without local DB client. If we find a way to do it in this method, I will let know here.

For your question about incremental backup with dump methods, this is possible to do incremental backup with MySQL and MariaDB plugins, if you have configured the binary log in following way:

https://bacularis.app/doc/plugins/database/mysql-backup-plugin.html#prepare-environment

You can see how it works on this video guide below. The video is for dump all databases separately method but is will work the same for the dump only selected databases method.

https://youtu.be/09BaFdNfwPI?t=368

The differential backups are possible to do as well.

The incremental and differential backups will be unsuccessful if you do not have the binary log configured as described above.

PostgreSQL backup and incremental

For PostgreSQL plugin, with the dump method it supports only full backups.

Best regards,
Marcin Haba (gani)

gani

Hi @michaeldolzani

I did retest on my side and I have an idea what can be wrong in your settings.

Could you check in your FileSet if the Plugin directive line starts with \ character?

For example this:

Plugin = \|/usr/share/bacularis/protected/Common/Bin/plugin ...some parameters

If it does not start with \ then please add it to the Plugin definition and then you will not need to have local db client binaries. This \ means that plugin command will be executed on the client host instead on the director host and by default Bacularis adds it.

In the past this function could work differently and this is why I pointed the dump all databases separately method in my previous message.

Thanks in advance for your try and feedback.

Best regards,
Marcin Haba (gani)

michaeldolzani

Hi Marcin,

thank you for your suggestion.

I checked the configuration as you described and, even after adding the \ at the beginning of the Plugin directive in the FileSet, on the client side I still need to have the MySQL or PostgreSQL binaries installed in order for the backup to work properly.

What I am trying to achieve, however, is a different approach: I would like the client machines not to require MySQL or PostgreSQL to be installed at all, because the databases are running exclusively inside Docker containers.

My goal is to have the database backups performed directly through Docker (for example by using docker exec inside the DB container to run the dump), without relying on MySQL/PostgreSQL client binaries installed on the host system.

In other words, I want the backup of the Docker containers that host the databases to be handled using Docker itself as the “transport layer”, avoiding any dependency on locally installed DB client tools on the client machine.

If you have suggestions on how to adapt the plugin or the FileSet to work in this way, they would be very helpful.

Thank you again for your support.

Best regards

gani

Hi @michaeldolzani,

Thanks for explaining it in details. It looks that originaly I understood you incorrectly because I differently understood your meaning of the host machine.

To be sure that we see the same, in this environment the host machine is the system working in the container, right? This is the system where you have the db server? And you want to back up it remotely without db client dependencies installed inside the container?

Thanks in advance for confirming it.

Best regards,
Marcin Haba (gani)

michaeldolzani

Hi Marcin,

To clarify further, my goal is to back up databases that are running inside Docker containers without installing any database binaries on the host machine where Docker is running.

Ideally, I would like a plugin that can connect to the host machine and perform a dump of all MySQL and PostgreSQL databases present in multiple containers, without requiring MySQL or PostgreSQL binaries installed on the Docker host itself.

This way, the backup would rely entirely on the containers themselves rather than on any client tools installed on the host.

Best regards,

gani

Thanks. OK, we see it.

Best regards,
Marcin Haba (gani)