Multiple catalogs (databases)

Plural

HI Marcin (@gani),

Let me explain my goal first. We have a cliente (let's call it "external") in our pós-production facilities that want to keep they backups as archive. I need to provide then a separate catalog (database) at the end of the production.
Until now I performed the following actions:

I create a new bacula database called "external" using the bacula scripts.
I create the external catalogo in bacularis.
I create the external API host in Security.
I give access in my user account to the external API host. Now when I login I get the host pulldown to choose the host.
I choose the host external an login.
In the Clients page, I create a new client and give them the external Catalog.

At this point I can not see the new client in the Clients page. The client appears in the Director configuration and it's available in the job configuration but is not available in the Restore page.

What I'm doing wrong?

Best regards,
Rui

gani

Hi Rui (@Plural)

OK, I see your need. Thanks. So, you need to archive data from the one Bacula client for long term use, right?

In this settings I think you are doing it correctly with one missing compoment. The most important point is to have two Bacularis API:

1) one already existing for main Bacula Catalog
2) one new for the external Bacula Catalog

This 1) you already have. This 2) is something to create. Using Bacularis installation with binary packages this can be hard to create one more Bacularis API on the same host but different port. This second Bacularis API could be installed in another way if you want to do that. After the newly created API host connection in web interface will connect directly to this new API host.

At the moment I tried this environment with two Catalogs and two APIs. This looks to be quite fine with the Restore wizard exception. Jobs for the external client are listed connectly in Restore wizard, but there is a problem with listing content of the jobs in the restore file browser. This is because if we have multiple catalogs in one Director, then some Bacula bconsole commands, which require Catalog access, ask question about which Catalog should be used.

In case the restore wizard there are couple of commands (ex. Bvfs commands) which asks question - which catalog should be used. The Catalog settings in bconsole is done by use command. The only problem with this command is that Bacularis does not support it.

*.bvfs_update jobid="1"
You must specify a "use <catalog-name>" command before continuing.
Could not find a Catalog resource

I tried to use Console ACLs to limit Catalogs for this second API host to only one Catalog. This way the bconsole should not ask questions about multiple catalogs because it will have access to only one. But as I experienced at the moment, this does not work even if we give access to only one Catalog. Probably creating bconsole wrapper for this external Bacularis API could work where this wrapper opens bconsole and executes use command to select the Catalog for all this bconsole session. But I didn't try this wrapper.

In Bacularis we could think about adding this Catalog selection support. This might be not bad feature. But for now I am thinking about the original problem that we are trying to solve and I am wondering if using one Catalog and setting enough big job/file retention time for this external client would be sufficient? Did you take into account the retention values settings for that?

Best regards,
Marcin Haba (gani)

Plural

HI Marcin (@gani),

Thank you for your views about this. Let's talk in abstract for future implementations. What you propose is:
New API (e.g port 9098)-> New Catalog in configured in the Catalog page of the API Settings) -> API Host (pointing to localhost:9098)

It's possible to do this instead?:
Same API (port 9097) -> Multiple Catalogs, i.e. databases, in the Catalog page of the API Settings -> API Host with a new field to refer to the catalog.

All catalog related function will refer to the correct catalog when we change the API host in the API Host pulldown.

I'm saying this because in my point of view, a Rest API is just a proxy for a database, if you change the database but the database have the same structure, the API queries to the database are the same, just returns different results.

What do you you thing, this not solve the need for the use <catalog> command, and at the same time segregate one catalog from another?

Best regards,
Rui

gani

Hi Rui (@Plural),

Thanks for your response and comments.

I proposed using two API hosts for current Bacularis that does not have the multiple catalogs support in one API. This is a workaround but, as I checked and described, Bacula still asks about selecting catalog in bconsole commands (even with Bacula ACLs with one Catalog assigned). I think that adding wrapper to bconsole could solve it that will type this use command before normal Bconsole commands.

For support for multiple catalogs in one Bacularis API, yes, this not require two API hosts and this is something that Bacularis does not support yet.

In my message I think that I described it not clearly that they are two solutions: for current Bacularis and for Bacularis in the future.

Plural I'm saying this because in my point of view, a Rest API is just a proxy for a database, if you change the database but the database have the same structure, the API queries to the database are the same, just returns different results.

Yes. This is correct. Bacula does this switching in the same way by use bconsole command. It just changes db.

Plural What do you you thing, this not solve the need for the use <catalog> command, and at the same time segregate one catalog from another?

This use command is needed for more Catalogs than one. For one Catalog Bacula does the db auto-selection. My idea for this current Bacularis (not for future implementation) was to assign in Console ACLs only one Catalog for this new API host. This way Bacula should do this auto-selection in the same way as it does if only one Catalog exists. But Bacula Director does not do that but asks about Catalog selection even if Console ACLs restricts to only have one Catalog.

Best regards,
Marcin Haba (gani)

Plural

Hi Marcin (@gani),

gani But Bacula Director does not do that but asks about Catalog selection even if Console ACLs restricts to only have one Catalog.

Ok, I forgot about that.

gani My idea for this current Bacularis (not for future implementation) was to assign in Console ACLs only one Catalog for this new API host.

For that I will need a dedicated user, right? This implementation also requires a new API service or we can use the default one? If it requires a new one, can you give me some guidance in that matter?

Best regards,
Rui

gani

Hi Rui (@Plural),

Plural For that I will need a dedicated user, right? This implementation also requires a new API service or we can use the default one? If it requires a new one, can you give me some guidance in that matter?

Yes, this requires a new API user in the new API instance. The web server user can stay the same, it will only have one more additional API host assigned.

For the API service, yes, it will require a new API service with a new port. After configuring this new instance, in the web interface the new API host connection will connect to it.

For setting up this new instance, I know that the best might be use just binary packages, but it can be not easy with packages, because they support one instance by default and they use system paths like /var/lib/bacularis /var/cache/bacularis etc. which for two Bacularis instances should be separated.

If you want, you can use installation with Composer for example in /var/www/bacularis:

https://bacularis.app/doc/brief/installation.html#install-using-composer

This way the second instance will be separated from the first one.

After setting it up, the last important thing (that I didn't try) is to use wrapper for Bconsole. This wrapper needs to get the original bconsole command from command line, add at the begining of command something like use MyCatalog, execute command and at the end the best strip the use command from output but maybe this is not necessary. We will see. Once the wrapper is ready, it needs to be used in this new API instance as the bconsole command definition.

Please let us know if you decided to configure it and if you were able to set it up. I am curious how it will work.

Thanks and good luck. In case problems, please let us know

Best regards,
Marcin Haba (gani)

Plural

Hi Marcin (@gani),

I already created a new API instance in the port 9098. I don't install the web interface for the Bacularis.

Some questions about the new API configuration:

Do I need to configure the Autochangers and Devices in the new API?
About,

gani Yes, this requires a new API user in the new API instance.

I can't see the relation between a user in the new API instance and the old one. Who can I assign the Console ACL to the user? Do I need to create a Dedicated Bconsole config file in the new API location pointing to the Console ACL directive of the bacula-dir.conf?

About the wrapper, if I understood well, I create a script and I replace the default "Bconsole binary file path" in the new API Settings -> Cosole tab?

Best regards,
Rui

gani

Hi Rui (@Plural)

For this wrapper, I think it can be something like this or something similar:

#!/bin/bash

catalog='MyCatalog'
read -t0.1 cmd
echo -e "use ${catalog}\n${cmd}" | bconsole $*

Best regards,
Marcin Haba (gani)

Plural

Hi Marcin,

gani More like:

#!/bin/bash

catalog='MyCatalog'
read -t0.1 cmd
echo -e "use catalog=${catalog}\n${cmd}" | bconsole $*

Best regards,
Rui

Plural

Hi Marcin (@gani),

In you example configuration you use:

composer create-project bacularis/bacularis-app

Can I use something like:

composer create-project external/bacularis-app

Or even better:

composer create-project bacularis/external1/bacularis-app
composer create-project bacularis/external2/bacularis-app
...

Best regards,
Rui

gani

Hi Rui (@Plural),

This bacularis/bacularis-app is the name of the project in Composer.

To checkout this project to bacularis/external1/ you create for example create this path and then run the command from the documentation.

mkdir -p bacularis/external1
cd bacularis/external1
composer create-project bacularis/bacularis-app

Best regards,
Marcin Haba (gani)

gani

Hi Rui (@Plural)

Plural I already created a new API instance in the port 9098. I don't install the web interface for the Bacularis.

Great.

Plural Do I need to configure the Autochangers and Devices in the new API?

No, this is not needed. You can manage autochanger from the main API host.

Plural I can't see the relation between a user in the new API instance and the old one.

This new user was created when you installed this new API instance. It was created in the install wizard.

If you have this new API instance ready, then next steps can be following:

Create new API host connection in the web interface (Security => API hosts => Add new API host)
Assign this new API host to your current web interface user (Security => Users => Edit user)
Now you should be able to switch between the API instances using the dropdown list with API hosts at the top left of the web interface.

The Console ACLs are not necessary to configure for this user. I needed to configure it because I wanted to force Bacula to use one Catalog. But you have this wrapper that will do it automatically, so this Console ACLs are not needed.

For your questions:

Plural Who can I assign the Console ACL to the user?

The Console ACLs are assigned by the Bacularis web server admin user, but in this case with wrapper I would propose to not use Console ACLs if they are not really needed for other purposes. In my checks I wanted to use Console ACLs because I didn't have wrapper yet.

Plural Do I need to create a Dedicated Bconsole config file in the new API location pointing to the Console ACL directive of the bacula-dir.conf?

You don't need to create dedicated Bconsole config. This main bconsole.conf is OK and gives full access for user.

Plural About the wrapper, if I understood well, I create a script and I replace the default "Bconsole binary file path" in the new API Settings -> Cosole tab?

Yes, Exactly.

Best regards,
Marcin Haba (gani)

gani

Plural More like:

Great. Thanks for this fix.

Best regards,
Marcin Haba (gani)

Plural

Hi Marcin (@gani),

We already have our multiple catalog running. Thank you for the support in this. I'm posting a small step by step configuration.

Creating a multiple catalog implementation for Bacula and Bacularis

As the main goal for this implementation was to have a external partner using our Bacula/Bacularis server with a different database, we will use external to refer to it.

Create a new database in the database engine (this is for PostGreSQL):

cd /usr/share/bacula-director
sudo -u postgres db_name=external /usr/share/bacula-director/create_postgresql_database
sudo -u postgres db_name=external /usr/share/bacula-director/make_postgresql_tables
sudo -u postgres db_name=external db_user=<some_user> db_password=<some_password> /usr/share/bacula-director/   grant_postgresql_privileges

Install the new API instance in the 9098 port. We followed this link: https://bacularis.app/doc/brief/installation.html#install-using-composer

Configure the new API Settings:

Configure the Catalog setting pointing to the new database.

Create a Wrapper command for the bconsole command and replace the default in the Console setting for all APIs, change the catalog name as it fits:

#!/bin/bash
catalog='MyCatalog'
cmd=$(cat)
echo -e "use catalog=${catalog}\n${cmd}" | /usr/sbin/bconsole "$@"

Create a API Host external in Bacularis Web Security pointing to the new API instance in port 9098.
Add the API Host external to the API default user.

Create a new Catalog directive in the director. We can use the Bacularis Web for that.

Catalog {
  Name = "External"
  Address = "localhost"
  Password = "password"
  User = "some_user"
  DbName = "external"
}

To use the new catalog a new client directive must be created pointing to the new catalog. If we are using the same client computer for multiple client/catalog directives, each client directive in the director must have a different port. In the client computer side, multiple FileDaemons must be created.

For our MacOS client implementation we have:

Two FileDaemon configuration files:

File for CLIENT1

# /Library/Preferences/bacula/bacula-fd.conf 
#
# List Directors who are permitted to contact this File daemon
#
Director {
  Name = bacula-dir
  Password = "password"
}
#
#
FileDaemon { 
  Name = CLIENT1
  FDport = 9102                
  WorkingDirectory = /usr/local/var/lib/bacula
  Pid Directory = /usr/local/var/run/bacula
  Maximum Concurrent Jobs = 20
  Maximum Network Buffer Size = 262144
  Heartbeat Interval = 60
}

# Send all messages except skipped files back to Director
Messages {
  Name = Standard
  director = baculaplr-dir = all, !skipped, !restored
}

File for CLIENT2

# /Library/Preferences/bacula/bacula-fd-2.conf                      # new file
#
# List Directors who are permitted to contact this File daemon
#
Director {
  Name = bacula-dir
  Password = "password"
   }
#
#
FileDaemon { 
  Name = CLIENT2                                                    # new client name
  FDport = 19102                                                    # new port
 WorkingDirectory = /usr/local/var/lib/bacula2                     # new directory
  Pid Directory = /usr/local/var/run/bacula2                        # new directory
  Maximum Concurrent Jobs = 1
  Maximum Network Buffer Size = 262144
  Heartbeat Interval = 60
}

# Send all messages except skipped files back to Director
Messages {
  Name = Standard
  director = baculaplr-dir = all, !skipped, !restored
}

Two plist files to start the daemons.

For CLIENT1

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Label</key>
        <string>org.bacula.bacula-fd</string>
        <key>ProgramArguments</key>
        <array>
                <string>/Applications/Bacula.app/Contents/Resources/BaculaBin/sbin/bacula-fd</string>
                <string>-f</string>
                <string>-c</string>
                <string>/Library/Preferences/bacula/bacula-fd.conf</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
</dict>
</plist>

For CLIENT2

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Label</key>
        <string>org.bacula.bacula-fd-2</string>
        <key>ProgramArguments</key>
         <array>
                <string>/Applications/Bacula.app/Contents/Resources/BaculaBin/sbin/bacula-fd</string>
                <string>-f</string>
                <string>-c</string>
                <string>/Library/Preferences/bacula/bacula-fd-2.conf</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
</dict>
</plist>

Create a new scratch pool Scratch-external. The job pools must have this scratch pool in the ScratchPool and RecyclePool directives.

Now, we must have labeled volumes in the new catalog. But first, for peace of mind, this volumes must be tagged in some way in the old catalog. For now, is better to do this from the bconsole.

Change catalog if necessary:

 use catalog=MyCatalog

If the volume needs to be purged, purged then as always (the purge command must be used with extreme caution! This will delete all job records for this volume from the catalog).

 purge volume=volume_name

We can change the scratch pool for this volume to the Scratch-external pool. This way Bacula do not use this volumes in the old catalog jobs.

update volume=volume_name pool=Scratch-external

This way we can track the volumes in the two catalogs, but this volumes can be remove from the old catalog using the bconsole delete command.!<

Next we import the volume to the new catalog:

use catalog=External
label barcodes slot=some_slot pool=Scratch-external
purge volume=volume_name

If the label command returns a error, we may need to rewind and write the eof in the tape. we can do this using the linux mt command.

Try to run a test job. In Bacularis change to the external API host (pulldown at the right top) to see the job in the Job History. If there is no job listed, this can be related to the Wrapper configuration.

Best regards,
Rui

gani

Hi Rui (@Plural),

Fantastic! Thank you for describing here all this process. Great job.

This will be very helpful for users.

Thanks!

Best regards,
Marcin Haba (gani)