This post was written to acknowledge bacula has saved my ass again. My main desktop machine failed, a new replacement was purchased and swapped in, new OS installed with same hostname and ip-address. Then installed puppet and let the puppet agent do its thing, one of its tasks was to install and configure bacula-fd. So within 10 minutes of starting the puppet agent I could use bacula to restore all custom server configuration files and everything under /home, as the last bacula backup ran only a few hours before the server died I lost zero data. As bacula is wonderfull it deserves another post.
First be aware bacula is most useful if you have multiple machines (physical or virtual) to backup to a seperate dedicated physical server.
The reason is simply your data is going to be available; if the backup server fails in any way your actual servers being backed up lose no data and you can quickly rebuild tha backup server and do a full backup of all the machines to recreate a backup point for them all (yes, you will have lost backups of individual files you may have accidentally deleted prior to that date), and if a server fails you can simply rebuild that and restore from the backup server.
If you have only one machine bacula would still be useful if you needed to restore one or more files from a specific date if needed should you accidentally delete or corrupt them, although obviously if that server fails you have lost everything.
This post is on how to set it up in a home lab environment. My environment is a Fedora30 backup server used to backup Fedora and CentOS servers. I use only one storage daemon with maximum concurrent jobs zet to 1 so jobs queue to ensure only one job runs at a time, and I use multiple schedules.
While it is a large post, with what may seem like a lot of manual effort and time to set bacula up, it is in fact extremely simple to setup and get running and you will probably spend less than 10 minutes in an editor on the configuration files, unless you have dozens of clients to add to be backed up.
So the ideal configuration would be
- a dedicated backup server with a lot of storage (or usb ports for a very large external disk), or at least one always-on machine that can do the backups
- multiple servers to be backed up
The requirement for a lot of storage or a very large external disk and considerations to take into account are
- even though bacula does compress the data used by backups you must remember that multiple backups will be kept depending upon your configuration, for example you may do a full backup once every 30 days and incremental backups across those 30 days, if you change a lot of large files dailt you will need more space
- you should size for the maximum, for example if you are backing up machines with 1Tb disks that are only 30% used in the 30 day backup cycle mentioned above your backups would probably only take around 400Gb or less per machine (I backup 9 machines with a combined actual filesystem usage of around 3.5Tb using the 30 day cycle mentioned above and all the backups fit (just) into 2Tb of space), however over time space usage on the machines being backed up will grow so you should size for the maximum space that could be used not what is currently used
- what I use as a backup storage device for my home lab is an external 2Tb disk, the major benefit of using a single external disk is that as backup space needed grows you can simply replace it with a larger device on the same mount point (after copying the backup data off the old disk onto it of course)
- do not grow space by linking multiple disks into a pool or union, because if any single one of those disks fail you have lost all your backups anyway, better to use a single large disk external or otherwise so if it fails you simply replace it without needing to reinstate pool or union structures
- if the backup device used by bacula on the backup machine fails with data loss and needs to be replaced with an empty device you need to drop and recreate the database and delete all the work files it uses, then do full backups on all the machines. The reason for that is you do not want the bacula backup service to try to append to or recall from data that no longer exists
- if sized correctly the disk based virtual backup volumes will just be re-used resulting in static space usage once they have all been used and start recycling. Personally I hit an issue when a few 100Gb that should have been excluded got backed up and I had to manually label a few extra volumes, to avoid having to manually delete those when they eventually expired I changed my configuration to “Action On Purge = Truncate” which does not affect existing volumes, only new volumes created so for existing volumes from bconsole update them all as per this example : “update volume=Inc-0091 ActionOnPurge=Truncate”
Update: 14Mar2021 new commands are introduced in more recent versions of bacula
to work in conjunction with the "Action on purge = truncate" option. My Catalog
backup job now looks like this.
Job {
Name = "BackupCatalog"
JobDefs = "DefaultJob"
Client = vmhost1-fd
Level = Full
FileSet="Catalog"
Schedule = "WeeklyCycleAfterBackup"
RunBeforeJob = "/usr/libexec/bacula/make_catalog_backup.pl bacula"
# I take a copy of the catalog backup and move it to my database dump/backup directory
RunAfterJob = "/var/spool/bacula/marks_post_BackupCatalog"
Write Bootstrap = "/var/spool/bacula/%n.bsr"
Priority = 11 # run after main backup
RunScript {
RunsWhen=Before
RunsOnClient=No
Console = "prune expired volume yes"
}
RunScript {
RunsWhen=After
RunsOnClient=No
Console = "purge volume action=all allpools storage=File"
Console = "truncate storage=File pool=File-Incr-Pool"
Console = "truncate storage=File pool=File-Full-Pool"
}
}
The actual software requirements needed to run bacula are
- on the backup server itself
- the bacula-fd service, this is on all servers and is used as the client to perform backups/restores
- the bacula-sd service, this is the storage service, it requires the bacup device to be available before it is started. Note that the bacula-sd service can be on a seperate machine, and you can in fact have multiple servers running a bacula-sd service in your environment which allows not just spreading the load but the setup of copy and migration jobs to get ‘offsite’ backups, nice but not really relevant for a home lab
- the bacula-dir service, this is the director service that runs the backup jobs on schedule and runs ad-hoc backup and restore jobs
- the bacula-console package, the command line interface needed to run restore jobs, control scheduled jobs and servers (cancel backups, disable/enable servers to be backed up etc)
- mariadb is what I prefer as the database to be used, it also supports postgress and sqlite3 but as I use mariadb only that is covered in this post
when using external disks the bacula-sd and bacula-dir services should be started (in that order) only after the external disks have been mounted so do not autostart those if using external disks
- on all the servers to be backed up
- the bacula-fd service, this is on all servers and is used as the client to perform backups/restores
- the bacula-console package, the command line interface needed to run restore jobs for the server, and all the admin functions mentioned in the server section
- optionally install the bacula-console-bat package on desktops as it needs gtk for a gui interface, personally I find the command line interface easier to use
The configuration that should be set before running bacula should be
- On the backup server itself, the one running the bacula-dir service
- look in /usr/libexec/bacula, you need to check/edit run the scripts create_mysql_database, make_mysql_tables, grant_mysql_privileges to create the database to be used
- On the backup server itself in /etc/bacula
- bacula-sd.conf : storage configuration
- set all the passwords, they must match the ones used in bacula-dir.conf, bacula-fd.conf, bconsole.conf
- set the archive device file path, it must be a mounted filesystem with enough space to hold all your backups, this file path cannot easily be changed later so get it correct now. This may be external media as long as you can guarantee it will be mounted before the bacula-sd service starts after reboots, make sure automatic mount is set to yes and always open is set to no
- comment out all the autochange/tape entries. If these are not commented the bacula-sd service will hand scanning for tape devices on startup
- bacula-dir.conf : jobs, filesets, schedule, clients (servers to backup), storage servers to use, sql database user/password/port, volume pools, pretty much everything
- set all the passwords, they must match the ones used in bacula-sd.conf, bacula-fd.conf, bconsole.conf
- search for the Catalog entry, set the user/password/databasename etc to match the database you created earlier. Note that the supplied example with dbuser and dbpassword parameters does not work, use something like
Catalog { Name = bacula dbname = "bacula" user = "bacula" password = "bacula" DB address = "vmhost1" DB port = "3306" } - make sure the ‘mail’ address is set correctly to something that works, bacula manual “intervention required” errors are emailed. note: mail address configuration occurs twice in the file, setup both
- set up multiple schedules to be used, you do not want all your backups run simultaneously (although they can queue). Note: a job defined without a schedule will not be automatically scheduled byt can still be run on demand
- set up your fileset entries to define the filesystems to be backed up. Bacula will not bacup multiple filesystems so if you have a standard fedora installation with a seperate /home directory you will need at least two filesets, one for / and one for /home. It is in the fileset definition you specify exclude entries, customise the exclude entries for your servers before running any backups; for example if you run lots of virt-manager VMs you may want to exclude /usr/lib/libvirt/images which may contain many hundreds of GBs of data in disk images that if the VMs are running and changing the virtual disk access time would be backed up with every incremental backup (basically exclude huge files that change often other than important things like database backups); if you do want to backup things like that create a seperate fileset and non-scheduled job to use the fileset and do ad-hoc backups rather than every day
- update the “default” jobsdef entry as it can be used by all your jobs
- edit the job definition for RestoreFiles which is the default job used when restores are requested, specifically change the “Where” entry from /tmp/bacula-restores to somewhere you have a lot of space, I change it to /var/tmp/bacula-restores; the difference being on Fedors /tmp is a ‘tmpfs’ filesystem using real memory and /var/tmp is on disk; I can restore 100Gb onto disk but not into memory :-). Note: CentOS7 has /tmp on disk so would be ok, but the job definition is used by all clients so cater for all your client OSs
- create client entries for every server you intend to backup, the password configured in the client entry has to match that set in the bacula-fd.conf on the client server. Unless you are in a super secure environment use the same password for all client entries or al least the same format as that will make it much simpler to manage rolling out the bacula-fd.conf file to every client. Also ensure your client names follow a fixed naming convention, for example servers wally and fred could have client name entries of wally-fd and fred-fd, making automated client deployment easier
- Now you can create backup “job” definitions for the clients. Simple standards would be if the client is a Fedora server named Wally you would have two jobs BackupWally and BackupWallyHome if you created seperate filesets for the two filesystems, and may even have them on different schedules. If you add a schedule entry to the job it will run automatically when scheduled, no schedule entry and the jobs are only available for ad-hoc backup requests
- check all the entries to make sure ‘max full interval’,’file retention’,’job retention’ values throughout the file all make sense. For example a full backup retention of 14 days with an incremental backup retention of 30 days does not make sense, bacula is smart enough to determine during an incremental run that it has no full backup and if your job definition is setup correctly will run a full backup instead of an incremental you may not want a full backup every 14 days, so make sure your full backup expiries closely match your incremental ones
- open firewall ports 9101, 9102, 9103 (bacula-dir, bacula-fd, bacula-sd)
- for a home lab set “Maximum Concurrent Jobs = 1” in the Director entry of bacula-dir.conf, this ensures only one backup job runs at a time and others are queued after each other, to avoid stress on your lab network and backup server disks
- in the BackupCatalog job the default RunAfter job deletes the disk file sql backup file created after bacula has backed it up, personally I replace this job with a custom one that takes a backup of it (/var/spool/bacula/bacula.sql) to the same external storage device I use for the bacula virtual backup volumes before deleting it as it seems silly to me to only be able to access it to rebuild bacula if bacula is actually running and able to restore it, so I keep a copy. You may want to do the same
- bacula-sd.conf : storage configuration
- On the all servers to be backed up including the backup server itself in /etc/bacula
- bacula-fd.conf
- set all the passwords, they must match the ones used in bacula-dir.conf
- in the filedaemon section set the ‘name’ to match the client name you used when defining the client on the backup server in bacula-dir.conf, they must match
- open firewall port 9102 (bacula-fd)
- bconsole.conf
- set the hostname of the backup server and the director passord to match the one you set in bacula-dir.conf
- ensure the /etc/bacula/bconsole.conf file is readable by all users that need to perform restores and issue admin commands to the director, you should probably restrict access as the interface grants full admin access although retricting access means users cannot restore their own files and must always bug an admin user to do so, not as mch of an issue in a home lab enviroment, personally I recomend securing all conf files root:bacula 640 and add backup admins (only yourself in a home lab case) to the bacula group
- optionally if you installed the bacula-console-bat package ‘cp -p bconsole.conf bat.conf’, as they are identical in configuration
- bacula-fd.conf
On the client servers you can ‘systemctl enable bacula-fd’ and ‘systemctl start bacula-fd’ immediately on all the client servers, there will be no activity from them until the bacula director requests a backup, whether manual or scheduled.
I would suggest manually logging into mariadb to make sure the userid/password works and you can access the bacula tables before starting anything on the backup server.
When happy make sure the archive file path is mounted and ‘systemctl enable bacula-sd’ and ‘systemctl start bacula-sd’. Check messages in /var/log/messages and ‘systemctl status bacula-sd’ to make sure there were no errors.
When happy ‘systemctl enable bacula-dir’ and ‘systemctl start bacula-dir’, again check messages for startup errors.
If all OK they will start automatically on a system reboot.
It is important to note that if your storage configuration is configured to use external media you should not enable the bacula-sd and bacula-dir services to autostart on a reboot as there is no guarantee the external media will spin up in time to be mounted and available, when using external storage manually start the services in the order bacula-sd then bacula-dir after checking the external media is available.
Tips on configuraion
- I use passwords that include the server name, so template deployment can be done onto servers
- I use bacula-fd client names that include the server name, so template deployment can be done onto servers
- as the SD task can be run and used on multiple servers should you have a huge home lab, include the server name in the name of the bacula-sd name so template deployment can be done onto servers
- if your bacula-dir.conf file is getting too large to maintain the @ symbol can be used to include files (ie: @/etc/bacula/client1.conf in bacula-dir.conf would include that file when the director starts) so you can split sections out. Also as the @ symbol can also be used within included files to include additional files it is easily possible to create structured/managed configurations.
Security tips, and issues
- users cannot be allowed to restore their own files, simply because bconsole is a full admin interface and you cannot limit a user to only restore functions whilst omitting destructive functions
- another reason general users must be kept away is that all the configuration files for bacula have hard coded passwords to chat between services and to use the sql database, note however this is not unique to bacula and most *nix applications have passwords stored in configuration files, sometimes hashed and sometimes not
- these are not security issues reguardless of what an auditor may tell you, as long as the configuration files are secured so only admin users have access to read them (which is needed to perform their jobs) there is no exposure to non-admin users so not an issue. However as not all admin users (in fedora those in the wheel group) will be tasked with doing restores add only those that need access to the bacula group also, the config files should all be root:bacula 640 (the bacula user must be able to read the files so leave the group bacula, not root or wheel or bacula can never start, unless you make the perms 644 instead of 640 and let everybody on the server read the config files, do not do that)
- the only issue is that in a secure world sysadmins have to do all the restores, not ideal but common across every *nix backup solution I have ever seen
Tips on usage
- bconsole can do everything
- if you are running short on disk space on the bacula storage device use ‘list volumes’ to check for any expired volumes you can delete with the delete volume command. This should not be necessary if you have configured the volumes for truncation however
- make damb sure you have changed the default restore location as mentioned in the setup steps if you are to restore GBs of data
- when using bconsole to restore files if you are not sure of the exact filenames use the “6: Select backup for a client before a specified time” option, this builds a file list and drops you into a ‘shell’ where you can cd between backed up directories and select files or entire directories to ‘mark’ for restore
- scheduling issues, if you have a few jobs scheduled to run at 17:00 daily and the director is stopped over the 17:00 period, when it is restarted the jobs will still be scheduled to run at 17:00, the next day. There is no catchup on missed jobs so just do not shutdown the director over backup periods
- setup a job to periodically check the last 40 or so messages in /var/log/bacula/bacula.log to check for manual intervention cases, I have a nrpe job that checks for manual ‘label’ requests required that occur when max allowed volumes has been exceeded and one must be manually created. All errors such as that will be emailed also, so ensure you have the email address setup correctly, but who checks their emails all day long
Update: 14Mar2021 – changes in bacula no longer log mount messages to the log file, I now have created a new user and mount (any action) events are mailed to that user and my nrpe check job now checks if a file exists for that user under /var/spool/mail and raises an alert if it does
