Borg Backup without root

I use Borg Backup as my current backup software. I'm not going into details about using borg here, I will show you how you can write a systemd service that can backup all files on your system without running as root. This also applies to other backup software not just borg.
My borg backup service has gone through multiple version until I reached a state that I was happy with. And to be honest, I'm very pleased with what I came up with in the end. I had two goals that took me some time to achieve. First I did not want to run borg as root and second I wanted it to be modular so other software could "inject" itself to be also backed up when the borg service runs.
This is how my current systemd service looks:
[Unit]
Description=Borg Backup
[Service]
Type=oneshot
User=borg
Group=borg
SupplementaryGroups=postdrop
ExecStart=/usr/local/sbin/backup.sh
ExecStartPost=/usr/bin/touch /backup.last
# filesystem access
ProtectSystem=strict
PrivateTmp=true
PrivateDevices=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true
ReadWritePaths=/backup/ /home/borg/.cache/borg/ /home/borg/.config/borg/ /var/spool/postfix/maildrop/ /backup.last
# network
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
# misc
SystemCallArchitectures=native
NoNewPrivileges=true
RestrictRealtime=true
MemoryDenyWriteExecute=true
LockPersonality=true
ProtectHostname=true
RemoveIPC=true
RestrictSUIDSGID=true
ProtectClock=true
# capabilities
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
AmbientCapabilities=CAP_DAC_READ_SEARCH
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
This service is run daily by a systemd timer:
[Unit]
Description=Borg Backup
[Timer]
OnCalendar=daily
[Install]
WantedBy=timers.target
Basics
My backup works as follows: The borg backup script backups all specified directories and also the /backup directory. The /backup directory can be used by other software to write files they want to be backed up. The /backup directory is owned by the borg user and group. After the backup script finishes the systemd service touches the file /backup.last which can be used by monitoring software to check when the last successful backup was created. I use a temporary text file /backup/emailmessage.txt
for borg and other services to write their log to, that will be send via email when the backup script finishes or encounters an error.
#!/bin/bash
set -e
set -u
set -o pipefail
_mail() {
mail -s "$SUBJECT" "$EMAIL" < "$EMAILMESSAGE"
truncate -s 0 "$EMAILMESSAGE"
}
_infomail() {
SUBJECT="borgbackup on $THISHOST finished successfully"
_mail
}
_info(){
echo "$@"
printf "$(date "+%Y-%m-%d %R:%S") %s\n" "$@" >> "$EMAILMESSAGE"
}
_err() {
_info "$@"
SUBJECT="ERROR during backup on $THISHOST"
_mail
exit 1
}
_init() {
THISHOST={{ inventory_hostname }}
EMAIL="{{ backup_mail }}"
EMAILMESSAGE="/backup/emailmessage.txt"
REPOSITORY="{{ backup_repo }}"
BACKUPPFADE="{{ backup_paths }}"
}
_main() {
_init
_create
_check
_prune
_infomail
}
_create() {
_info "Running borg create"
borg create -v --stats --compression lz4 --noatime "$REPOSITORY::$THISHOST-$(date +%Y-%m-%d-%R)" $BACKUPPFADE 1>>"$EMAILMESSAGE" 2>>"$EMAILMESSAGE"; OUT=$?
if test $OUT -eq 0; then
_info "borg create successful"
else
_err "borg create had problems, borg statuscode = $OUT"
fi
}
_check() {
_info "Running borg check"
borg check -v "$REPOSITORY"; OUT=$?
if test $OUT -eq 0; then
_info "borg check successful"
else
_err "borg check had problems, borg statuscode = $OUT"
fi
}
_prune() {
_info "Running borg prune"
borg prune -v --list "$REPOSITORY" --prefix "$THISHOST"- --keep-daily=30 --keep-weekly=12 --keep-monthly=12 1>>"$EMAILMESSAGE" 2>>"$EMAILMESSAGE"; OUT=$?
if test $OUT -eq 0; then
_info "borg prune successful"
else
_err "borg prune had problems, borg statuscode = $OUT"
fi
}
_main
Not running as root
This was the easier of my two goals. Borg needs root to access and read all files on the system, or to be more specific, all files you want to backup. Also services not running as root can't send mails via sendmail
To achieve this, you can give a service the capability to open all directories and read all files in the system. The needed capability is CAP_DAC_READ_SEARCH and can be added to the service with AmbientCapabilities=CAP_DAC_READ_SEARCH
.
To send mails the service needs to be able to create files in /var/spool/postfix/maildrop which can be achieved by adding the service to the postdrop group via SupplementaryGroups=postdrop
. Because I also use ProtectSystem=strict
I also need to add the path to ReadWritePaths. The service also needs to access netlink sockets which can be achieved by adding it to RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
.
Modularity
I use ansible to configure my servers and I don't want to edit my borg backup role or the backup script each time I add a new role / software that I want to backup. When my service was still running as root I could do this by using run-parts in my main backup script and then each role / software could create a backup file in some directory and it would be executed. Now that the script no longer has root privileges the scripts started by run-parts would often be missing privileges they need. For example to backup postgresql databases you need to run pg_dump as the postgres user.
[Unit]
Description=Backup PostgreSQL
[Service]
Type=oneshot
User=postgres
Group=postgres
SupplementaryGroups=borg
ExecStart=/usr/local/sbin/backup-postgresql.sh
# filesystem access
ProtectSystem=strict
PrivateTmp=true
PrivateDevices=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true
ReadWritePaths=/backup/
# network
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
# misc
SystemCallArchitectures=native
NoNewPrivileges=true
RestrictRealtime=true
MemoryDenyWriteExecute=true
LockPersonality=true
RemoveIPC=true
My solution is to run the additional backup scripts via their own service files which include all the privileges needed. These service files are injected into the main backup service through a systemd conf files located in /etc/systemd/system/backup.service.d/. The file contains one line in the [Service]
section: ExecStartPre=!/bin/systemctl start backup-postgresql.service
. Normally this command would be executed as the user borg, because of the User=borg and fail, since only root can use systemctl start. The ! at the start of the command can be used to execute this single command as root instead without running the wholes service as root. This injection via /etc/systemd/system/backup.service.d/ can be done by any number of services. All ExecStartPre= command have to exit successfully or the backup service as a whole fails. I could prefix the ExecStartPre= comamnds with a - (minus) but I prefer the whole service to fail because I monitor successfully backups via the /backup.last file.
#!/bin/bash
set -e
set -u
set -o pipefail
_info(){
echo "$@"
printf "$(date "+%Y-%m-%d %R:%S") %s\n" "$@" >> /backup/emailmessage.txt
}
_err() {
_info "$@"
exit 1
}
_main() {
_info "Create Postgres Dump of all Databases"
DBS=$(psql -t -A -c 'SELECT datname FROM pg_database' | grep -P -v '(template0|template1|postgres)')
for DB in $DBS; do
pg_dump -Fc "$DB" -f /backup/postgres-"$DB".pg_dump 2>/backup/pg_dump.log; OUT=$?
if test $OUT -eq 0; then
_info "Postgres Dump of $DB successful"
else
_err "Postgres Dump $DB had problems, pg_dump statuscode = $OUT"
fi
done
_info "Postgres Dump of all Databases successful"
}
_main
The additional systemd services need to be able to write to /backup which can be achieved by adding the group borg to the service with SupplementaryGroups=borg
.
Sandboxing
Not all sandboxing options can be used. ProtectHome=true prevents the backup of the user home directories, PrivateUsers=true prevents borg from preserving the correct file permissions, LockPersonality=true prevents sending mails.
The directories /home/borg/.cache/borg and /home/borg/.config/borg need to be added to ReadWritePaths= to allow borg to work.
The current version of my Borg Backup systemd service can always be found in my GitHub repository: https://github.com/stephan13360/systemd-services/tree/master/borg