Borg Backup without root

I use Borg Backup as my current backup software. I'm not going into details about using borg here, I will show you how you can write a systemd service that can backup all files on your system without running as root. This also applies to other backup software not just borg.

My borg backup service has gone through multiple version until I reached a state that I was happy with. And to be honest, I'm very pleased with what I came up with in the end. I had two goals that took me some time to achieve. First I did not want to run borg as root and second I wanted it to be modular so other software could "inject" itself to be also backed up when the borg service runs.

This is how my current systemd service looks:

[Unit]
Description=Borg Backup

[Service]
Type=oneshot
User=borg
Group=borg
SupplementaryGroups=postdrop
ExecStart=/usr/local/sbin/backup.sh
ExecStartPost=/usr/bin/touch /backup.last

# filesystem access
ProtectSystem=strict
PrivateTmp=true
PrivateDevices=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true
ReadWritePaths=/backup/ /home/borg/.cache/borg/ /home/borg/.config/borg/ /var/spool/postfix/maildrop/ /backup.last

# network
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK

# misc
SystemCallArchitectures=native
NoNewPrivileges=true
RestrictRealtime=true
MemoryDenyWriteExecute=true
LockPersonality=true
ProtectHostname=true
RemoveIPC=true
RestrictSUIDSGID=true
ProtectClock=true

# capabilities
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
AmbientCapabilities=CAP_DAC_READ_SEARCH
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

This service is run daily by a systemd timer:

[Unit]
Description=Borg Backup

[Timer]
OnCalendar=daily

[Install]
WantedBy=timers.target

Basics

My backup works as follows: The borg backup script backups all specified directories and also the /backup directory. The /backup directory can be used by other software to write files they want to be backed up. The /backup directory is owned by the borg user and group. After the backup script finishes the systemd service touches the file /backup.last which can be used by monitoring software to check when the last successful backup was created. I use a temporary text file /backup/emailmessage.txt for borg and other services to write their log to, that will be send via email when the backup script finishes or encounters an error.

#!/bin/bash

set -e
set -u
set -o pipefail

_mail() {
    mail -s "$SUBJECT" "$EMAIL" < "$EMAILMESSAGE"
    truncate -s 0 "$EMAILMESSAGE"
}

_infomail() {
    SUBJECT="borgbackup on $THISHOST finished successfully"
    _mail
}

_info(){
    echo "$@"
    printf "$(date "+%Y-%m-%d %R:%S") %s\n" "$@" >> "$EMAILMESSAGE"
}

_err() {
    _info "$@"
    SUBJECT="ERROR during backup on $THISHOST"
    _mail
    exit 1
}

_init() {
    THISHOST={{ inventory_hostname }}
    EMAIL="{{ backup_mail }}"
    EMAILMESSAGE="/backup/emailmessage.txt"

    REPOSITORY="{{ backup_repo }}"
    BACKUPPFADE="{{ backup_paths }}"
}

_main() {
    _init
    _create
    _check
    _prune
    _infomail
}

_create() {
    _info "Running borg create"

    borg create -v --stats --compression lz4 --noatime "$REPOSITORY::$THISHOST-$(date +%Y-%m-%d-%R)" $BACKUPPFADE 1>>"$EMAILMESSAGE" 2>>"$EMAILMESSAGE"; OUT=$?

    if test $OUT -eq 0; then
        _info "borg create successful"
    else
        _err "borg create had problems, borg statuscode = $OUT"
    fi
}

_check() {
    _info "Running borg check"

    borg check -v "$REPOSITORY"; OUT=$?

    if test $OUT -eq 0; then
        _info "borg check successful"
    else
        _err "borg check had problems, borg statuscode = $OUT"
    fi
}

_prune() {
    _info "Running borg prune"

    borg prune -v --list "$REPOSITORY" --prefix "$THISHOST"- --keep-daily=30 --keep-weekly=12 --keep-monthly=12 1>>"$EMAILMESSAGE" 2>>"$EMAILMESSAGE"; OUT=$?

    if test $OUT -eq 0; then
        _info "borg prune successful"
    else
        _err "borg prune had problems, borg statuscode = $OUT"
    fi
}

_main

Not running as root

This was the easier of my two goals. Borg needs root to access and read all files on the system, or to be more specific, all files you want to backup. Also services not running as root can't send mails via sendmail

To achieve this, you can give a service the capability to open all directories and read all files in the system. The needed capability is CAP_DAC_READ_SEARCH and can be added to the service with AmbientCapabilities=CAP_DAC_READ_SEARCH.

To send mails the service needs to be able to create files in /var/spool/postfix/maildrop which can be achieved by adding the service to the postdrop group via SupplementaryGroups=postdrop. Because I also use ProtectSystem=strict I also need to add the path to ReadWritePaths. The service also needs to access netlink sockets which can be achieved by adding it to RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK.

Modularity

I use ansible to configure my servers and I don't want to edit my borg backup role or the backup script each time I add a new role / software that I want to backup. When my service was still running as root I could do this by using run-parts in my main backup script and then each role / software could create a backup file in some directory and it would be executed. Now that the script no longer has root privileges the scripts started by run-parts would often be missing privileges they need. For example to backup postgresql databases you need to run pg_dump as the postgres user.

[Unit]
Description=Backup PostgreSQL

[Service]
Type=oneshot
User=postgres
Group=postgres
SupplementaryGroups=borg
ExecStart=/usr/local/sbin/backup-postgresql.sh

# filesystem access
ProtectSystem=strict
PrivateTmp=true
PrivateDevices=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true
ReadWritePaths=/backup/

# network
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK

# misc
SystemCallArchitectures=native
NoNewPrivileges=true
RestrictRealtime=true
MemoryDenyWriteExecute=true
LockPersonality=true
RemoveIPC=true

My solution is to run the additional backup scripts via their own service files which include all the privileges needed. These service files are injected into the main backup service through a systemd conf files located in /etc/systemd/system/backup.service.d/. The file contains one line in the [Service] section: ExecStartPre=!/bin/systemctl start backup-postgresql.service. Normally this command would be executed as the user borg, because of the User=borg and fail, since only root can use systemctl start. The ! at the start of the command can be used to execute this single command as root instead without running the wholes service as root. This injection via /etc/systemd/system/backup.service.d/ can be done by any number of services. All ExecStartPre= command have to exit successfully or the backup service as a whole fails. I could prefix the ExecStartPre= comamnds with a - (minus) but I prefer the whole service to fail because I monitor successfully backups via the /backup.last file.

#!/bin/bash

set -e
set -u
set -o pipefail

_info(){
    echo "$@"
    printf "$(date "+%Y-%m-%d %R:%S") %s\n" "$@" >> /backup/emailmessage.txt
}

_err() {
    _info "$@"
    exit 1
}

_main() {
    _info "Create Postgres Dump of all Databases"

    DBS=$(psql -t -A -c 'SELECT datname FROM pg_database' | grep -P -v '(template0|template1|postgres)')

    for DB in $DBS; do
        pg_dump -Fc "$DB" -f /backup/postgres-"$DB".pg_dump 2>/backup/pg_dump.log; OUT=$?
        if test $OUT -eq 0; then
            _info "Postgres Dump of $DB successful"
        else
            _err "Postgres Dump $DB had problems, pg_dump statuscode = $OUT"
        fi
    done

    _info "Postgres Dump of all Databases successful"
}

_main

The additional systemd services need to be able to write to /backup which can be achieved by adding the group borg to the service with SupplementaryGroups=borg.

Sandboxing

Not all sandboxing options can be used. ProtectHome=true prevents the backup of the user home directories, PrivateUsers=true prevents borg from preserving the correct file permissions, LockPersonality=true prevents sending mails.

The directories /home/borg/.cache/borg and /home/borg/.config/borg need to be added to ReadWritePaths= to allow borg to work.

The current version of my Borg Backup systemd service can always be found in my GitHub repository: https://github.com/stephan13360/systemd-services/tree/master/borg