I am a big fan of the backup tool borgbackup. Borg is space efficient as it supports compression and deduplication and is also secure, so you don’t have to trust the storage provider where your backup will be stored.

In this short guide we will setup automatic backups for a server. But most of it can also be used to backup your clients. An addition to borg I use the wrapper borgmatic which will remove any custom scripts needed around borg. Everything related to backups will be stored in a single configuration file that configures borgmatic which will then perform borg operations.

As my homeserver is an Arch Linux machine this guide is specific for Arch, but as most distributions nowadays are also systemd based there should not be much of a difference than the used package manager or the versions of some packages.

Install needed tools

As both borg and borgmatic are in the official repositories they are easily installed with:

pacman -S borg borgmatic

Configure borgmatic

Borgmatic is configured in a single configuration file that will be stored in /etc/borgmatic/config.yaml. You can also use any other location and provide the path via -c path/to/my/config. A minimal configuration could be like this:

location:
    source_directories:
        - /home
        - /mnt/storage

    repositories:
        - user@my-backup-host:/mnt/backups/server

retention:
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6

storage:
    encryption_passphrase: "my-secret-1337"
    compression: zstd,22

consistency:
    checks:
        - repository
        - archives

This means backup everything in the directories home and mnt/storage to /mnt/backups/server on the host my-backup-host. We also want to keep 7 daily backups, 4 weekly backups and 6 monthly backups. The password to access backups is my-secret-1337 and the used compression is zstd with the highest compression (ranges from 0 (almost no compression, but fast) to 22 (the highest compression, but slow)). Read more about the compression options borg provides here.

Setup remote server and ssh access

In this case we will backup to a remote server via ssh. Therefore generate an ssh key pair on the server you want to backup with ssh-keygen. And copy the public key to the remote server with ssh-copy-id user@my-backup-host. After that you should be able to login to the remote backup server.

You also need borg installed on the remote machine (in my case also Arch Linux). As the backups created by borg are stored encrypted you can use the same backup server for multiple clients. But please mind that the backups are only encrypted if specified at the initial creation of the repository, which we will handle now.

Setup borg repository

Borg provides different encryption models which are documented here. In my case I want encrypted and authenticated backups these options are also the fastest on most modern AMD/Intel CPUs (as the documentation states). As we use borgmatic as a borg wrapper we use borgmatic init instead borg borg init.

borgmatic init -e repokey-blake2

Start the initial backup

Backups can now be started with borgmatic. If you want further information like which file is currently backed up or some statistics at the end use borgmatic --files --stats.

Automate backups with systemd

These systemd units are based on the offical ones from borgmatic. I prefer systemd-timers over a simple cron daemon for two reasons:

Systemd will ensure that the job will be executed even if the system is down at the specific time the job was scheduled.
Systemd will ensure all logs are written into the journal. With that they easy to inspect via journalctl -u borgmatic.

Store this unit as borgmatic.service in /etc/systemd/system.

[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target
ConditionACPower=true

[Service]
Type=oneshot

LockPersonality=true
MemoryDenyWriteExecute=no
NoNewPrivileges=yes
PrivateDevices=yes
PrivateTmp=yes
ProtectClock=yes
ProtectControlGroups=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
ProtectSystem=full

CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW

Nice=19
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOWeight=100

Restart=no
LogRateLimitIntervalSec=0

ExecStartPre=sleep 1m
ExecStart=systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" borgmatic --syslog-verbosity 1

Reload all units with systemctl daemon-reload and test the unit with systemctl start borgmatic. You should see that a backup is running via journalctl -u borgmatic -f.

If everything is running as expected create a borgmatic.timer with the following content:

[Unit]
Description=Run borgmatic backup

[Timer]
OnCalendar=*-*-* 4:00:00
Persistent=true

[Install]
WantedBy=timers.target

OnCalendar uses the format WeekDay Year-Month-Day Hour:Minute:Second so if you want a backup running every day at 2 am: *-*-* 2:00:00. If you want to run multiple backups simply add addition OnCalendar entries to the timer.

Enable the timer with systemctl enable borgmatic.timer and check its status:

systemctl list-timers
NEXT                         LEFT     LAST                         PASSED       UNIT                         ACTIVATES
Mon 2021-04-19 00:00:00 CEST 8h left  Sun 2021-04-18 00:00:10 CEST 15h ago      shadow.timer                 shadow.service
Mon 2021-04-19 00:40:11 CEST 8h left  Mon 2021-04-12 00:32:56 CEST 6 days ago   fstrim.timer                 fstrim.service
Mon 2021-04-19 04:00:00 CEST 12h left n/a                          n/a          borgmatic.timer              borgmatic.service
Mon 2021-04-19 12:59:22 CEST 21h left Sun 2021-04-18 12:59:22 CEST 2h 46min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service

4 timers listed.

Your server will now create a backup every night at 4:00 am.

Check existing backups

To list all existing backups use borgmatic list.

Restore a backup

In my opinion the easiest way to restore a backup is to use the borg mount feature. borg mount allows you to mount a created backup to a specified directory. On Arch you have to install python-llfuse with pacman -S python-llfuse. For example:

# borgmatic list
my-backup-server:my-repo: Listing archives
Server-2021-04-16T16:36:35.431066       Fri, 2021-04-16 16:36:36 [0a1b2933a2f1cab81ab164c7ef01370793c5d688710b1c29e94d8d1a8f22c414]
Server-2021-04-18T12:52:59.997607       Sun, 2021-04-18 12:53:01 [9e2b0d1b635c36ede5653135821cdcc9c5181615c5cc0b4b8b06ef1e81ff29b8]

# mount last backup to ~/mount
borg mount my-backup-server:my-repo::Server-2021-04-18T12:52:59.997607  ~/mount

You can now access the backup in ~/mount. Simply copy the files wherever you want.

Conclusion

borg and borgmatic are perfect tools to automate backups. There are advanced features like append only backups to make it impossible to delete backups if a server has been compromised. Overall I am very happy with by backup setup as it requires no maintenance and simply does its job. And that’s exactly how backups should be - just working and boring.

Install needed tools#

Configure borgmatic#

Setup remote server and ssh access#

Setup borg repository#

Start the initial backup#

Automate backups with systemd#

Check existing backups#

Restore a backup#

Conclusion#