Back up your server

A backup is a copy of your data somewhere else, that you've tested you can get back. We don't back up servers for you yet (it's coming), so this guide sets up nightly backups you run yourself.

What to back up#

The system reinstalls in minutes. What you can't get back is this:

Set up restic#

restic encrypts backups before they leave the server, stores only what changed, and keeps a history you can restore from by date, in any S3-compatible bucket or on another server over SFTP.

sudo apt install -y restic

sudo dnf install -y epel-release && sudo dnf install -y restic

Create a bucket with your storage provider, and an access key for it. You need the endpoint, the bucket name, the key ID and the secret.

Make a folder for restic's settings, readable only by root:

sudo install -d -m 700 /etc/restic

Make a long random password that encrypts the backups:

sudo sh -c 'openssl rand -base64 32 > /etc/restic/password && chmod 600 /etc/restic/password'

Save the storage details, with your own in place of the examples:

sudo tee /etc/restic/env <<'EOF'
RESTIC_REPOSITORY=s3:https://s3.example.com/web-01-backups
RESTIC_PASSWORD_FILE=/etc/restic/password
AWS_ACCESS_KEY_ID=replace-with-your-key-id
AWS_SECRET_ACCESS_KEY=replace-with-your-secret
EOF

sudo chmod 600 /etc/restic/env

The AWS_ names work for any S3-compatible storage. For SFTP instead, set RESTIC_REPOSITORY=sftp:[email protected]:/backups/web-01, leave out the two keys, and make sure the server can log in there with an SSH key.

Now become root, load the settings, and create the repository:

sudo -i

set -a; . /etc/restic/env; set +a

restic init

The backup script#

This dumps your databases, backs up the folders that exist, and keeps one snapshot a day for a week, a week for a month, and a month for six months:

sudo tee /usr/local/bin/backup <<'EOF'
#!/bin/sh
set -eu
set -a; . /etc/restic/env; set +a

# Database dumps go here first. Uncomment the one you use.
mkdir -p /var/backups/db
# runuser -u postgres -- pg_dumpall > /var/backups/db/postgres.sql
# mysqldump --all-databases --single-transaction > /var/backups/db/mysql.sql

paths=""
for p in /etc /home /root /opt /var/www /var/backups/db; do
  [ -e "$p" ] && paths="$paths $p"
done

restic backup --exclude-caches $paths
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
EOF

sudo chmod 700 /usr/local/bin/backup

Add any other data folders to the for line, then run it once by hand:

sudo /usr/local/bin/backup

The first run sends everything. After that, only changes go.

Run it every night#

A systemd timer runs it at 3:30am, give or take half an hour, or when the server is back if it was off.

sudo tee /etc/systemd/system/backup.service <<'EOF'
[Unit]
Description=Back up the server with restic
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup
EOF

sudo tee /etc/systemd/system/backup.timer <<'EOF'
[Unit]
Description=Back up the server every night

[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload && sudo systemctl enable --now backup.timer

See when it runs next, and how the last run went:

systemctl list-timers backup.timer

sudo journalctl -u backup -n 50 --no-pager

Restore something#

A backup you've never restored from is a hope. Try it now, and every few months.

As root, with the settings loaded (sudo -i, then the set -a line above), list your snapshots:

restic snapshots

See what's in the latest one:

restic ls latest /etc/caddy

Restore a folder into /tmp/restore, so you can look before copying anything back:

restic restore latest --target /tmp/restore --include /etc/caddy

The files land under /tmp/restore/etc/caddy. To restore from an older date, use a snapshot's ID from the list instead of latest.

Every so often, check the repository itself is healthy:

restic check

Simpler: a copy on another machine#

rsync can copy your folders to another always-on Linux machine. It keeps only the latest copy, so a mistake is copied over too the next night. Better than nothing; restic is better.

rsync -aAX --delete /etc /home /opt /var/www [email protected]:/backups/web-01/

See Move files to and from your server for how rsync works.