← All notes

rsync vs rclone for nightly offsite backups

We moved a 2TB nightly backup from a rented box to object storage. The obvious question was whether to keep rsync and mount the bucket, or switch to rclone. The answer turned out to depend entirely on what a restore looks like.

Where rsync still wins

rsync's delta algorithm only pays off when both ends can read the file. Against a real filesystem over ssh, changing 40MB inside a 3GB file transfers roughly 40MB. It also preserves hardlinks, xattrs, and sparse files, which matters more than people expect when the thing you are backing up is a VM image.

rsync -aHAX --delete --numeric-ids \
      --partial --info=progress2 \
      /srv/data/ backup@offsite:/pool/data/

Where it falls apart

Point rsync at a FUSE-mounted bucket and the delta algorithm becomes a liability: it reads the whole remote file to compute checksums, so you pay full egress to upload a small change. The nightly job went from 20 minutes to over four hours.

rclone, with the flags that matter

rclone speaks the storage API directly and compares by size and modtime instead of rolling checksums. The defaults are conservative; these three changed the runtime the most for us:

FlagEffect
--transfers=16parallel uploads, the single biggest win
--fast-listone listing pass instead of per-directory calls
--backup-dirmoves overwritten files aside instead of destroying them
rclone sync /srv/data remote:backups/data \
  --transfers=16 --checkers=32 --fast-list \
  --backup-dir=remote:backups/history/$(date +%F) \
  --log-file=/var/log/rclone/nightly.log --log-level INFO

--backup-dir is the important one. Plain sync is a mirror, which means a corrupted source faithfully replicates the corruption and you have no previous copy. With a dated backup dir you get cheap versioning without paying for full snapshots.

The part everyone skips

Restore timing. Uploading 2TB overnight is fine; pulling it back during an incident at 40MB/s is nine hours you did not budget for. Test the restore path once a quarter and time it, or you do not have a backup, you have a receipt.