Fast backup to external drive with lz4

The simplest backup is to just tar-gz your home directory, but it’s painfullyslow. A friend put me on to lz4 which isn’t installed by default in ubuntu &mint, it moves the bottleneck from the compressor to disk i/o for my spinningrust usb disks.

The default output of this is basically silence for two hours while it runswhich isn’t great. Enter pv (pipeviewer)which can show you progress based on byte count.

Install lz4 and pv with:

sudo apt install liblz4-tool pv

And then pipe the output of tar through pv and into lz4

Here’s a quick script for backing up my home folder to an external drive;customize to suit your needs:

#!/bin/sh
# https://gist.github.com/timabell/68d112d66623d9a4a3643c86a93debee#file-backup-sh
echo "Opening/creating backup folder..."
mountpoint=/media/tim/backup/
cd "$mountpoint"
mkdir -p fox
cd fox
base=/home/
src=tim
echo "Getting source folder size..."
size_bytes=`du -sb "$base/$src" | awk '{print $1}'`
echo "$size_bytes bytes to backup."
echo "Backing up..."
tar -cpC $base $src -P | pv -s "$size_bytes" | lz4 >> "$(date -d "today" +"%Y%m%d-%H%M")-home.tar.lz4"
echo "Done."

This script is also available here:https://gist.github.com/timabell/68d112d66623d9a4a3643c86a93debee#file-backup-sh

Things to watch out for

Progress is inaccurate and variable because it’s based on input bytes processed vs total, but the speed is limited by output bytes to the spinning rust backup disk, and the ratio varies with the compressability of the input data.
Finding a file will require reading or decompressing the entire archive because of the way tar works.

Keeping it simple

There’s lots of great backup tools for backing up home but most of them createobscure custom formatted backups, often with incremental backups and chunked upfiles for efficient use of space. While that’s great I know from experiencethat you don’t always have all the right tools available to unpack such thingswhen you really need them, and there’s always the chance that you are missingan important incremental. In short complexity is worrying when it comes tobackup (and especially restore).