duplicity: simple and powerful backups from the CLI

Translation of the original in Catalan. Read it in: CatalàCastellano

Duplicity is a python based application that for the moment has no graphical frontend, but like the good things in life it is used from the CLI. It is aimed at backups and its main strength lies in relying on rdiffdir/rdiff/rsync to make differential/incremental backups. This may not seem like any distinctive feature if it were not because it supports a good bunch of systems to store the backups on:

  • ssh/scp

  • local file access

  • rsync

  • ftp

  • HSI

  • WebDAV

  • Amazon S3

So far I have already been able to check that both via Amazon S3 and via FTP, it works wonderfully. This, as you can see, is a different story, since making a system of differential and/or incremental backups against an FTP or against S3 is not trivial. It is also made following all the standards, it is efficient in bandwidth management and it supports public key and private key encryption.

Next I attach a small cookbook of the parameters that I use the most:

  • Make a backup against an FTP of the whole system, the backup will be made incremental as long as the oldest incremental copy is not more than 7 days old, then a 'full' copy will be made. It says that the files that will be uploaded to the FTP have a size of 10Mb. The /dev, /proc, /tmp, /mnt, /media directories will not be copied

PASSPHRASE=clausimetrica duplicity incremental --full-if-older-than 7D -v4 --volsize 10 --exclude=/dev --exclude=/proc --exclude=/tmp --exclude=/mnt --exclude=/media / ftp://user:pass@hostname
  • See a list of the files we have made a copy of:

PASSPHRASE=clausimetrica duplicity list-current-files ftp://user:pass@hostname
  • Check which copies are available in the repository:

PASSPHRASE=clausimetrica duplicity collection-status  ftp://user:pass@hostname

To restore the backups, you have to think that often what you want is to recover some file or directory and not the whole copy. So here these details will be taken into account.

  • We recover the whole backup entirely:

PASSPHRASE=clausimetrica duplicity ftp://user:pass@hostname /tmp/restore
  • We recover a file or directory:

PASSPHRASE=clausimetrica duplicity --file-to-restore nom_fitxer_o_directori_a_recuperar ftp://user:pass@hostname /tmp/restore
  • We recover a specific file or directory from 3 days ago:

PASSPHRASE=clausimetrica duplicity -t 3D --file-to-restore nom_fitxer_o_directori_a_recuperar ftp://user:pass@hostname /tmp/restore

As you can check the use of the tool is quite simple and easy to place inside a crontab, but really if you need more complex combinations you have to take a good look at man duplicity.

UPDATE (2015/08/17) How to install duplicity on Ubuntu 12.04

apt-get install -y python-paramiko python-gobject-2 python-dev build-essential python-pip python-software-properties
add-apt-repository ppa:duplicity-team
apt-get update
apt-get install -y duplicity

UPDATE (2016/08/02) Full backup of Ubuntu Linux

PASSPHRASE=your_password duplicity incremental --full-if-older-than 7D -v4 --volsize 10 --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / rsync://the_user:the_password@host_server::/path/

3 comments

Comments from the old blog, as they were written. Comments are closed.

  1. Gerard Castell

    Hem pareix una aplicació fantastica, esitic pensant d'implementar un sistema de copies de seguretat per al meu sistema (debian) i m'agrada la idea de poder conectar amb amazon S3, saps on puc trovar la documentació amb exemples de conexió?? Salut company :D

  2. Oriol Rius

    La documentació del duplicity??? doncs a la pàgina del mateix. Si vols exemples per això he fet l'article per tenir exemples sinó hauràs de tirar de 'man'. Ho sento però no sé més del tema.

  3. Oriol Rius

    Un gran complement per aquesta eina de backup és: trickle que ens permet limitar l'ampla de banda de pujada i així no consumir-lo totalment durant el procés de còpia.

    Un parell de bones referències per coneixer millor trickle:

    http://www.tuxradar.com/content/control-your-bandwidth-trickle
    http://monkey.org/~marius/pages/?page=trickle