Finding files with duplicate content on linux...

Translation of the original in Catalan. Read it in: CatalàCastellano

It's often hard to find tools or small scripts that do apparently such stupid things as the one mentioned. In fact for windows I've found some programs that look good although I haven't tried them yet. But for linux it took me a bit longer until I found fdupes (local). It's programmed in C and it's really very easy to use and quite powerful.

I've also found a small bash script that does a rather rudimentary search but is useful in some cases:

OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -print0 |
  xargs -0 -n1 md5sum |
    sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
    sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF;
chmod a+x $OUTF; ls -l $OUTF

The page where I found this script is Unix shell script for removing duplicate files.

1 comment

Comments from the old blog, as they were written. Comments are closed.

  1. —

    Jo utilitzo la comanda 'repeats' per buscar fitxers duplicats, està inclosa dins de littleutils, que encara no está a gentoo. Vaig fer l'ebuild fa temps però encara no l'han inclos al portage. Pots veure'l aquí:

    http://bugs.gentoo.org/show_bug.cgi?id=109072...