rdiff-backup millions of small files and memory usage

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

rdiff-backup millions of small files and memory usage

Grégory Pakosz

I'm trying using rdiff-backup to backup an XFS partition which contains millions of small 2K-4K files.
Exactly 63 766 856 files.

I'm using rdiff-backup 1.2.8 on Debian Wheezy with the following packages:
ii  librsync1:amd64                    0.9.7-9
ii  rdiff-backup                       1.2.8-7

I'm using the following options:
--create-full-path --no-hard-links --exclude-other-filesystems --no-file-statistics --verbosity 4

rdiff-backup starts backing up the partition and I can see the memory usage slowly growing. E.g. after running for 1h30 it takes 4GB of RAM according to htop. Ultimately it gets killed after having exhausted the 16GB of available RAM.

Can someone please explain me what data rdiff-backup keeps in memory while backing up?
Any idea on how I can prevent the memory usage to balloon?


rdiff-backup-users mailing list at [hidden email]
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki