I have been a very happy user of rdiff-backup for about a year now.
Recently though I noticed that certain directories are not in my backup. I did a 'diff -rq' on some of the source/backup directories and (although I haven't checked extensively) all the missing ones seem to be empty directory trees. I am currently running rdiff-backup 1.0.1-r1 on a Gentoo Linux box (2.6.12-gentoo-r10 on AMD64). The source data itself was restored after a disk crash, some time towards the end of August, so I can only assume this problem emerged some time after that (since I know that some of the empty directory trees are more than a year old and otherwise I don't see how they would have been restored in the first place). I tried running a small backup of just a single directory (one that contained some of the missing directories) and the odd thing is that worked as expected, i.e. it backed up everything including the empty sub-directory trees. Could this have something to do with volumes of data? Looking at my Gentoo package installation log I see the history is as follows: rdiff-backup-0.13.6 -> installed 8 September 2005 rdiff-backup-1.0.1-r1 -> installed 22 September 2005 So I'm guessing this is a problem that came with upgrading to 1.0.1. Perhaps it would be better for me to back out to 0.13.6? (Unfortunately the most recent versions are not yet available on 64-bit Gentoo.) I tried searching Google & the rdiff-backup-user archives but didn't come up with anything. If it would help, I can post my rdiff-backup command or any other relevant information. Thanks Toni _______________________________________________ rdiff-backup-users mailing list at [hidden email] http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki |
>>>>> Toni Price <[hidden email]>
>>>>> wrote the following on Sun, 01 Jan 2006 09:02:02 +0000 > I have been a very happy user of rdiff-backup for about a year now. > Recently though I noticed that certain directories are not in my backup. > I did a 'diff -rq' on some of the source/backup directories and > (although I haven't checked extensively) all the missing ones seem to be > empty directory trees. All I can say is I don't see why rdiff-backup wouldn't back up empty directories. I've never personally noticed this problem. I'm not sure what to do unless you can isolate or reproduce the problem. Are you sure its a bug? Maybe it's just some unexpected --include/--exclude interaction, or a permission problem? > So I'm guessing this is a problem that came with upgrading to > 1.0.1. Perhaps it would be better for me to back out to 0.13.6? > (Unfortunately the most recent versions are not yet available on > 64-bit Gentoo.) I tried searching Google & the rdiff-backup-user > archives but didn't come up with anything. If it would help, I can > post my rdiff-backup command or any other relevant information. 0.13.6 was the last development release before the stable 1.0.x series, so 1.0.1 is just 0.13.6 with bugfixes. So it probably won't help if you downgrade. -- Ben Escoto _______________________________________________ rdiff-backup-users mailing list at [hidden email] http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki |
Ben Escoto wrote:
>>>>>>Toni Price <[hidden email]> >>>>>>wrote the following on Sun, 01 Jan 2006 09:02:02 +0000 >>>>>> >>>>>> >>I have been a very happy user of rdiff-backup for about a year now. >>Recently though I noticed that certain directories are not in my backup. >>I did a 'diff -rq' on some of the source/backup directories and >>(although I haven't checked extensively) all the missing ones seem to be >>empty directory trees. >> >> > >All I can say is I don't see why rdiff-backup wouldn't back up empty >directories. I've never personally noticed this problem. I'm not >sure what to do unless you can isolate or reproduce the problem. > >Are you sure its a bug? Maybe it's just some unexpected >--include/--exclude interaction, or a permission problem? > > No - I'm not at all sure it's a bug! Though it seems very unlikely to me that it could be a permissions problem. The permissions on some of the missing directories look like this: (...) drwx------ 2 tprice users 4096 Jan 9 2005 <...> drwxr-xr-x 3 tprice users 4096 Jan 9 2005 <...> drwxrwxrwx 2 tprice users 4096 May 28 2004 <...> drwx------ 5 tprice users 4096 Jan 27 2005 <...> drwxr-xr-x 2 tprice users 4096 Jan 30 2005 <...> drwxr-xr-x 2 tprice users 4096 Jan 30 2005 <...> (...) As far as includes and excludes are concerned, I wouldn't have guessed they were causing this but I'll try and do some testing to see if I can pinpoint anything. > > >>So I'm guessing this is a problem that came with upgrading to >>1.0.1. Perhaps it would be better for me to back out to 0.13.6? >>(Unfortunately the most recent versions are not yet available on >>64-bit Gentoo.) I tried searching Google & the rdiff-backup-user >>archives but didn't come up with anything. If it would help, I can >>post my rdiff-backup command or any other relevant information. >> >> > >0.13.6 was the last development release before the stable 1.0.x >series, so 1.0.1 is just 0.13.6 with bugfixes. So it probably won't >help if you downgrade. > > > > _______________________________________________ rdiff-backup-users mailing list at [hidden email] http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki |
Ben Escoto wrote:
> >> >> Are you sure its a bug? Maybe it's just some unexpected >> --include/--exclude interaction, or a permission problem? >> >> Ok, I have managed to repeat the problem on some dummy data I've created. I've attached a copy of the data with a bash script to illustrate what's happening ... this data (sort of) mimics my own data structure, which is why I've done it like this. (If you untar it and run test-rdiff-backup.sh from the directory where it's been untarred, it should copy some data to /tmp/rdiff-data and run two backups to /tmp/rdiff-test/bak). It's entirely possible that I simply don't understand properly how the --includes and --excludes work, but this behaviour seems odd to me. My data structure looks as follows: rdiff-data/ |-- docs | |-- technotes | | |-- anotherdir | | | |-- empty_tree_2 | | | | |-- dir1 | | | | | `-- structure | | | | | |-- sub1 | | | | | `-- sub2 | | | | | `-- subsub | | | | `-- dir2 | | | | |-- lower1 | | | | `-- lower2 | | | `-- somefile.txt | | |-- empty_tree_1 | | | `-- sub1 | | | |-- sibling1 | | | | `-- nextdir | | | `-- sibling2 | | `-- misc | | |-- dir1 | | | `-- fox-and-dogs.txt | | `-- dir2 | | `-- sub1 | | `-- stuff.txt | |-- tmp | | |-- tempdir1 | | |-- tempfile1.txt | | `-- tempfile2.txt | `-- various | |-- another | | |-- empty_tree_3 | | | `-- testdir1 | | | `-- testdir2 | | | |-- testsub1 | | | `-- testsub2 | | `-- moreStuff.txt | |-- file1.txt | `-- file2.txt |-- mediafiles | `-- audio | |-- Classical | | |-- audio1 | | `-- audio2 | |-- Jazz | | `-- audio | `-- Playlists | |-- playlist1.txt | `-- playlist2.txt |-- ref | `-- ref.txt `-- software `-- sofware-stuff.txt 37 directories, 15 files Within this there are three empty directory trees, rdiff-data/docs/technotes/empty_tree_1/ rdiff-data/docs/technotes/anotherdir/empty_tree_2/ rdiff-data/docs/various/another/empty_tree_3 Basically, if I run the following command: /usr/bin/rdiff-backup \ --exclude /tmp/rdiff-data/docs/tmp/'*' \ --include /tmp/rdiff-data/mediafiles/audio/Playlists \ --exclude /tmp/rdiff-data/mediafiles/audio/'*' \ --exclude /tmp/rdiff-data/ref/'*' \ --exclude /tmp/rdiff-data/software/'*' \ /tmp/rdiff-data /tmp/rdiff-test/bak/data1 it works as expected, whilst if I change it as follows (the change I've made here is from /tmp/rdiff-data to '**' in the --include) /usr/bin/rdiff-backup \ --exclude /tmp/rdiff-data/docs/tmp/'*' \ --include '**'/mediafiles/audio/Playlists \ --exclude /tmp/rdiff-data/mediafiles/audio/'*' \ --exclude /tmp/rdiff-data/ref/'*' \ --exclude /tmp/rdiff-data/software/'*' \ /tmp/rdiff-data /tmp/rdiff-test/bak/data2 then it all goes a bit weird, and excludes empty_tree_1, empty_tree_2 & empty_tree_3 from the backup (as well as excluding tmp, ref and software altogether, which is not my intention). Here's the output I get when running test-rdiff-backup.sh. (Hopefully it will produce the same results when run in a different environment): ===== START output ================================================= Files only in /tmp/rdiff-data after [correct] backup: Only in /tmp/rdiff-data/docs/tmp: tempdir1 Only in /tmp/rdiff-data/docs/tmp: tempfile1.txt Only in /tmp/rdiff-data/docs/tmp: tempfile2.txt Only in /tmp/rdiff-data/mediafiles/audio: Classical Only in /tmp/rdiff-data/mediafiles/audio: Jazz Only in /tmp/rdiff-data/ref: ref.txt Only in /tmp/rdiff-data/software: sofware-stuff.txt Files only in /tmp/rdiff-test/bak/data1 after [correct] backup: Only in /tmp/rdiff-test/bak/data1: rdiff-backup-data ------------------------------------------------ Files only in /tmp/rdiff-data after [incorrect] backup: Only in /tmp/rdiff-data/docs/technotes/anotherdir: empty_tree_2 Only in /tmp/rdiff-data/docs/technotes: empty_tree_1 Only in /tmp/rdiff-data/docs: tmp Only in /tmp/rdiff-data/docs/various/another: empty_tree_3 Only in /tmp/rdiff-data/mediafiles/audio: Classical Only in /tmp/rdiff-data/mediafiles/audio: Jazz Only in /tmp/rdiff-data: ref Only in /tmp/rdiff-data: software Files only in /tmp/rdiff-test/bak/data2 after [incorrect] backup: Only in /tmp/rdiff-test/bak/data2: rdiff-backup-data ===== END output ================================================== What I don't understand is, why would the following include: --include '**'/mediafiles/audio/Playlists cause rdiff-backup to exclude something like "/tmp/rdiff-data/docs/technotes/anotherdir/empty_tree_2"? Am I simply missing something about how --include should work? Thanks ... Toni _______________________________________________ rdiff-backup-users mailing list at [hidden email] http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki |
>>>>> Toni Price <[hidden email]>
>>>>> wrote the following on Tue, 03 Jan 2006 21:00:06 +0000 > > Ok, I have managed to repeat the problem on some dummy data I've created. ... > It's entirely possible that I simply don't understand properly how the > --includes and --excludes work, but this behaviour seems odd to me. You were absolutely right, under some conditions when you use an include like "--include **XXX", empty directories may be skipped, and this bug may have first appeared in version 1.0.0 (which fixed another selection bug). I think this patch will fix the problem, but you may want to test it on your directory structure: http://cvs.savannah.nongnu.org/viewcvs/rdiff-backup/rdiff_backup/selection.py?root=rdiff-backup&r1=1.41&r2=1.42&makepatch=1&diff_format=u -- Ben Escoto _______________________________________________ rdiff-backup-users mailing list at [hidden email] http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki |
Ben Escoto wrote:
>>>>>>Toni Price <[hidden email]> >>>>>>wrote the following on Tue, 03 Jan 2006 21:00:06 +0000 >>>>>> >>>>>> >>Ok, I have managed to repeat the problem on some dummy data I've created. >> >> >... > > >>It's entirely possible that I simply don't understand properly how the >>--includes and --excludes work, but this behaviour seems odd to me. >> >> > >You were absolutely right, under some conditions when you use an >include like "--include **XXX", empty directories may be skipped, and >this bug may have first appeared in version 1.0.0 (which fixed another >selection bug). > >I think this patch will fix the problem, but you may want to test it >on your directory structure: > >http://cvs.savannah.nongnu.org/viewcvs/rdiff-backup/rdiff_backup/selection.py?root=rdiff-backup&r1=1.41&r2=1.42&makepatch=1&diff_format=u > > > > Well, it's taken me an inordinate amount of time to get back on this one, but just a heads-up really to say this seems to have fixed the problem on my backup (tested by upgrading to version 1.0.4, not by applying the patch myself) -- thanks! Toni _______________________________________________ rdiff-backup-users mailing list at [hidden email] http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki |
Free forum by Nabble | Edit this page |