comments about rdiff-backup

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

comments about rdiff-backup

Faheem Mitha

Hello,

I've just started using rdiff-backup. I've had a few difficulties, but
nothing major. I thought I would make a few comments about these.

I've being trying to set up remote unattended backups using Dean Gaudet's
writeup, including use of LVM snapshots. This seems to work Ok for the
most part. However, I did some stress testing by having rdiff-backup do
backups of big changes at 5 min intervals.

This predictably means that (since the previous job has not completed) at
some point two different instances of rdiff-backup will be running and
trying to write to the same directory. Perhaps problems were to be
expected and in fact rdiff-backup produced errors and appeared to lose
data.

I managed to work around this by having the script check whether the LVM
snapshot were mounted first, since this would mean the previous backups
were still in progress. However, I wonder whether it would not be possible
for rdiff-backup to detect that another rdiff-backup process was writing
to the same directory and exit gracefully? Perhaps this would be difficult
to do in a portable way.

Secondly, I noticed that the --remove-older-than option removes files in
the increments directory, but does not touch any of the log and statistics
files that are generated, eg session_statistics, error_log,
mirror_metadata etc.

Would it not be reasonable for this command to remove all these files for
the relevant dates as well? If not, what is the recommended way to remove
these files?

Thirdly, when using --remove-older-than, if there are no increments that
need to be removed, I see a message, apparently on standard error, saying
something like

"No increments older than Tue Nov 22 11:46:48 2005 found, exiting."

Isn't this somewhat unnecessary? This seems hardly to merit a message at
all, let alone one on standard error.

Thanks for reading. I look forward to hearing back. Please cc me on any
reply. Thanks.

                                                                  Faheem.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: comments about rdiff-backup

Keith Edmunds
Faheem Mitha wrote:
> This predictably means that (since the previous job has not completed) at
> some point two different instances of rdiff-backup will be running and
> trying to write to the same directory
.
.
> I wonder whether it would not be possible
> for rdiff-backup to detect that another rdiff-backup process was writing
> to the same directory and exit gracefully?

I see this as down to the system administrator to manage, and much
easier too. For example, suppose I have two backup jobs, one backing up
user xxx and a second backing up user yyy. Those two backup jobs might
writing, respectively, to:

        /backups/home/xxx/...
        /backups/home/yyy...

That won't cause a problem at all. Now suppose on your system you backup
the whole of /home. A second invocation is started, so at one point
there are two rdiff-backup processes, and they are writing to the
directories shown above. How should rdiff-backup decide that the former
pair of processes are safe to run, but the latter pair are not? Yes, it
can be done by parsing command lines of other processes - but even that
may not be possible if each of user xxx and yyy are running the backup
process under their own UIDs.

What I do:

        #!/bin/bash
        # Start of script
        ACTIVE_FLAG=/var/run/backup_in_progress
        [ -e $ACTIVE_FLAG] && exit 1 || touch $ACTIVE_FLAG
        .
        .
        # End of script
        rm $ACTIVE_FLAG

Actually it's a little more sophisticated than that, but you get the idea.

Keith

--
Keith Edmunds

+---------------------------------------------------------------------+
|  Tiger Computing Ltd  |  Helping businesses make the most of Linux  |
|  "The Linux Company"  |    http://www.TheLinuxConsultancy.co.uk     |
+---------------------------------------------------------------------+


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: comments about rdiff-backup

Faheem Mitha


On Tue, 22 Nov 2005, Keith Edmunds wrote:

> I see this as down to the system administrator to manage, and much easier
> too. For example, suppose I have two backup jobs, one backing up user xxx and
> a second backing up user yyy. Those two backup jobs might writing,
> respectively, to:
>
> /backups/home/xxx/...
> /backups/home/yyy...
>
> That won't cause a problem at all. Now suppose on your system you backup the
> whole of /home. A second invocation is started, so at one point there are two
> rdiff-backup processes, and they are writing to the directories shown above.
> How should rdiff-backup decide that the former pair of processes are safe to
> run, but the latter pair are not? Yes, it can be done by parsing command
> lines of other processes - but even that may not be possible if each of user
> xxx and yyy are running the backup process under their own UIDs.

Actually, one could use some kind of locking mechanism, since the
directory being written to by rdiff-backup is well-defined by the
existence of the rdiff-backup-data directory at the top level.

This is hardly an ideal solution though, since one problem with locking is
guaranteeing that it be released correctly.

> What I do:
>
> #!/bin/bash
> # Start of script
> ACTIVE_FLAG=/var/run/backup_in_progress
> [ -e $ACTIVE_FLAG] && exit 1 || touch $ACTIVE_FLAG
> .
> .
> # End of script
> rm $ACTIVE_FLAG
>
> Actually it's a little more sophisticated than that, but you get the idea.

Yes, fair enough. I'm doing something similar in concept, but using mounts
of LVM volumes instead. The trouble with what you are doing it that if the
script exits unexpectedly, the ACTIVE_FLAG file might still remain, no?

I was just wondering if this issue had been considered, and what the
concensus thinking about it was.

Thanks for your comments.
                                                                    Faheem.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: comments about rdiff-backup

Ben Escoto
In reply to this post by Faheem Mitha
>>>>> Faheem Mitha <[hidden email]>
>>>>> wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)
>
> I managed to work around this by having the script check whether the
> LVM snapshot were mounted first, since this would mean the previous
> backups were still in progress. However, I wonder whether it would
> not be possible for rdiff-backup to detect that another rdiff-backup
> process was writing to the same directory and exit gracefully?
> Perhaps this would be difficult to do in a portable way.

The development version writes its PID to the current_mirror file in
the backup repository.  Some future version should check that PID and
abort if the process is still running (unless you run --force or
something)

> Secondly, I noticed that the --remove-older-than option removes files in
> the increments directory, but does not touch any of the log and statistics
> files that are generated, eg session_statistics, error_log,
> mirror_metadata etc.

Are you sure it doesn't remove those?  It seems to from a few quick
tests I did.  In fact, I just found a bug, if you pick a time later
than the current mirror time it will delete those too :-(

> Thirdly, when using --remove-older-than, if there are no increments that
> need to be removed, I see a message, apparently on standard error, saying
> something like
>
> "No increments older than Tue Nov 22 11:46:48 2005 found, exiting."
>
> Isn't this somewhat unnecessary? This seems hardly to merit a message at
> all, let alone one on standard error.

Ok, I'll remove that message.


--
Ben Escoto

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: comments about rdiff-backup

Blair Zajac
Ben Escoto wrote:

>>>>>>Faheem Mitha <[hidden email]>
>>>>>>wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)
>>Thirdly, when using --remove-older-than, if there are no increments that
>>need to be removed, I see a message, apparently on standard error, saying
>>something like
>>
>>"No increments older than Tue Nov 22 11:46:48 2005 found, exiting."
>>
>>Isn't this somewhat unnecessary? This seems hardly to merit a message at
>>all, let alone one on standard error.
>
>
> Ok, I'll remove that message.

Hold on!  I like that message.  In my nightly crontab, I delete any backups
older than 60 days, so I would like to see if there were any deleted.

Regards,
Blair

--
Blair Zajac, Ph.D.
<[hidden email]>
Subversion and Orca training and consulting
http://www.orcaware.com/svn/


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: comments about rdiff-backup

Faheem Mitha
In reply to this post by Ben Escoto


On Tue, 22 Nov 2005, Ben Escoto wrote:

>>>>>> Faheem Mitha <[hidden email]>
>>>>>> wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)
>>
>> I managed to work around this by having the script check whether the
>> LVM snapshot were mounted first, since this would mean the previous
>> backups were still in progress. However, I wonder whether it would
>> not be possible for rdiff-backup to detect that another rdiff-backup
>> process was writing to the same directory and exit gracefully?
>> Perhaps this would be difficult to do in a portable way.
>
> The development version writes its PID to the current_mirror file in
> the backup repository.  Some future version should check that PID and
> abort if the process is still running (unless you run --force or
> something)

Yes, that sounds like a good idea.

>> Secondly, I noticed that the --remove-older-than option removes files in
>> the increments directory, but does not touch any of the log and statistics
>> files that are generated, eg session_statistics, error_log,
>> mirror_metadata etc.
>
> Are you sure it doesn't remove those?  It seems to from a few quick
> tests I did.  In fact, I just found a bug, if you pick a time later
> than the current mirror time it will delete those too :-(

Pretty sure. I've been doing extensive testing. However, I'm running the
current Debian version in Sarge, namely 1.0.1. Is it possible this is
something that has been fixed recently?

In any case, if you can point me to the code that removes the
log/statistics files in the code (I looked in 1.0.1 but could not find
anything) then that would be very helpful.

Hmm, now looking in 1.1.2, but can't see anything there either.

Thanks.                                                        Faheem.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: comments about rdiff-backup

Faheem Mitha
In reply to this post by Blair Zajac


On Tue, 22 Nov 2005, Blair Zajac wrote:

> Ben Escoto wrote:
>>>>>>> Faheem Mitha <[hidden email]>
>>>>>>> wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)
>>> Thirdly, when using --remove-older-than, if there are no increments that
>>> need to be removed, I see a message, apparently on standard error, saying
>>> something like
>>>
>>> "No increments older than Tue Nov 22 11:46:48 2005 found, exiting."
>>>
>>> Isn't this somewhat unnecessary? This seems hardly to merit a message at
>>> all, let alone one on standard error.
>>
>>
>> Ok, I'll remove that message.
>
> Hold on!  I like that message.  In my nightly crontab, I delete any backups
> older than 60 days, so I would like to see if there were any deleted.

Ok, but couldn't this go to standard output rather than standard error
instead? It is not an error message.

                                                                 Faheem.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: comments about rdiff-backup

Blair Zajac
Faheem Mitha wrote:

>
>
> On Tue, 22 Nov 2005, Blair Zajac wrote:
>
>> Ben Escoto wrote:
>>
>>>>>>>> Faheem Mitha <[hidden email]>
>>>>>>>> wrote the following on Tue, 22 Nov 2005 13:19:56 -0500 (EST)
>>>>
>>>> Thirdly, when using --remove-older-than, if there are no increments
>>>> that need to be removed, I see a message, apparently on standard
>>>> error, saying something like
>>>>
>>>> "No increments older than Tue Nov 22 11:46:48 2005 found, exiting."
>>>>
>>>> Isn't this somewhat unnecessary? This seems hardly to merit a
>>>> message at all, let alone one on standard error.
>>>
>>>
>>>
>>> Ok, I'll remove that message.
>>
>>
>> Hold on!  I like that message.  In my nightly crontab, I delete any
>> backups older than 60 days, so I would like to see if there were any
>> deleted.
>
>
> Ok, but couldn't this go to standard output rather than standard error
> instead? It is not an error message.

Fine with me.

Regards,
Blair


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: comments about rdiff-backup

Faheem Mitha
In reply to this post by Faheem Mitha


On Tue, 22 Nov 2005, Faheem Mitha wrote:

>>> Secondly, I noticed that the --remove-older-than option removes files in
>>> the increments directory, but does not touch any of the log and statistics
>>> files that are generated, eg session_statistics, error_log,
>>> mirror_metadata etc.
>>
>> Are you sure it doesn't remove those?  It seems to from a few quick
>> tests I did.  In fact, I just found a bug, if you pick a time later
>> than the current mirror time it will delete those too :-(
>
> Pretty sure. I've been doing extensive testing. However, I'm running the
> current Debian version in Sarge, namely 1.0.1. Is it possible this is
> something that has been fixed recently?
>
> In any case, if you can point me to the code that removes the log/statistics
> files in the code (I looked in 1.0.1 but could not find anything) then that
> would be very helpful.
>
> Hmm, now looking in 1.1.2, but can't see anything there either.

Ok, further info.

I have two test backups running every 5 minutes. I have it set to clear
out everything more than an hr old.

I enabled the one hr removal after the backups had been running for a
while, and it does not clear the log and statistics files in that case.

I am not sure whether it cleared the increments.

However, I deleted one on the backups and restarted it with the 1 hr
removal enabled, and now it is deleting the log/statistics files
correctly.

It looks to me like there is a bug somewhere.

                                                                Faheem.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki