Safe to remove old increments while a backup runs?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Safe to remove old increments while a backup runs?

Sam's Lists
I was removing some old increments on my backup server using:
 
rdiff-backup --remove-older-than 3M --force root

While this was running a cronjob on a remote computer started backing up to my backup server (the same directory).

This is perfectly safe right?

Thanks!


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Safe to remove old increments while a backup runs?

Thomas Harold
On 3/26/2014 4:59 AM, Sam's Lists wrote:
> I was removing some old increments on my backup server using:
>  
> rdiff-backup --remove-older-than 3M --force root
>
> While this was running a cronjob on a remote computer started backing up
> to my backup server (the same directory).
>
> This is perfectly safe right?

Well, hopefully the remote computer would abort saying that the
destination rdiff-backup directory is in use... it's generally not safe
to have two rdiff-backup commands hitting the folder at the same time.
And I'm pretty sure that rdiff-backup checks for that.

In our backup scripts, we always do everything from the client end in
the backup script.  So the origin is in control of everything.  The
sequence of commands that the origin PC runs are:

- Run the rdiff-backup
- Run a verify on the last N increments
- Age out older increments

We also take a bit of care when rsync'ing the backups to the offsite
location.  We check whether there has been any activity in the
rdiff-backup directory in the last N minutes (usually 10) before we
snapshot the filesystem and send the rdiff-backup offsite.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Safe to remove old increments while a backup runs?

Dominic Raferd-3

On 26/03/2014 09:41, Thomas Harold wrote:
On 3/26/2014 4:59 AM, Sam's Lists wrote:
I was removing some old increments on my backup server using:
 
rdiff-backup --remove-older-than 3M --force root

While this was running a cronjob on a remote computer started backing up
to my backup server (the same directory).

This is perfectly safe right?
Well, hopefully the remote computer would abort saying that the
destination rdiff-backup directory is in use... it's generally not safe
to have two rdiff-backup commands hitting the folder at the same time.
And I'm pretty sure that rdiff-backup checks for that.

In our backup scripts, we always do everything from the client end in
the backup script.  So the origin is in control of everything.  The
sequence of commands that the origin PC runs are:

- Run the rdiff-backup
- Run a verify on the last N increments
- Age out older increments

We also take a bit of care when rsync'ing the backups to the offsite
location.  We check whether there has been any activity in the
rdiff-backup directory in the last N minutes (usually 10) before we
snapshot the filesystem and send the rdiff-backup offsite.

Similar to your last tip, my offsite backup script aborts before taking the snapshot unless it gets null output from command:

ps h -C rdiff-backup|grep -E -v " (--restore|-r |--list|-l |--compare)"

The idea is to check that there are no ongoing rdiff-backup activities except for certain 'safe' ones such as restore, list, compare.

Dominic
--
TimeDicer: Free File Recovery from Whenever

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Safe to remove old increments while a backup runs?

Sam's Lists
In reply to this post by Thomas Harold



On Wed, Mar 26, 2014 at 2:41 AM, Thomas Harold <[hidden email]> wrote:
On 3/26/2014 4:59 AM, Sam's Lists wrote:
> I was removing some old increments on my backup server using:
>
> rdiff-backup --remove-older-than 3M --force root
>
> While this was running a cronjob on a remote computer started backing up
> to my backup server (the same directory).
>
> This is perfectly safe right?

Well, hopefully the remote computer would abort saying that the
destination rdiff-backup directory is in use... it's generally not safe
to have two rdiff-backup commands hitting the folder at the same time.
And I'm pretty sure that rdiff-backup checks for that.

Unfortunately, it doesn't seem to test for that. 

It seems like this might be safe behavior though. I mean just thinking about the way the older backups must be stored. I'm not sure why it would matter if deletes were being made when the backup happened. 

Anyway, what should I do now? Is there an easy way to tell if my backup was corrupted?

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Safe to remove old increments while a backup runs?

Thomas Harold
On 3/26/2014 7:04 PM, Sam's Lists wrote:
>
> Anyway, what should I do now? Is there an easy way to tell if my backup
> was corrupted?

You can use the rdiff-backup "--verify" option to test your latest
backup.  To test multiple increments, you have to do a bit of scripting.
 Or feed the dates one at a time into --verify-at arguments.

...

Here is a working fragment from our bash-based backup script.  It is
designed to verify the last 5 increments plus the 'now' increment after
the backup is run.  All of the $RDIFFBACKUP, $SED, $SORT, $GREP
variables are pointers to the various executables.  You will need to
customize this code to match your environment, but it shows how to parse
--list-increments and feed it back into a --verify-at command.

$OFFHOST is the FQDN of our backup server (or can be left blank to
backup to a local file system).  $OFFBASE is the full path to the backup
area, with a trailing '/' on it.  And $DIR is the directory within the
backup area being verified.

--------------------------------------

if [[ ! $VALIDATELASTN =~ ^[0-9]+$ ]]; then VALIDATELASTN=5; fi

RDIFFINCREMENTS=$(
    $RDIFFBACKUP --list-increments ${OFFHOST}::${OFFBASE}${DIR} | \
    $GREP -o 'increments\..*\.dir' | \
    $SED 's/^increments\.//g' | $SED 's/\.dir$//g' | \
    $SORT | $TAIL -n $VALIDATELASTN
    )

RDIFFINCREMENTS+=' now'

echo "$RDIFFBACKUP verify last $VALIDATELASTN increments"
for INCREMENTTIME in ${RDIFFINCREMENTS}; do

    echo "Verify backup at: ${INCREMENTTIME}"

    $RDIFFBACKUP --verify-at "${INCREMENTTIME}" ${OFFHOST}::${OFFBASE}${DIR}

    status=$?
    if [ $status -ne 0 ]; then
        echo "" >> $TMPFILE
        echo "$RDIFFBACKUP verify failed with status: $status"
        echo "$RDIFFBACKUP --verify-at "${INCREMENTTIME}"
${OFFHOST}::${OFFBASE}${DIR} failed with status: $status" >> $TMPFILE
        let ERRCNT+=1
        continue
    fi

done

--------------------------------------

Notes:

Running --verify-at requires a lot of disk activity, especially as you
get deeper and deeper into the past.  This goes much faster if your /tmp
(or wherever you tell rdiff-backup to use as a temporary directory) is
on a different spindle then your backups, or is located on a SSD.

In order to verify a past increment, rdiff-backup has to reconstruct the
file.  The file system where your /tmp directory is located will need to
be large enough to hold the largest file being tested.  A safer rule of
thumb is a 2x or 3x multiplier.  I run into this issue when verifying
backups containing virtual machine images (which are ~30GB).  We were
blowing out the free space on our /tmp which was initially only a 16GB
file system.  We have since moved /tmp to it's own RAID array (RAID-1 on
a pair of Intel SSDs) and made it much larger.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki