initial rdiff-backup for large repository over Internet (and connection lost)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

initial rdiff-backup for large repository over Internet (and connection lost)

me-96
Hi all,

I'm an happy rdiff-backup user for many years.

Years ago I setup a rdiff-backup from a WWW server to a local server.
Internet connection is limited to 2Mbps between the 2 servers.
As /var/www was initially empty, rdiff-backup as done its backup job
everyday without problem, with many sites added in /var/www.

Now /var/www is 14G large and I need to fresh start the backup
(destination directory has been deleted).

Bad thing is that trying to first sync source and destination via the
command :
nice -n 19 rdiff-backup --force $SOURCE $DEST::$DESTDIR
always ends with network (Internet error) like these :
Found interrupted initial backup. Removing...
Write failed: Broken pipe
Fatal Error: Lost connection to the remote system

What would be proper way to first sync rdiff-backup ?
Would a previous rsync help, so that only meta-datas would be written by
rdiff ?

Thanks in advance for your thought.
Nicolas


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: initial rdiff-backup for large repository over Internet (and connection lost)

Adrian Klaver-2
On 02/24/2016 01:57 PM, Nicolas wrote:

> Hi all,
>
> I'm an happy rdiff-backup user for many years.
>
> Years ago I setup a rdiff-backup from a WWW server to a local server.
> Internet connection is limited to 2Mbps between the 2 servers.
> As /var/www was initially empty, rdiff-backup as done its backup job
> everyday without problem, with many sites added in /var/www.
>
> Now /var/www is 14G large and I need to fresh start the backup
> (destination directory has been deleted).
>
> Bad thing is that trying to first sync source and destination via the
> command :
> nice -n 19 rdiff-backup --force $SOURCE $DEST::$DESTDIR
> always ends with network (Internet error) like these :
> Found interrupted initial backup. Removing...
> Write failed: Broken pipe
> Fatal Error: Lost connection to the remote system
>
> What would be proper way to first sync rdiff-backup ?
> Would a previous rsync help, so that only meta-datas would be written by
> rdiff ?

Take a look here:

http://www.nongnu.org/rdiff-backup/rdiff-backup.1.html

FILE SELECTION

I would use the --include-filelist option and associated file to have
rdiff back up /var/www a subdirectory at a time. In other words add a
subdirectory to the file and run rdiff. After that completes, add
another subdirectory and rerun, etc. Basically build up the destination
directory incrementally.

>
> Thanks in advance for your thought.
> Nicolas
>
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


--
Adrian Klaver
[hidden email]

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: initial rdiff-backup for large repository over Internet (and connection lost)

Adrian Klaver-2
In reply to this post by me-96
On 02/24/2016 01:57 PM, Nicolas wrote:

> Hi all,
>
> I'm an happy rdiff-backup user for many years.
>
> Years ago I setup a rdiff-backup from a WWW server to a local server.
> Internet connection is limited to 2Mbps between the 2 servers.
> As /var/www was initially empty, rdiff-backup as done its backup job
> everyday without problem, with many sites added in /var/www.
>
> Now /var/www is 14G large and I need to fresh start the backup
> (destination directory has been deleted).
>
> Bad thing is that trying to first sync source and destination via the
> command :
> nice -n 19 rdiff-backup --force $SOURCE $DEST::$DESTDIR
> always ends with network (Internet error) like these :
> Found interrupted initial backup. Removing...
> Write failed: Broken pipe
> Fatal Error: Lost connection to the remote system
>
> What would be proper way to first sync rdiff-backup ?
> Would a previous rsync help, so that only meta-datas would be written by
> rdiff ?

I do not use rdiff much any more, as I switched over to
borg-backup(http://borgbackup.readthedocs.org/en/stable/), so I am a
little rusty on my advice. In addition to my previous post, I now
remember something else I used to do. That is rdiff from the source to a
local destination, then rsync the local destination to the remote
destination, finally switch the backup to use the remote instead of the
local destination. rsync is more forgiving of network interruptions.

>
> Thanks in advance for your thought.
> Nicolas
>
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL:
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


--
Adrian Klaver
[hidden email]

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: initial rdiff-backup for large repository over Internet (and connection lost)

Dominic Raferd-3


On 25 February 2016 at 01:19, Adrian Klaver <[hidden email]> wrote:

On 02/24/2016 01:57 PM, Nicolas wrote:
Hi all,

I'm an happy rdiff-backup user for many years.

Years ago I setup a rdiff-backup from a WWW server to a local server.
Internet connection is limited to 2Mbps between the 2 servers.
As /var/www was initially empty, rdiff-backup as done its backup job
everyday without problem, with many sites added in /var/www.

Now /var/www is 14G large and I need to fresh start the backup
(destination directory has been deleted).

Bad thing is that trying to first sync source and destination via the
command :
nice -n 19 rdiff-backup --force $SOURCE $DEST::$DESTDIR
always ends with network (Internet error) like these :
Found interrupted initial backup. Removing...
Write failed: Broken pipe
Fatal Error: Lost connection to the remote system

What would be proper way to first sync rdiff-backup ?
Would a previous rsync help, so that only meta-datas would be written by
rdiff ?

I do not use rdiff much any more, as I switched over to borg-backup(http://borgbackup.readthedocs.org/en/stable/), so I am a little rusty on my advice. In addition to my previous post, I now remember something else I used to do. That is rdiff from the source to a local destination, then rsync the local destination to the remote destination, finally switch the backup to use the remote instead of the local destination. rsync is more forgiving of network interruptions.


I think that is good advice, it is the strategy I follow with my TimeDicer - a local 'primary' rdiff-backup server and a remote 'mirror' server that is synchronized by a script on the primary which uses rsync.

Alternatively it is possible to tweak the ssh settings to help rdiff-backup maintain a session over the internet, still I don't think it is such a robust solution.



_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: initial rdiff-backup for large repository over Internet (and connection lost)

David Precious-2
In reply to this post by me-96
Hi Nicolas,

On Thu, 25 Feb 2016 08:57:20 +1100
Nicolas <[hidden email]> wrote:
> Now /var/www is 14G large and I need to fresh start the backup
> (destination directory has been deleted).
[...]
> What would be proper way to first sync rdiff-backup ?
> Would a previous rsync help, so that only meta-datas would be written
> by rdiff ?

If I remember rightly, what I did when I set up my large backup was use
rdiff-backup to back up to a local rdiff-backup target on a spare drive
(so all data and metadata was there), then sneakernet that to the box
in a datacentre that I wanted to back up to, then future backups over
the Internet to that target worked fine.

For 14G, I'd say you could just do the backup locally on the source
server, assuming you have enough disk space, rsync the whole thing to
the remote box (being able to continue if it's interrupted), then point
rdiff-backup at it afterwards - I'd expect that to work fine.

_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: initial rdiff-backup for large repository over Internet (and connection lost)

Alvin Starr
This is actually an excellent idea.
I do some support work for a backup company and they regularly do
something like this.
They have a courier deliver a USB drive to the customer and an "initial
backup" copy is put on the drive and this drive is sent back to the
company where someone plugs it in and copies the data.

Rdiff onto a USB drive would be a good way to back the system up and 14G
of data would fit nicely onto a USB flash drive.


On 02/25/2016 05:34 AM, David Precious wrote:

> Hi Nicolas,
>
> On Thu, 25 Feb 2016 08:57:20 +1100
> Nicolas <[hidden email]> wrote:
>> Now /var/www is 14G large and I need to fresh start the backup
>> (destination directory has been deleted).
> [...]
>> What would be proper way to first sync rdiff-backup ?
>> Would a previous rsync help, so that only meta-datas would be written
>> by rdiff ?
> If I remember rightly, what I did when I set up my large backup was use
> rdiff-backup to back up to a local rdiff-backup target on a spare drive
> (so all data and metadata was there), then sneakernet that to the box
> in a datacentre that I wanted to back up to, then future backups over
> the Internet to that target worked fine.
>
> For 14G, I'd say you could just do the backup locally on the source
> server, assuming you have enough disk space, rsync the whole thing to
> the remote box (being able to continue if it's interrupted), then point
> rdiff-backup at it afterwards - I'd expect that to work fine.
>
> _______________________________________________
> rdiff-backup-users mailing list at [hidden email]
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki


--
Alvin Starr                   ||   voice: (905)513-7688
Netvel Inc.                   ||   Cell:  (416)806-0133
[hidden email]              ||


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki