Remote encrypted backup with slow connection.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Remote encrypted backup with slow connection.

Piotr Karbowski
Hi there!

I want to store my backups on remote machine, with sshfs and dmcrypt
(sshfs, losetup, cryptsetup luksOpen, mount etc.) but I need reduce
traffic, if I will just run rdiff on mounted dmcrypted-volume under
sshfs, to check if files was changed and generate diff ti must read
whole file, read == download but it will kill me, I think about two
stage backup, backup files to local rdiff-backup dir and then sync
local rdiff-backup dir with remote server but how? If I will use for
example rsync it still need to check whole files for changes (read,
download it) and upload only new. I hope you will understand what I
need and help me.


--
Pozdrawiam.
Piotr Karbowski.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Remote encrypted backup with slow connection.

Piotr Karbowski
On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller <[hidden email]> wrote:
> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote:
>> local rdiff-backup dir with remote server but how? If I will use for
>> example rsync it still need to check whole files for changes (read,
>> download it) and upload only new. I hope you will understand what I
>> need and help me.
>
> rsync won't check whole files unless you give the -c flag. Otherwise, it
> just compares metadata. I don't know if that's also the case with
> rdiff-backup, but I assume so.So

So I need to know how rdiff default compares data, if by size and
mod-time, it will not be so painful but still it will download changed
files to generate diff.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Re: Remote encrypted backup with slow connection.

Dominic Raferd-2
Piotr Karbowski wrote:

> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller <[hidden email]> wrote:
>  
>> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote:
>>    
>>> local rdiff-backup dir with remote server but how? If I will use for
>>> example rsync it still need to check whole files for changes (read,
>>> download it) and upload only new. I hope you will understand what I
>>> need and help me.
>>>      
>> rsync won't check whole files unless you give the -c flag. Otherwise, it
>> just compares metadata. I don't know if that's also the case with
>> rdiff-backup, but I assume so.
>>    
>
> So I need to know how rdiff default compares data, if by size and
> mod-time, it will not be so painful but still it will download changed
> files to generate diff.
Rdiff-backup is designed to be ultra-efficient at this activity. It only
sends the changes in a file over the wire, not the whole file. To do
this it uses the librsync library which is effectively the same as
rsync. You can read more about the technique at
http://en.wikipedia.org/wiki/Rsync. rdiff-backup does not use file times
to determine whether to do backups. It can backup very large files with
small changes very quickly.

Dominic


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Re: Remote encrypted backup with slow connection.

Piotr Karbowski
On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd <[hidden email]> wrote:

> Piotr Karbowski wrote:
>>
>> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller <[hidden email]> wrote:
>>
>>>
>>> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote:
>>>
>>>>
>>>> local rdiff-backup dir with remote server but how? If I will use for
>>>> example rsync it still need to check whole files for changes (read,
>>>> download it) and upload only new. I hope you will understand what I
>>>> need and help me.
>>>>
>>>
>>> rsync won't check whole files unless you give the -c flag. Otherwise, it
>>> just compares metadata. I don't know if that's also the case with
>>> rdiff-backup, but I assume so.
>>>
>>
>> So I need to know how rdiff default compares data, if by size and
>> mod-time, it will not be so painful but still itefficient  will download changed
>> files to generate diff.
>
> Rdiff-backup is designed to be ultra-efficient at this activity. It only
> sends the changes in a file over the wire, not the whole file. To do this it
> uses the librsync library which is effectively the same as rsync. You can
> read more about the technique at http://en.wikipedia.org/wiki/Rsync.
> rdiff-backup does not use file times to determine whether to do backups. It
> can backup very large files with small changes very quickly.
>
> Dominic
>

You dont understand me, rdiff-backup is efficient, but to make diff it
must read WHOLE file, on remote nfs or sshfs it is SLOOOOW and
painful.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Re: Remote encrypted backup with slow connection.

Dominic Raferd-2
Piotr Karbowski wrote:

> On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd <[hidden email]> wrote:
>  
>> Piotr Karbowski wrote:
>>    
>>> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller <[hidden email]> wrote:
>>>
>>>      
>>>> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote:
>>>>
>>>>        
>>>>> local rdiff-backup dir with remote server but how? If I will use for
>>>>> example rsync it still need to check whole files for changes (read,
>>>>> download it) and upload only new. I hope you will understand what I
>>>>> need and help me.
>>>>>
>>>>>          
>>>> rsync won't check whole files unless you give the -c flag. Otherwise, it
>>>> just compares metadata. I don't know if that's also the case with
>>>> rdiff-backup, but I assume so.
>>>>
>>>>        
>>> So I need to know how rdiff default compares data, if by size and
>>> mod-time, it will not be so painful but still itefficient  will download changed
>>> files to generate diff.
>>>      
>> Rdiff-backup is designed to be ultra-efficient at this activity. It only
>> sends the changes in a file over the wire, not the whole file. To do this it
>> uses the librsync library which is effectively the same as rsync. You can
>> read more about the technique at http://en.wikipedia.org/wiki/Rsync.
>> rdiff-backup does not use file times to determine whether to do backups. It
>> can backup very large files with small changes very quickly.
>>
>> Dominic
>>
>>    
>
> You dont understand me, rdiff-backup is efficient, but to make diff it
> must read WHOLE file, on remote nfs or sshfs it is SLOOOOW and
> painful
Sorry I get it now. But I think rdiff-backup and rsync require a
separate computer at the remote end in order to optimise transfers, so
if you are just accessing a remote share using sshfs or similar then
they can still work of course but as you realise they will be slow. I
guess it is not possible for you to run rdiff-backup (or rsync) at the
remote end as well?

You could run rdiff-backup locally to create a backup store and then
mirror this store to the remote share using rcp. Still it will be slow
because rdiff-backup always stores the latest copy of each file in full
and so if this changes even slightly then the whole file will must be
transferred by rcp.

Duplicity http://duplicity.nongnu.org/ might work better for you,
because it uses forward diffs. Also its archives are secure.

Although not directly relevant I found a page here
http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch
to greatly speed up OpenSSH in some situations.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Re: Remote encrypted backup with slow connection.

Piotr Karbowski
On Tue, Sep 22, 2009 at 10:51 AM, Dominic Raferd <[hidden email]> wrote:

> Piotr Karbowski wrote:
>>
>> On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd <[hidden email]>
>> wrote:
>>
>>>
>>> Piotr Karbowski wrote:
>>>
>>>>
>>>> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller <[hidden email]>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> local rdiff-backup dir with remote server but how? If I will use for
>>>>>> example rsync it still need to check whole files for changes (read,
>>>>>> download it) and upload only new. I hope you will understand what I
>>>>>> need and help me.
>>>>>>
>>>>>>
>>>>>
>>>>> rsync won't check whole files unless you give the -c flag. Otherwise,
>>>>> it
>>>>> just compares metadata. I don't know if that's also the case with
>>>>> rdiff-backup, but I assume so.
>>>>>
>>>>>
>>>>
>>>> So I need to know how rdiff default compares data, if by size and
>>>> mod-time, it will not be so painful but still itefficient  will download
>>>> changed
>>>> files to generate diff.
>>>>
>>>
>>> Rdiff-backup is designed to be ultra-efficient at this activity. It only
>>> sends the changes in a file over the wire, not the whole file. To do this
>>> it
>>> uses the librsync library which is effectively the same as rsync. You can
>>> read more about the technique at http://en.wikipedia.org/wiki/Rsync.
>>> rdiff-backup does not use file times to determine whether to do backups.
>>> It
>>> can backup very large files with small changes very quickly.
>>>
>>> Dominic
>>>
>>>
>>
>> You dont understand me, rdiff-backup is efficient, but to make diff it
>> must read WHOLE file, on remote nfs or sshfs it is SLOOOOW and
>> painful
>
> Sorry I get it now. But I think rdiff-backup and rsync require a separate
> computer at the remote end in order to optimise transfers, so if you are
> just accessing a remote share using sshfs or similar then they can still
> work of course but as you realise they will be slow. I guess it is not
> possible for you to run rdiff-backup (or rsync) at the remote end as well?
>
> You could run rdiff-backup locally to create a backup store and then mirror
> this store to the remote share using rcp. Still it will be slow because
> rdiff-backup always stores the latest copy of each file in full and so if
> this changes even slightly then the whole file will must be transferred by
> rcp.
>
> Duplicity http://duplicity.nongnu.org/ might work better for you, because it
> uses forward diffs. Also its archives are secure.
>
> Although not directly relevant I found a page here
> http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to
> greatly speed up OpenSSH in some situations.

Duplicity is interesing project. What you think about using
rdiff-backup to create local backup, for example in /backups and then
send this /backups to remote server by duplicity? As far as I know
duplicity is encrypted so I DONT need using encfs, dmcrypt or other -
only ssh access is needed (realy I dont need duplicity on remote
server?).

I just want be able to send _ENCRYPTED_ backups to remote server where
I have only ssh access (sftp/scp work).


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Re: Remote encrypted backup with slow connection.

Dominic Raferd-2
Piotr Karbowski wrote:

> On Tue, Sep 22, 2009 at 10:51 AM, Dominic Raferd <[hidden email]> wrote:
>  
>> Piotr Karbowski wrote:
>>    
>>> On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd <[hidden email]>
>>> wrote:
>>>
>>>      
>>>> Piotr Karbowski wrote:
>>>>
>>>>        
>>>>> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>
>>>>>          
>>>>>> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote:
>>>>>>
>>>>>>
>>>>>>            
>>>>>>> local rdiff-backup dir with remote server but how? If I will use for
>>>>>>> example rsync it still need to check whole files for changes (read,
>>>>>>> download it) and upload only new. I hope you will understand what I
>>>>>>> need and help me.
>>>>>>>
>>>>>>>
>>>>>>>              
>>>>>> rsync won't check whole files unless you give the -c flag. Otherwise,
>>>>>> it
>>>>>> just compares metadata. I don't know if that's also the case with
>>>>>> rdiff-backup, but I assume so.
>>>>>>
>>>>>>
>>>>>>            
>>>>> So I need to know how rdiff default compares data, if by size and
>>>>> mod-time, it will not be so painful but still itefficient  will download
>>>>> changed
>>>>> files to generate diff.
>>>>>
>>>>>          
>>>> Rdiff-backup is designed to be ultra-efficient at this activity. It only
>>>> sends the changes in a file over the wire, not the whole file. To do this
>>>> it
>>>> uses the librsync library which is effectively the same as rsync. You can
>>>> read more about the technique at http://en.wikipedia.org/wiki/Rsync.
>>>> rdiff-backup does not use file times to determine whether to do backups.
>>>> It
>>>> can backup very large files with small changes very quickly.
>>>>
>>>> Dominic
>>>>
>>>>
>>>>        
>>> You dont understand me, rdiff-backup is efficient, but to make diff it
>>> must read WHOLE file, on remote nfs or sshfs it is SLOOOOW and
>>> painful
>>>      
>> Sorry I get it now. But I think rdiff-backup and rsync require a separate
>> computer at the remote end in order to optimise transfers, so if you are
>> just accessing a remote share using sshfs or similar then they can still
>> work of course but as you realise they will be slow. I guess it is not
>> possible for you to run rdiff-backup (or rsync) at the remote end as well?
>>
>> You could run rdiff-backup locally to create a backup store and then mirror
>> this store to the remote share using rcp. Still it will be slow because
>> rdiff-backup always stores the latest copy of each file in full and so if
>> this changes even slightly then the whole file will must be transferred by
>> rcp.
>>
>> Duplicity http://duplicity.nongnu.org/ might work better for you, because it
>> uses forward diffs. Also its archives are secure.
>>
>> Although not directly relevant I found a page here
>> http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to
>> greatly speed up OpenSSH in some situations.
>>    
>
> Duplicity is interesing project. What you think about using
> rdiff-backup to create local backup, for example in /backups and then
> send this /backups to remote server by duplicity? As far as I know
> duplicity is encrypted so I DONT need using encfs, dmcrypt or other -
> only ssh access is needed (realy I dont need duplicity on remote
> server?).
>
> I just want be able to send _ENCRYPTED_ backups to remote server where
> I have only ssh access (sftp/scp work).
I have not used duplicity myself, I use rdiff-backup. But I am not sure
you need to run rdiff-backup first, I think duplicity may make its own
local copies of backup increments so that it can send future increments
without having to access the earlier increments from the remote share.
And yes duplicity sends encrypted files so you don't need other encryption.

Because duplicity uses forward diffs you have to keep all backups
forever, and if there is any corruption of a file you lose all backups
that occurred *after* the date of this file. Rdiff-backup uses reverse
diffs so corruption, if it occurs, affects backups *before* the date of
the corrupted backup. With rdiff-backup you can delete backups before a
certain date (though in my experience the storage is so efficient it is
not usually worth bothering), which is not possible with duplicity.

Still it sounds like duplicity would better suit your needs.


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Re: Remote encrypted backup with slow connection.

Piotr Karbowski
On Tue, Sep 22, 2009 at 4:36 PM, Dominic Raferd <[hidden email]> wrote:

> Piotr Karbowski wrote:
>>
>> On Tue, Sep 22, 2009 at 10:51 AM, Dominic Raferd <[hidden email]>
>> wrote:
>>
>>>
>>> Piotr Karbowski wrote:
>>>
>>>>
>>>> On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd <[hidden email]>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> Piotr Karbowski wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> local rdiff-backup dir with remote server but how? If I will use for
>>>>>>>> example rsync it still need to check whole files for changes (read,
>>>>>>>> download it) and upload only new. I hope you will understand what I
>>>>>>>> need and help me.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> rsync won't check whole files unless you give the -c flag. Otherwise,
>>>>>>> it
>>>>>>> just compares metadata. I don't know if that's also the case with
>>>>>>> rdiff-backup, but I assume so.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> So I need to know how rdiff default compares data, if by size and
>>>>>> mod-time, it will not be so painful but still itefficient  will
>>>>>> download
>>>>>> changed
>>>>>> files to generate diff.
>>>>>>
>>>>>>
>>>>>
>>>>> Rdiff-backup is designed to be ultra-efficient at this activity. It
>>>>> only
>>>>> sends the changes in a file over the wire, not the whole file. To do
>>>>> this
>>>>> it
>>>>> uses the librsync library which is effectively the same as rsync. You
>>>>> can
>>>>> read more about the technique at http://en.wikipedia.org/wiki/Rsync.
>>>>> rdiff-backup does not use file times to determine whether to do
>>>>> backups.
>>>>> It
>>>>> can backup very large files with small changes very quickly.
>>>>>
>>>>> Dominic
>>>>>
>>>>>
>>>>>
>>>>
>>>> You dont understand me, rdiff-backup is efficient, but to make diff it
>>>> must read WHOLE file, on remote nfs or sshfs it is SLOOOOW and
>>>> painful
>>>>
>>>
>>> Sorry I get it now. But I think rdiff-backup and rsync require a separate
>>> computer at the remote end in order to optimise transfers, so if you are
>>> just accessing a remote share using sshfs or similar then they can still
>>> work of course but as you realise they will be slow. I guess it is not
>>> possible for you to run rdiff-backup (or rsync) at the remote end as
>>> well?
>>>
>>> You could run rdiff-backup locally to create a backup store and then
>>> mirror
>>> this store to the remote share using rcp. Still it will be slow because
>>> rdiff-backup always stores the latest copy of each file in full and so if
>>> this changes even slightly then the whole file will must be transferred
>>> by
>>> rcp.
>>>
>>> Duplicity http://duplicity.nongnu.org/ might work better for you, because
>>> it
>>> uses forward diffs. Also its archives are secure.
>>>
>>> Although not directly relevant I found a page here
>>> http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch to
>>> greatly speed up OpenSSH in some situations.
>>>
>>
>> Duplicity is interesing project. What you think about using
>> rdiff-backup to create local backup, for example in /backups and then
>> send this /backups to remote server by duplicity? As far as I know
>> duplicity is encrypted so I DONT need using encfs, dmcrypt or other -
>> only ssh access is needed (realy I dont need duplicity on remote
>> server?).
>>
>> I just want be able to send _ENCRYPTED_ backups to remote server where
>> I have only ssh access (sftp/scp work).
>
> I have not used duplicity myself, I use rdiff-backup. But I am not sure you
> need to run rdiff-backup first, I think duplicity may make its own local
> copies of backup increments so that it can send future increments without
> having to access the earlier increments from the remote share. And yes
> duplicity sends encrypted files so you don't need other encryption.
>
> Because duplicity uses forward diffs you have to keep all backups forever,
> and if there is any corruption of a file you lose all backups that occurred
> *after* the date of this file. Rdiff-backup uses reverse diffs so
> corruption, if it occurs, affects backups *before* the date of the corrupted
> backup. With rdiff-backup you can delete backups before a certain date
> (though in my experience the storage is so efficient it is not usually worth
> bothering), which is not possible with duplicity.
>
> Still it sounds like duplicity would better suit your needs.
>

So best will be using duplicity to make local backup and run rsync to
send new files (diffs) to remote server. and if backup will be TOO big
just mv backup backup_old and start new backup (every 4 weeks for
example)


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Re: Remote encrypted backup with slow connection.

Dominic Raferd-2
Piotr Karbowski wrote:
> So best will be using duplicity to make local backup and run rsync to
> send new files (diffs) to remote server. and if backup will be TOO big
> just mv backup backup_old and start new backup (every 4 weeks for
> example)
>  
I think so. There is a duplicity mailing list where you could probably
get more help: http://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
Reply | Threaded
Open this post in threaded view
|

Re: Re: Remote encrypted backup with slow connection.

Piotr Karbowski
On Tue, Sep 22, 2009 at 5:45 PM, Dominic Raferd <[hidden email]> wrote:

> Piotr Karbowski wrote:
>>
>> So best will be using duplicity to make local backup and run rsync to
>> send new files (diffs) to remote server. and if backup will be TOO big
>> just mv backup backup_old and start new backup (every 4 weeks for
>> example)
>>
>
> I think so. There is a duplicity mailing list where you could probably get
> more help: http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

Thanks for you all!


_______________________________________________
rdiff-backup-users mailing list at [hidden email]
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki