Minimizing risk on incremental

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Minimizing risk on incremental

duplicity-talk mailing list

As written on Wikipedia, about Duplicity:

“Chains consisting of a full backup and a series of incremental backups can be recovered to the point in time that any of the incremental steps were taken. If any of the incremental backups are missing then the incremental backups following it cannot be reconstructed.”

 

Im concerned about those chains, in my particular implementation.

Im using rsync.net as backend, and they automatically create a daily snapshot of whatever we put there.

Each snapshot is a “picture of the file system” as it existed when was taken.

 

If I just do simple rsync to rsync.net, only the changes are transmitted, but the filesystem is a “copy” of the source, not an “incremental backup chain”

Therefore, the rsync.net’s daily snapshot have only the changes, and grows by the size of the data changed every day, letting me recover any snapshot at any time, even if the snapshots in between are lost/deleted.

 

If I use Duplicity, I will be sending incremental every day, and if one of those days is missing or corrupted, all my snapshots are worthless, right?

For example, lets say I have 400 GB of original data, and 2 GB of changes per day.

 

Day 1: Duplicity, first backup, totaling 400 GB (full)

Day 1: rsync.net snapshot (full is recoverable)

Day 2: Duplicity, incremental, totaling 2 GB (incremental)

Day 2: rsync.net snapshot (full is recoverable, incremental might be broken)

Day 3: Duplicity, incremental, totaling 2 GB (incremental)

Day 2: rsync.net snapshot (full is recoverable, incremental might be broken because Day 2 is broken/missing)

Etc

 

If I continue forever, then I might be on day 200 and be able to recovery only full back up from Day1.

What am I missing here?

 

I guess I could do full backup, every week, but that will cause a full 400GB of data every week, so in few weeks I will have a few Terabytes of data on rsync.net.

 

Is there any way to make duplicity just sync the file system, and avoid the incremental part?

The attractive part of duplicity in my case is the encryption, but it seems the “incremental backup” is not ideal for me.

 

Thoughts?

Thanks!

Dotty

 

 

 

 

 

 

 


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Minimizing risk on incremental

duplicity-talk mailing list
You can use the par2 wrapper backend that takes a little more space but provides redundancy if one of the chain files gets corrupted somehow (flips a bit...Solar radiation? Bad disk?)

~mark

On Apr 22, 2017 6:15 AM, "Manuel Morales via Duplicity-talk" <[hidden email]> wrote:

As written on Wikipedia, about Duplicity:

“Chains consisting of a full backup and a series of incremental backups can be recovered to the point in time that any of the incremental steps were taken. If any of the incremental backups are missing then the incremental backups following it cannot be reconstructed.”

 

Im concerned about those chains, in my particular implementation.

Im using rsync.net as backend, and they automatically create a daily snapshot of whatever we put there.

Each snapshot is a “picture of the file system” as it existed when was taken.

 

If I just do simple rsync to rsync.net, only the changes are transmitted, but the filesystem is a “copy” of the source, not an “incremental backup chain”

Therefore, the rsync.net’s daily snapshot have only the changes, and grows by the size of the data changed every day, letting me recover any snapshot at any time, even if the snapshots in between are lost/deleted.

 

If I use Duplicity, I will be sending incremental every day, and if one of those days is missing or corrupted, all my snapshots are worthless, right?

For example, lets say I have 400 GB of original data, and 2 GB of changes per day.

 

Day 1: Duplicity, first backup, totaling 400 GB (full)

Day 1: rsync.net snapshot (full is recoverable)

Day 2: Duplicity, incremental, totaling 2 GB (incremental)

Day 2: rsync.net snapshot (full is recoverable, incremental might be broken)

Day 3: Duplicity, incremental, totaling 2 GB (incremental)

Day 2: rsync.net snapshot (full is recoverable, incremental might be broken because Day 2 is broken/missing)

Etc

 

If I continue forever, then I might be on day 200 and be able to recovery only full back up from Day1.

What am I missing here?

 

I guess I could do full backup, every week, but that will cause a full 400GB of data every week, so in few weeks I will have a few Terabytes of data on rsync.net.

 

Is there any way to make duplicity just sync the file system, and avoid the incremental part?

The attractive part of duplicity in my case is the encryption, but it seems the “incremental backup” is not ideal for me.

 

Thoughts?

Thanks!

Dotty

 

 

 

 

 

 

 


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Minimizing risk on incremental

duplicity-talk mailing list

That’s a good ideal also, but like you said, it will require more space.

 

Duplicity seems to be great, in the traditional backup sense, the problem here is that IM using ZFS snapshots, which basically provide the same, but avoiding the broken chain issues.

In my case then I don’t know, if I use duplicity I get the nice encryption but I introduce a backup chain in my logic, and if I don’t use, then the data inside my snapshots will not be encrypted.

 

Is there any way to disable the chain in duplicity and have it behave just like rsync but just adding encryption?

 

 

 

From: Mark Grandi <[hidden email]>
Date: Saturday, April 22, 2017 at 2:39 PM
To: Discussion of the backup program duplicity <[hidden email]>
Cc: Manuel Morales <[hidden email]>
Subject: Re: [Duplicity-talk] Minimizing risk on incremental

 

You can use the par2 wrapper backend that takes a little more space but provides redundancy if one of the chain files gets corrupted somehow (flips a bit...Solar radiation? Bad disk?)

 

~mark

 

On Apr 22, 2017 6:15 AM, "Manuel Morales via Duplicity-talk" <[hidden email]> wrote:

As written on Wikipedia, about Duplicity:

“Chains consisting of a full backup and a series of incremental backups can be recovered to the point in time that any of the incremental steps were taken. If any of the incremental backups are missing then the incremental backups following it cannot be reconstructed.”

 

Im concerned about those chains, in my particular implementation.

Im using rsync.net as backend, and they automatically create a daily snapshot of whatever we put there.

Each snapshot is a “picture of the file system” as it existed when was taken.

 

If I just do simple rsync to rsync.net, only the changes are transmitted, but the filesystem is a “copy” of the source, not an “incremental backup chain”

Therefore, the rsync.net’s daily snapshot have only the changes, and grows by the size of the data changed every day, letting me recover any snapshot at any time, even if the snapshots in between are lost/deleted.

 

If I use Duplicity, I will be sending incremental every day, and if one of those days is missing or corrupted, all my snapshots are worthless, right?

For example, lets say I have 400 GB of original data, and 2 GB of changes per day.

 

Day 1: Duplicity, first backup, totaling 400 GB (full)

Day 1: rsync.net snapshot (full is recoverable)

Day 2: Duplicity, incremental, totaling 2 GB (incremental)

Day 2: rsync.net snapshot (full is recoverable, incremental might be broken)

Day 3: Duplicity, incremental, totaling 2 GB (incremental)

Day 2: rsync.net snapshot (full is recoverable, incremental might be broken because Day 2 is broken/missing)

Etc

 

If I continue forever, then I might be on day 200 and be able to recovery only full back up from Day1.

What am I missing here?

 

I guess I could do full backup, every week, but that will cause a full 400GB of data every week, so in few weeks I will have a few Terabytes of data on rsync.net.

 

Is there any way to make duplicity just sync the file system, and avoid the incremental part?

The attractive part of duplicity in my case is the encryption, but it seems the “incremental backup” is not ideal for me.

 

Thoughts?

Thanks!

Dotty

 

 

 

 

 

 

 


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Minimizing risk on incremental

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On Sat, 22 Apr 2017 11:39:24 -0700
Mark Grandi via Duplicity-talk <[hidden email]> wrote:

> You can use the par2 wrapper backend that takes a little more space
> but provides redundancy if one of the chain files gets corrupted
> somehow (flips a bit...Solar radiation? Bad disk?)

I'm interested in this wrapper but can't find a reference to it.
My version is 0.6.22

harry


> ~mark
>
> On Apr 22, 2017 6:15 AM, "Manuel Morales via Duplicity-talk" <
> [hidden email]> wrote:
>
> > As written on Wikipedia, about Duplicity:
> >
> > “Chains consisting of a full backup and a series of incremental
> > backups can be recovered to the point in time that any of the
> > incremental steps were taken. If any of the incremental backups are
> > missing then the incremental backups following it cannot be
> > reconstructed.”
> >
> >
> >
> > Im concerned about those chains, in my particular implementation.
> >
> > Im using rsync.net as backend, and they automatically create a daily
> > snapshot of whatever we put there.
> >
> > Each snapshot is a “picture of the file system” as it existed when
> > was taken.
> >
> >
> >
> > If I just do simple rsync to rsync.net, only the changes are
> > transmitted, but the filesystem is a “copy” of the source, not an
> > “incremental backup chain”
> >
> > Therefore, the rsync.net’s daily snapshot have only the changes, and
> > grows by the size of the data changed every day, letting me recover
> > any snapshot at any time, even if the snapshots in between are
> > lost/deleted.
> >
> >
> >
> > If I use Duplicity, I will be sending incremental every day, and if
> > one of those days is missing or corrupted, all my snapshots are
> > worthless, right?
> >
> > For example, lets say I have 400 GB of original data, and 2 GB of
> > changes per day.
> >
> >
> >
> > Day 1: Duplicity, first backup, totaling 400 GB (full)
> >
> > Day 1: rsync.net snapshot (full is recoverable)
> >
> > Day 2: Duplicity, incremental, totaling 2 GB (incremental)
> >
> > Day 2: rsync.net snapshot (full is recoverable, incremental might be
> > broken)
> >
> > Day 3: Duplicity, incremental, totaling 2 GB (incremental)
> >
> > Day 2: rsync.net snapshot (full is recoverable, incremental might be
> > broken because Day 2 is broken/missing)
> >
> > Etc
> >
> >
> >
> > If I continue forever, then I might be on day 200 and be able to
> > recovery only full back up from Day1.
> >
> > What am I missing here?
> >
> >
> >
> > I guess I could do full backup, every week, but that will cause a
> > full 400GB of data every week, so in few weeks I will have a few
> > Terabytes of data on rsync.net.
> >
> >
> >
> > Is there any way to make duplicity just sync the file system, and
> > avoid the incremental part?
> >
> > The attractive part of duplicity in my case is the encryption, but
> > it seems the “incremental backup” is not ideal for me.
> >
> >
> >
> > Thoughts?
> >
> > Thanks!
> >
> > Dotty
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > [hidden email]
> > https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
> >


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Minimizing risk on incremental

duplicity-talk mailing list
On 23.04.2017 07:10, arie via Duplicity-talk wrote:
>> You can use the par2 wrapper backend that takes a little more space
>> but provides redundancy if one of the chain files gets corrupted
>> somehow (flips a bit...Solar radiation? Bad disk?)
> I'm interested in this wrapper but can't find a reference to it.
> My version is 0.6.22

Harry,

0.6.22 is very old. use your interest as a signal to update to the latest stable 0.7.12, which holds 3,5years fixes and improvements.

if you want to keep your old version, just in case something does not work as expected, you may want to try to install the tarball in parallel as described here under TIPS->INSTALL MULTIPLE VERSIONS
  duply.net/wiki/index.php/Duply-documentation

..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Minimizing risk on incremental

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
hey Manuel,

On 22.04.2017 15:15, Manuel Morales via Duplicity-talk wrote:
> I guess I could do full backup, every week, but that will cause a full 400GB of data every week, so in few weeks I will have a few Terabytes of data on rsync.net.
>

check the ml or the search engine of your choice. this topic comes up every now and then and generally, yes find a pattern of fulls/incrs that fit's your storage constraints and max chain length (==risk or corruption).

>
> Is there any way to make duplicity just sync the file system, and avoid the incremental part?
>
> The attractive part of duplicity in my case is the encryption, but it seems the “incremental backup” is not ideal for me.

putting files on backends simply is most compatible way to use virtually any backend (duplicity even supports imap) and that leads to backup chains. incrementals are essentially what you describe as syncing.

problem is duplicity treating backends as dumb, so there is no way to efficiently
 duplicate/verify the old full
or
 consolidate a new full from the latest chain state
on the backend only without transferring the whole shebang back to the source machine.

but that will probably never change, as the target backend is treated as insecure and should not be allowed to decrypt your backups.

in summary:

if you want to stick w/ duplicity, consider
 to do regular fulls
and
 verify regularly
and
 check your backup logs
.
the verify makes sure that your backup is restorable and not corrupted up to now. manually start a new chain in case your verify fails at some point.

..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Minimizing risk on incremental

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On Sun, 23 Apr 2017 12:18:58 +0200
"edgar.soldin--- via Duplicity-talk" <[hidden email]> wrote:

> On 23.04.2017 07:10, arie via Duplicity-talk wrote:
> >> You can use the par2 wrapper backend that takes a little more space
> >> but provides redundancy if one of the chain files gets corrupted
> >> somehow (flips a bit...Solar radiation? Bad disk?)
> > I'm interested in this wrapper but can't find a reference to it.
> > My version is 0.6.22
>
> Harry,
>
> 0.6.22 is very old. use your interest as a signal to update to the
> latest stable 0.7.12, which holds 3,5years fixes and improvements.
>
> if you want to keep your old version, just in case something does not
> work as expected, you may want to try to install the tarball in
> parallel as described here under TIPS->INSTALL MULTIPLE VERSIONS
> duply.net/wiki/index.php/Duply-documentation
>
> ..ede/duply.net
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Thank you, I was afraid an update to 0.7.12 on my Fedora 18
would lead to the well-known dependency hell.
The documentation that allows parallel trials is most welcome.

harry

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Minimizing risk on incremental

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On Sat, Apr 22, 2017 at 09:15:17AM -0400, Manuel Morales via Duplicity-talk wrote:
> The attractive part of duplicity in my case is the encryption, but it seems the “incremental backup” is not ideal for me.

You may want to check out borgbackup: https://borgbackup.readthedocs.io/en/stable/

Borg backups can be encrypted, and each backup archive is a full backup
of the input files.  But Borg uses rsync-like chunking and deduplication
to store only new chunks as you add archives, so you get full backup
archives for the space cost of an incremental.  See the borg docs for
more detail.

Unlike Duplicity, borg needs smarts on the backend (for the
deduplication), so has a much smaller set of supported backends.  
However, you mentioned that you use rsync.net, which is one of the
providers that supports borg: http://rsync.net/products/attic.html

I don't have any experience with using borg with rsync.net, but I do use
borg and am happy with it.  Please email me directly or contact borg
support channels if you have questions, to avoid further off-topic
conversation on this mailing list.

--
Ed Blackman

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Minimizing risk on incremental

duplicity-talk mailing list

I will definitely test it

 

Thanks a lot for the suggestion

 

Dotty

 

 

From: Ed Blackman <[hidden email]>
Date: Monday, April 24, 2017 at 11:22 AM
To: Manuel Morales via Duplicity-talk <[hidden email]>
Cc: Manuel Morales <[hidden email]>
Subject: Re: [Duplicity-talk] Minimizing risk on incremental

 

On Sat, Apr 22, 2017 at 09:15:17AM -0400, Manuel Morales via Duplicity-talk wrote:

The attractive part of duplicity in my case is the encryption, but it seems the “incremental backup” is not ideal for me.

 

You may want to check out borgbackup: https://borgbackup.readthedocs.io/en/stable/

 

Borg backups can be encrypted, and each backup archive is a full backup

of the input files.  But Borg uses rsync-like chunking and deduplication

to store only new chunks as you add archives, so you get full backup

archives for the space cost of an incremental.  See the borg docs for

more detail.

 

Unlike Duplicity, borg needs smarts on the backend (for the

deduplication), so has a much smaller set of supported backends.  

However, you mentioned that you use rsync.net, which is one of the

providers that supports borg: http://rsync.net/products/attic.html

 

I don't have any experience with using borg with rsync.net, but I do use

borg and am happy with it.  Please email me directly or contact borg

support channels if you have questions, to avoid further off-topic

conversation on this mailing list.

 

--

Ed Blackman

 


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Minimizing risk on incremental

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
Hello Manuel,


> On 22.04.2017 15:15, Manuel Morales via Duplicity-talk wrote:
>
>> Is there any way to make duplicity just sync the file system, and avoid the incremental part?
>>
>> The attractive part of duplicity in my case is the encryption, but it seems the “incremental backup” is not ideal for me.
> in summary:
>
> if you want to stick w/ duplicity, consider
>   to do regular fulls
> and
>   verify regularly
> and
>   check your backup logs
>

In practice I address a similar problem to yours by running duplicity to
a local folder (/external HDD etc) and then rsyncing the archive files
to various places. That way you can verify every day/week and know that
you have a working backup chain, but you only have to send the deltas to
the remote location. Clearly this does not protect you if one of your
archive files is corrupted etc on the remote server, unless this is
picked up in the next rsync.

Kind regards,

Aaron

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk