How to use duplicity for local backup

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

How to use duplicity for local backup

duplicity-talk mailing list
Hello, I want to ask opinion about backup tactic please.

I found a backup system that using only rsync to a local server. A few
scripts run on the server that create tree of historical directrories
that are filled with hard links to the main rsync target. I never saw
this but my understand of the benefit:
* saving space for historical backups because they only use inodes
* simple scheme to use rsync and ln
* But am I correct, to understand if a file *change* then the historical
data is lost so its a shortcoming of the scheme.

Is this scheme popular? Is it useful in comparison to local-rsync
variation of duplicity? For example, lower CPU needed because not
creating archive files and encrypting them. Can duplicity have gpg turn
off when it is not necessary? I prefer duplicity to replace the scheme
but have limited storage and limited CPU cycles.

Could you guide to help make duplicity a comparable option in that
context? Thank you.

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: How to use duplicity for local backup

duplicity-talk mailing list
hi MRob,

On 24.04.2020 19:50, MRob via Duplicity-talk wrote:
> Hello, I want to ask opinion about backup tactic please.
>
> I found a backup system that using only rsync to a local server. A few scripts run on the server that create tree of historical directrories that are filled with hard links to the main rsync target. I never saw this but my understand of the benefit:
> * saving space for historical backups because they only use inodes
> * simple scheme to use rsync and ln

off-topic but interestingly there is a popular backup strategy like that.

on the backend (where the backups are stored)
- create mirror/snapshot of the last backup folder (eg. using file systems like btrfs, zfs or using 'rsync --link-dest=...' below)
on the backend or locally
- run rsync and synchronize the new backend folder
repeat

using cow(copy-on-write) filesystem you can even have rsync not to create a new file but just replace chunks of big ones, saving even more space.

> * But am I correct, to understand if a file *change* then the historical data is lost so its a shortcoming of the scheme.

not necessarily as rsync by default creates a new file if it detected changes in the old one, hence your old hardlinks won't be touched in that case

> Is this scheme popular?

relatively. i for one use it in some places.

>Is it useful in comparison to local-rsync variation of duplicity?

not comparable to what duplicity does. duplicity creates final volumes of backup data that are merely dumped on some backend.

>For example, lower CPU needed because not creating archive files and encrypting them.

as said. completely different approach. you need eg. a backend that you trust your unencrypted data with.

>Can duplicity have gpg turn off when it is not necessary?

yes. but then you may as well check out thy myriad of backup solutions out there. one might be a better fit for you.

>I prefer duplicity to replace the scheme but have limited storage and limited CPU cycles.

why, if it is working now?

> Could you guide to help make duplicity a comparable option in that context? Thank you.

again, not comparable sorry.


have fun ..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: How to use duplicity for local backup

duplicity-talk mailing list
On 2020-04-25 10:18, edgar.soldin--- via Duplicity-talk wrote:

> hi MRob,
>
> On 24.04.2020 19:50, MRob via Duplicity-talk wrote:
>> Hello, I want to ask opinion about backup tactic please.
>>
>> I found a backup system that using only rsync to a local server. A few
>> scripts run on the server that create tree of historical directrories
>> that are filled with hard links to the main rsync target. I never saw
>> this but my understand of the benefit:
>> * saving space for historical backups because they only use inodes
>> * simple scheme to use rsync and ln
>
> off-topic but interestingly there is a popular backup strategy like
> that.
>
> on the backend (where the backups are stored)
> - create mirror/snapshot of the last backup folder (eg. using file
> systems like btrfs, zfs or using 'rsync --link-dest=...' below)
> on the backend or locally
> - run rsync and synchronize the new backend folder
> repeat

The system Im looking at isn't that. It uses:

1. rsync from server to local backup server
2. "cp -l" for making incrementals

>> * But am I correct, to understand if a file *change* then the
>> historical data is lost so its a shortcoming of the scheme.
>
> not necessarily as rsync by default creates a new file if it detected
> changes in the old one, hence your old hardlinks won't be touched in
> that case

However with "cp -l" can I assume changes are lost?

>> Is this scheme popular?
>
> relatively. i for one use it in some places.
>
>> Is it useful in comparison to local-rsync variation of duplicity?
>
> not comparable to what duplicity does. duplicity creates final volumes
> of backup data that are merely dumped on some backend.
>
>> For example, lower CPU needed because not creating archive files and
>> encrypting them.
>
> as said. completely different approach. you need eg. a backend that
> you trust your unencrypted data with.

Yes it is local system, trusted. In such case is duplicity not the right
Tool?

>> Can duplicity have gpg turn off when it is not necessary?
>
> yes. but then you may as well check out thy myriad of backup solutions
> out there. one might be a better fit for you.

The point of my thread. Looking for advice here, please.

Maybe I change "cp -l" to "rsync --list-dest" it will be more CPU and
space efficent than duplicity without gpg?

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: How to use duplicity for local backup

duplicity-talk mailing list


> On Apr 27, 2020, at 10:37 AM, MRob via Duplicity-talk <[hidden email]> wrote:
>
>>> Can duplicity have gpg turn off when it is not necessary?
>> yes. but then you may as well check out thy myriad of backup solutions
>> out there. one might be a better fit for you.
>
> The point of my thread. Looking for advice here, please.

Actually this list is probably not the best source for advice on what software does a different task than duplicity….  I can tell you why I use duplicity but I don’t have a lot of expertise on why I don’t need duplicity.

You are looking for a general backup solution without encryption.  There are lots of resources and commercial systems that will tell you how to do that.  Duplicities strength is the encryption and optimization for untrusted off site storage.  If you want something else, then you should probably be asking other folks than us.  That is just not our groove, man!

-Scott



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: How to use duplicity for local backup

duplicity-talk mailing list
On 2020-04-27 14:46, Scott Hannahs wrote:

>> On Apr 27, 2020, at 10:37 AM, MRob via Duplicity-talk
>> <[hidden email]> wrote:
>>
>>>> Can duplicity have gpg turn off when it is not necessary?
>>> yes. but then you may as well check out thy myriad of backup
>>> solutions
>>> out there. one might be a better fit for you.
>>
>> The point of my thread. Looking for advice here, please.
>
> Actually this list is probably not the best source for advice on what
> software does a different task than duplicity….  I can tell you why I
> use duplicity but I don’t have a lot of expertise on why I don’t need
> duplicity.
>
> You are looking for a general backup solution without encryption.

Sorry for off topic. But Edgar's reply shows that there is that
expertise on this list.

Original I thought duplicity can be tuned for trusted local backup. At
minimum I think duplicity experts can tell opinion if duplicity
can/cannot be as CPU and space efficient as rsync-based approach Edgar
described.

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: How to use duplicity for local backup

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On 2020-04-27 14:46, Scott Hannahs wrote:

>> On Apr 27, 2020, at 10:37 AM, MRob via Duplicity-talk
>> <[hidden email]> wrote:
>>
>>>> Can duplicity have gpg turn off when it is not necessary?
>>> yes. but then you may as well check out thy myriad of backup
>>> solutions
>>> out there. one might be a better fit for you.
>>
>> The point of my thread. Looking for advice here, please.
>
> Actually this list is probably not the best source for advice on what
> software does a different task than duplicity….  I can tell you why I
> use duplicity but I don’t have a lot of expertise on why I don’t need
> duplicity.
>
> You are looking for a general backup solution without encryption.

Sorry for off topic. But Edgar's reply shows that there is that
expertise on this list.

Original I thought duplicity can be tuned for trusted local backup. At
minimum I think duplicity experts can tell opinion if duplicity
can/cannot be as CPU and space efficient as rsync-based approach Edgar
described.

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: How to use duplicity for local backup

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On Fri, 24 Apr 2020, MRob via Duplicity-talk wrote:

> Can duplicity have gpg turn off when it is not necessary?

Yes, with the (clearly documented) --no-encryption option.

Maybe it would be better if you just try this out and see if it meets your
needs, instead of arguing or trying to guess as to what will or won't be
the best solution?

--
Nate Eldredge
[hidden email]


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: How to use duplicity for local backup

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On Fri, 24 Apr 2020, MRob via Duplicity-talk wrote:

> Hello, I want to ask opinion about backup tactic please.
>
> I found a backup system that using only rsync to a local server. A few
> scripts run on the server that create tree of historical directrories
> that are filled with hard links to the main rsync target. I never saw
> this but my understand of the benefit:
> * saving space for historical backups because they only use inodes
> * simple scheme to use rsync and ln
> * But am I correct, to understand if a file *change* then the historical
> data is lost so its a shortcoming of the scheme.

Well, it could work either way.  Maybe it only keeps the latest version of
files that change, maybe it keeps all past versions.  I can't tell from
your description alone.  It'd help if you would provide a link to the
system you are talking about.

Duplicity uses the rdiff algorithm to only track the changes in a file's
contents.  So if you have a 1 GB file and you add 1 KB, duplicity will
only need an additional 1 KB (or so) to make an incremental backup of that
file, and you only use a total of 1 GB + 1 KB of backup storage, while
maintaining the ability to recover either the old or the new version.
With the system you describe, it sounds like either you lose the old
version, or you use 2 GB of backup storage.

Note this does not apply if files are moved: duplicity currently doesn't
detect moved files, and treats them as if the file was deleted and an
unrelated file created under a new name.  If you have a 1 GB file and then
rename it, even if you don't change the contents, duplicity will use 2 GB
of backup storage.  It's not clear how the other system you describe would
handle that.

> Is this scheme popular?

I've never heard of it before.

> Is it useful in comparison to local-rsync

They seem very different.

--
Nate Eldredge
[hidden email]


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: How to use duplicity for local backup

duplicity-talk mailing list
On 2020-04-27 15:51, Nate Eldredge wrote:

> On Fri, 24 Apr 2020, MRob via Duplicity-talk wrote:
>
>> Hello, I want to ask opinion about backup tactic please.
>>
>> I found a backup system that using only rsync to a local server. A few
>> scripts run on the server that create tree of historical directrories
>> that are filled with hard links to the main rsync target. I never saw
>> this but my understand of the benefit:
>> * saving space for historical backups because they only use inodes
>> * simple scheme to use rsync and ln
>> * But am I correct, to understand if a file *change* then the
>> historical data is lost so its a shortcoming of the scheme.
>
> Well, it could work either way.  Maybe it only keeps the latest
> version of files that change, maybe it keeps all past versions.  I
> can't tell from your description alone.  It'd help if you would
> provide a link to the system you are talking about.

Custom scripts thats why I want to replace it. (1) rsync to local backup
server (2) "cp -l" on backup server to making incrementals

> Duplicity uses the rdiff algorithm to only track the changes in a
> file's contents.  So if you have a 1 GB file and you add 1 KB,
> duplicity will only need an additional 1 KB (or so) to make an
> incremental backup of that file, and you only use a total of 1 GB + 1
> KB of backup storage, while maintaining the ability to recover either
> the old or the new version. With the system you describe, it sounds
> like either you lose the old version, or you use 2 GB of backup
> storage.

Thank you. Is correct to say duplicity will use more CPU on source
server because creating tarballs and doing diffs while plain rsync'ing
may only cost more network traffic for sending uncompressed files?

Sound like Edgar's idea to use "rsync --link-dest" for incrementals on
backup server is best idea

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk