Re: Problems with backup after using rdiff-backup-delete.py

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

Brian Gupta-2
So I work with OP, and am trying to sort out our path forward.

We have been using rdiff-backup for over 7 years, and we generally
keep backups indefinitely. A few years ago or so, we had to remove
certain PII data from our backups, for contractual reasons. The
ability to do these selective deletions is now an ongoing requirement.

We contracted with rdiff-backup's primary maintainer at the time,
sol1, to add this functionality for us, and they offered to write it
as an open source contribution, as they said it was a very common
request. When they did this work for us, there was no indication that
it wouldn't be production ready. (We paid extra for a thorough
validation.)

This is something I understand has now changed, as they are no longer
the primary maintainer and are disparaging the tool on a public
mailing list. [1] No bad blood here, we understand that open source is
hard, and companies' priorities change.

At the end of the day, we need a file-based backup system, that is
relatively space efficient, supports remote network backups, supports
frequent backups, allows indefinite storage, and allows us to
completely purge directories.

Ideally we don't want to change backup systems. Can rdiff-backup be
this system, with a little extra dev effort?

If so, would anyone be willing to help us sort this out on a contract
basis? Need:
1) fix current broken backup state
2) fix existing delete tool, or write a new one. In either case tool
should be up to the standards of the rdiff-backup community, and
considered production ready.

Thanks,
Brian Gupta

[1] - https://lists.nongnu.org/archive/html/rdiff-backup-users/2019-11/msg00003.html

Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

Patrik Dufresne-2
Hello Brian,

I'm the main developer for rdiffweb <https://github.com/ikus060/rdiffweb>
and Minarca <https://github.com/ikus060/minarca>, and more recently I'm
getting involved in rdiff-backup development with small contributions.
Through my software company, I'm already offering professional support for
few customers for rdiffweb/Minarca and I do extends the support to
rdiff-backup. If you contact me directly [hidden email], we could
further discuss how we can help you.

On Fri, Nov 29, 2019 at 12:06 PM Brian Gupta <[hidden email]>
wrote:

> So I work with OP, and am trying to sort out our path forward.
>
> We have been using rdiff-backup for over 7 years, and we generally
> keep backups indefinitely. A few years ago or so, we had to remove
> certain PII data from our backups, for contractual reasons. The
> ability to do these selective deletions is now an ongoing requirement.
>
> We contracted with rdiff-backup's primary maintainer at the time,
> sol1, to add this functionality for us, and they offered to write it
> as an open source contribution, as they said it was a very common
> request. When they did this work for us, there was no indication that
> it wouldn't be production ready. (We paid extra for a thorough
> validation.)
>
> This is something I understand has now changed, as they are no longer
> the primary maintainer and are disparaging the tool on a public
> mailing list. [1] No bad blood here, we understand that open source is
> hard, and companies' priorities change.
>
> At the end of the day, we need a file-based backup system, that is
> relatively space efficient, supports remote network backups, supports
> frequent backups, allows indefinite storage, and allows us to
> completely purge directories.
>
> Ideally we don't want to change backup systems. Can rdiff-backup be
> this system, with a little extra dev effort?
>
> If so, would anyone be willing to help us sort this out on a contract
> basis? Need:
> 1) fix current broken backup state
> 2) fix existing delete tool, or write a new one. In either case tool
> should be up to the standards of the rdiff-backup community, and
> considered production ready.
>
> Thanks,
> Brian Gupta
>
> [1] -
> https://lists.nongnu.org/archive/html/rdiff-backup-users/2019-11/msg00003.html
>
>

--
--
Patrik Dufresne Service Logiciel inc.
http://www.patrikdufresne.com <http://patrikdufresne.com/>/
514-971-6442
130 rue Doris
St-Colomban, QC J5K 1T9
Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

Robert Nichols-2
In reply to this post by Brian Gupta-2
On 11/29/19 11:00 AM, Brian Gupta wrote:

> So I work with OP, and am trying to sort out our path forward.
>
> We have been using rdiff-backup for over 7 years, and we generally
> keep backups indefinitely. A few years ago or so, we had to remove
> certain PII data from our backups, for contractual reasons. The
> ability to do these selective deletions is now an ongoing requirement.
>
> We contracted with rdiff-backup's primary maintainer at the time,
> sol1, to add this functionality for us, and they offered to write it
> as an open source contribution, as they said it was a very common
> request. When they did this work for us, there was no indication that
> it wouldn't be production ready. (We paid extra for a thorough
> validation.)
>
> This is something I understand has now changed, as they are no longer
> the primary maintainer and are disparaging the tool on a public
> mailing list. [1] No bad blood here, we understand that open source is
> hard, and companies' priorities change.
>
> At the end of the day, we need a file-based backup system, that is
> relatively space efficient, supports remote network backups, supports
> frequent backups, allows indefinite storage, and allows us to
> completely purge directories.
>
> Ideally we don't want to change backup systems. Can rdiff-backup be
> this system, with a little extra dev effort?
>
> If so, would anyone be willing to help us sort this out on a contract
> basis? Need:
> 1) fix current broken backup state
> 2) fix existing delete tool, or write a new one. In either case tool
> should be up to the standards of the rdiff-backup community, and
> considered production ready.
>
> Thanks,
> Brian Gupta
>
> [1] - https://lists.nongnu.org/archive/html/rdiff-backup-users/2019-11/msg00003.html

I do have a fairly massive (~640 lines, ~550 non-comment) shell script that I agree is ugly, but which has worked fine for me for quite a few years (last modified 7 years ago) and successfully purges both specific files and entire subtrees from an rdiff-backup archive. The principal limitations are that it does not support long_filename_data (refuses to run if any found) and does not handle Mac resource fork metadata (ignores it). I'll send it to anyone who wants to try it, or I can put it up for comment and/or ridicule if someone would tell me where and how to do that. All I can guarantee is that if it eats any babies, those will be the first babies it has ever eaten.

--
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.


Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

Patrik Dufresne
Hi Robert Nichols,

Your contribution is welcome. There is a folder with user's contribution in
rdiff-backup repositories. We could probably leave it their with some user
warning. So either submit a Pull Request in github or send it here as
attachment and will submit it for you.

https://github.com/rdiff-backup/rdiff-backup/tree/master/tools/misc


On Fri, Nov 29, 2019 at 1:54 PM Robert Nichols <[hidden email]>
wrote:

> On 11/29/19 11:00 AM, Brian Gupta wrote:
> > So I work with OP, and am trying to sort out our path forward.
> >
> > We have been using rdiff-backup for over 7 years, and we generally
> > keep backups indefinitely. A few years ago or so, we had to remove
> > certain PII data from our backups, for contractual reasons. The
> > ability to do these selective deletions is now an ongoing requirement.
> >
> > We contracted with rdiff-backup's primary maintainer at the time,
> > sol1, to add this functionality for us, and they offered to write it
> > as an open source contribution, as they said it was a very common
> > request. When they did this work for us, there was no indication that
> > it wouldn't be production ready. (We paid extra for a thorough
> > validation.)
> >
> > This is something I understand has now changed, as they are no longer
> > the primary maintainer and are disparaging the tool on a public
> > mailing list. [1] No bad blood here, we understand that open source is
> > hard, and companies' priorities change.
> >
> > At the end of the day, we need a file-based backup system, that is
> > relatively space efficient, supports remote network backups, supports
> > frequent backups, allows indefinite storage, and allows us to
> > completely purge directories.
> >
> > Ideally we don't want to change backup systems. Can rdiff-backup be
> > this system, with a little extra dev effort?
> >
> > If so, would anyone be willing to help us sort this out on a contract
> > basis? Need:
> > 1) fix current broken backup state
> > 2) fix existing delete tool, or write a new one. In either case tool
> > should be up to the standards of the rdiff-backup community, and
> > considered production ready.
> >
> > Thanks,
> > Brian Gupta
> >
> > [1] -
> https://lists.nongnu.org/archive/html/rdiff-backup-users/2019-11/msg00003.html
>
> I do have a fairly massive (~640 lines, ~550 non-comment) shell script
> that I agree is ugly, but which has worked fine for me for quite a few
> years (last modified 7 years ago) and successfully purges both specific
> files and entire subtrees from an rdiff-backup archive. The principal
> limitations are that it does not support long_filename_data (refuses to run
> if any found) and does not handle Mac resource fork metadata (ignores it).
> I'll send it to anyone who wants to try it, or I can put it up for comment
> and/or ridicule if someone would tell me where and how to do that. All I
> can guarantee is that if it eats any babies, those will be the first babies
> it has ever eaten.
>
> --
> Bob Nichols     "NOSPAM" is really part of my email address.
>                  Do NOT delete it.
>
>
>

--
Patrik Dufresne <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

ewl+rdiffbackup
In reply to this post by Patrik Dufresne-2
Hello,

if you plan to make a script an official part of the project, please
have a look at https://github.com/rdiff-backup/rdiff-backup/issues/187 
and the criteria highlighted for such a step.

KR, Eric

On 29/11/2019 18:16, Patrik Dufresne wrote:

> Hello Brian,
>
> I'm the main developer for rdiffweb <https://github.com/ikus060/rdiffweb>
> and Minarca <https://github.com/ikus060/minarca>, and more recently I'm
> getting involved in rdiff-backup development with small contributions.
> Through my software company, I'm already offering professional support for
> few customers for rdiffweb/Minarca and I do extends the support to
> rdiff-backup. If you contact me directly [hidden email], we could
> further discuss how we can help you.
>
> On Fri, Nov 29, 2019 at 12:06 PM Brian Gupta <[hidden email]>
> wrote:
>
>> So I work with OP, and am trying to sort out our path forward.
>>
>> We have been using rdiff-backup for over 7 years, and we generally
>> keep backups indefinitely. A few years ago or so, we had to remove
>> certain PII data from our backups, for contractual reasons. The
>> ability to do these selective deletions is now an ongoing requirement.
>>
>> We contracted with rdiff-backup's primary maintainer at the time,
>> sol1, to add this functionality for us, and they offered to write it
>> as an open source contribution, as they said it was a very common
>> request. When they did this work for us, there was no indication that
>> it wouldn't be production ready. (We paid extra for a thorough
>> validation.)
>>
>> This is something I understand has now changed, as they are no longer
>> the primary maintainer and are disparaging the tool on a public
>> mailing list. [1] No bad blood here, we understand that open source is
>> hard, and companies' priorities change.
>>
>> At the end of the day, we need a file-based backup system, that is
>> relatively space efficient, supports remote network backups, supports
>> frequent backups, allows indefinite storage, and allows us to
>> completely purge directories.
>>
>> Ideally we don't want to change backup systems. Can rdiff-backup be
>> this system, with a little extra dev effort?
>>
>> If so, would anyone be willing to help us sort this out on a contract
>> basis? Need:
>> 1) fix current broken backup state
>> 2) fix existing delete tool, or write a new one. In either case tool
>> should be up to the standards of the rdiff-backup community, and
>> considered production ready.
>>
>> Thanks,
>> Brian Gupta
>>
>> [1] -
>> https://lists.nongnu.org/archive/html/rdiff-backup-users/2019-11/msg00003.html
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

Robert Nichols-2
In reply to this post by Patrik Dufresne
On 11/29/19 1:10 PM, Patrik Dufresne wrote:

> Hi Robert Nichols,
>
> Your contribution is welcome. There is a folder with user's contribution in
> rdiff-backup repositories. We could probably leave it their with some user
> warning. So either submit a Pull Request in github or send it here as
> attachment and will submit it for you.
>
> https://github.com/rdiff-backup/rdiff-backup/tree/master/tools/misc
>
>
> On Fri, Nov 29, 2019 at 1:54 PM Robert Nichols <[hidden email]>
> wrote:
>>
>> I do have a fairly massive (~640 lines, ~550 non-comment) shell script
>> that I agree is ugly, but which has worked fine for me for quite a few
>> years (last modified 7 years ago) and successfully purges both specific
>> files and entire subtrees from an rdiff-backup archive. The principal
>> limitations are that it does not support long_filename_data (refuses to run
>> if any found) and does not handle Mac resource fork metadata (ignores it).
>> I'll send it to anyone who wants to try it, or I can put it up for comment
>> and/or ridicule if someone would tell me where and how to do that. All I
>> can guarantee is that if it eats any babies, those will be the first babies
>> it has ever eaten.
>>
>> --
>> Bob Nichols     "NOSPAM" is really part of my email address.
>>                   Do NOT delete it.

I've sent the script in private email to Patrik rather than force all the
list recipients to download it. Sorry about the delay. I've been very busy
lately.

--
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.


Reply | Threaded
Open this post in threaded view
|

Re: Problems with backup after using rdiff-backup-delete.py

Patrik Dufresne
I've submit your changes into a Pull Request here:
https://github.com/rdiff-backup/rdiff-backup/pull/205

On Wed, Dec 4, 2019 at 11:13 PM Robert Nichols <[hidden email]>
wrote:

> On 11/29/19 1:10 PM, Patrik Dufresne wrote:
> > Hi Robert Nichols,
> >
> > Your contribution is welcome. There is a folder with user's contribution
> in
> > rdiff-backup repositories. We could probably leave it their with some
> user
> > warning. So either submit a Pull Request in github or send it here as
> > attachment and will submit it for you.
> >
> > https://github.com/rdiff-backup/rdiff-backup/tree/master/tools/misc
> >
> >
> > On Fri, Nov 29, 2019 at 1:54 PM Robert Nichols <
> [hidden email]>
> > wrote:
> >>
> >> I do have a fairly massive (~640 lines, ~550 non-comment) shell script
> >> that I agree is ugly, but which has worked fine for me for quite a few
> >> years (last modified 7 years ago) and successfully purges both specific
> >> files and entire subtrees from an rdiff-backup archive. The principal
> >> limitations are that it does not support long_filename_data (refuses to
> run
> >> if any found) and does not handle Mac resource fork metadata (ignores
> it).
> >> I'll send it to anyone who wants to try it, or I can put it up for
> comment
> >> and/or ridicule if someone would tell me where and how to do that. All I
> >> can guarantee is that if it eats any babies, those will be the first
> babies
> >> it has ever eaten.
> >>
> >> --
> >> Bob Nichols     "NOSPAM" is really part of my email address.
> >>                   Do NOT delete it.
>
> I've sent the script in private email to Patrik rather than force all the
> list recipients to download it. Sorry about the delay. I've been very busy
> lately.
>
> --
> Bob Nichols     "NOSPAM" is really part of my email address.
>                  Do NOT delete it.
>
>
>

--
Patrik Dufresne <[hidden email]>