Direct usage via Python (i.e. import duplicity)?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Direct usage via Python (i.e. import duplicity)?

duplicity-talk mailing list
Hello,

I'm really interested in using Duplicity to replace my current homegrown backup system (based on full-file synchronization to S3 for important things and rsnapshot local backups for less-important). I was really happy to learn that Duplicity is written in Python, as that's my go-to language.

I was hoping that since Duplicity is written in Python, I could avoid the overhead of my current custom backup wrapper (for rsnapshot) that has to build command lines, run via subprocess, and then parse the output to get the statistics I care about (number and size of changed files, transfer time, etc.). However, even though I found a few Duplicity wrappers or front-ends written in Python, they all seem to execute duplicity via subprocess.

I was wondering if anyone has experience driving Duplicity natively via a Python wrapper by making the proper function calls (and then... I suppose patching some parts, since it seems like the status output/statistics isn't surfaced in a consumable way)? Is this something that's worth it for me to try, or will the mostly function-based design prohibit it?

Thanks,
Jason Antman


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Direct usage via Python (i.e. import duplicity)?

duplicity-talk mailing list
Hi,

All it would take is a bit of refactoring to do what you want.  A number of Python utils are set up with a minimal start script and the rest in the module.

If you have the time and the desire, we'd appreciate the work.

...Thanks,
...Ken


On Mon, Mar 26, 2018 at 5:01 PM, Jason Antman via Duplicity-talk <[hidden email]> wrote:
Hello,

I'm really interested in using Duplicity to replace my current homegrown backup system (based on full-file synchronization to S3 for important things and rsnapshot local backups for less-important). I was really happy to learn that Duplicity is written in Python, as that's my go-to language.

I was hoping that since Duplicity is written in Python, I could avoid the overhead of my current custom backup wrapper (for rsnapshot) that has to build command lines, run via subprocess, and then parse the output to get the statistics I care about (number and size of changed files, transfer time, etc.). However, even though I found a few Duplicity wrappers or front-ends written in Python, they all seem to execute duplicity via subprocess.

I was wondering if anyone has experience driving Duplicity natively via a Python wrapper by making the proper function calls (and then... I suppose patching some parts, since it seems like the status output/statistics isn't surfaced in a consumable way)? Is this something that's worth it for me to try, or will the mostly function-based design prohibit it?

Thanks,
Jason Antman


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Direct usage via Python (i.e. import duplicity)?

duplicity-talk mailing list
Hello Jason,

On Mar 30 2018, at 12:23 pm, Kenneth Loafman via Duplicity-talk <[hidden email]> wrote:
On Mon, Mar 26, 2018 at 5:01 PM, Jason Antman via Duplicity-talk <[hidden email]> wrote:

I was wondering if anyone has experience driving Duplicity natively via a Python wrapper by making the proper function calls (and then... I suppose patching some parts, since it seems like the status output/statistics isn't surfaced in a consumable way)? Is this something that's worth it for me to try, or will the mostly function-based design prohibit it?

All it would take is a bit of refactoring to do what you want.  A number of Python utils are set up with a minimal start script and the rest in the module.

If you have the time and the desire, we'd appreciate the work.

Just to reiterate Kenneth's point, we would love to see another contributor working on this aspect of duplicity. The code base was initially written a very long time ago on an early Python version and there are a few aspects that none of us would do the same way now.

I have also looked at refactoring the duplicity code to do the substantive work in a library and make the commandline a user of that (with duply another potential user). I intend to move the command line processing from optparse (deprecated) to argparse and argparse assumes a more modern approach.

As a first step, which may be enough for what you want, I would make a function that consumes something very similar to the existing commandline (after processing with argparse) and returns the statistics. That reduces the chance that a future refactoring will stop your code working, as while we make no guarantees around library stability, we keep the core commandline interface pretty stable. You would need to move the existing globals to be defaults to that function, but all of that would improve the existing code.

We need to put together some formal contributing documentation, but in the meantime:
https://lists.gnu.org/archive/html/duplicity-talk/2017-06/msg00000.html is an overview of the initial flow through the existing code, which may help you.
My comment here: https://answers.launchpad.net/duplicity/+question/660463 (comment 8) also gives some general tips to contributing to the project. Testing is going to be very important if you are replacing the overall application flow.

I would be happy to work with you on this, though I am pretty tight for time at the moment, so may take a little while to turn around comments/review.

Any questions, please ask.

Kind regards,

Aaron

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk