Why do I have a 19GB ~/.cache/duplicity folder?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Why do I have a 19GB ~/.cache/duplicity folder?

Timothee Besset-2
I'm finding that some systems that were backed up by duplicity are
running out of disk space after I upgraded the duplicity version. More
specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity folder?

I can't afford to have 20GB of disk occupied by my backup software. I
haven't found much information on the nature of that directory, except
that it's an addition in 0.6? Should it really be growing that big? Is
there a limit to how much it'll grow? Is it a case of backup biting it's
own tail like we had backups of /tmp trying to backup duplicity's own
temporary files at one point?

Best,
TTimo



_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Piotr Karbowski
Same here, but I have small .cache/duplicity dirs. What exacly is it
and why I need it? Alo, if I backup homedir with duplicity, duplicity
will default exclude ~/.cache/duplicity or should I do it?


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Cyril Russo

Le 10/02/2010 08:39, Piotr Karbowski a écrit :
> Same here, but I have small .cache/duplicity dirs. What exacly is it
> and why I need it? Alo, if I backup homedir with duplicity, duplicity
> will default exclude ~/.cache/duplicity or should I do it?
>
>    
Hi,

It's the archive used to restore upon network failure. Please have a
look at archive options if you want to disable it, but you'll loose
resumable backup.
Best regards,

Cyril



_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

edgar.soldin
In reply to this post by Piotr Karbowski
you will have to manually exclude it ..

it holds your backup chains index files (table of contents of the backup
repository). Caching them locally accelerates options like status and
incremental backup and others. These operations need to know what is
already backed up to work. If they are cached they do not need to be
transferred and decrypted every time again and again.

.. ede

On 10.02.2010 08:39, Piotr Karbowski wrote:
> Same here, but I have small .cache/duplicity dirs. What exacly is it
> and why I need it? Alo, if I backup homedir with duplicity, duplicity
> will default exclude ~/.cache/duplicity or should I do it?
>
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Tim Riemenschneider-2
In reply to this post by Timothee Besset-2
Timothee Besset schrieb:

> I'm finding that some systems that were backed up by duplicity are
> running out of disk space after I upgraded the duplicity version. More
> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity folder?
>
> I can't afford to have 20GB of disk occupied by my backup software. I
> haven't found much information on the nature of that directory, except
> that it's an addition in 0.6? Should it really be growing that big? Is
> there a limit to how much it'll grow? Is it a case of backup biting it's
> own tail like we had backups of /tmp trying to backup duplicity's own
> temporary files at one point?
>
>  
The problem seems to be, that duplicity does not cleanup signature-files
when removing old backups, so the signatures occupy more and more space.

To reproduce, I did some backups (locally), some with a forced full
backup and some incementals. Afterwards the backup looks like this:

$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 2 secondary backup chains.
Secondary chain 1 of 2:
-------------------------
Chain start time: Wed Feb 10 16:50:16 2010
Chain end time: Wed Feb 10 16:50:44 2010
Number of contained backup sets: 4
Total number of contained volumes: 4
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:50:16 2010                 1
         Incremental         Wed Feb 10 16:50:36 2010                 1
         Incremental         Wed Feb 10 16:50:41 2010                 1
         Incremental         Wed Feb 10 16:50:44 2010                 1
-------------------------

Secondary chain 2 of 2:
-------------------------
Chain start time: Wed Feb 10 16:51:01 2010
Chain end time: Wed Feb 10 16:51:39 2010
Number of contained backup sets: 4
Total number of contained volumes: 4
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:51:01 2010                 1
         Incremental         Wed Feb 10 16:51:36 2010                 1
         Incremental         Wed Feb 10 16:51:38 2010                 1
         Incremental         Wed Feb 10 16:51:39 2010                 1
-------------------------


Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:51:46 2010                 1
         Incremental         Wed Feb 10 16:52:04 2010                 1
-------------------------
No orphaned or incomplete backup sets found.
$ ls /tmp/duptest/
duplicity-full.20100210T155016Z.manifest.gpg
duplicity-full.20100210T155016Z.vol1.difftar.gpg
duplicity-full.20100210T155101Z.manifest.gpg
duplicity-full.20100210T155101Z.vol1.difftar.gpg
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg

Now I remove the old backups:

$ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
file:///tmp/duptest
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb 10 16:51:46 2010
Deleting backup sets at times:
Wed Feb 10 16:50:16 2010
Wed Feb 10 16:50:36 2010
Wed Feb 10 16:50:41 2010
Wed Feb 10 16:50:44 2010
Wed Feb 10 16:51:01 2010
Wed Feb 10 16:51:36 2010
Wed Feb 10 16:51:38 2010
Wed Feb 10 16:51:39 2010
Warning, found the following local orphaned signature files:
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz

Note the "locally orphaned files".
When I look at the status:

$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Synchronizing remote metadata to local cache...
Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:51:46 2010                 1
         Incremental         Wed Feb 10 16:52:04 2010                 1
-------------------------
No orphaned or incomplete backup sets found.

Note that duplicity did fetch the signatures of the 2 old (deleted!)
full-backups

When looking at the target, it becomes clear that "duplicity remove..."
did only remove the data (*vol*difftar-files), not the obsoleted signatures:

$ ls /tmp/duptest/
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg

These files are also present (in decrypted form) in the cache-dir. And
they stay there (or are refetched from the backup-target).
Only workaround: delete the unneeded signature-files on the target, then
duplicity also removes them from the cache (but be carefull to only
remove old and unneeded files):
$ cd /tmp/duptest/
:/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
duplicity-full-signatures.20100210T155101Z.sigtar.gpg
duplicity-new-signatures.20100210T1550*
duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
duplicity-new-signatures.20100210T15513*
:/tmp/duptest$ ls
duplicity-full.20100210T155146Z.manifest.gpg
duplicity-full.20100210T155146Z.vol1.difftar.gpg
duplicity-full-signatures.20100210T155146Z.sigtar.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
:/tmp/duptest$ cd -
~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
Synchronizing remote metadata to local cache...
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
(not authoritative at backend).
Deleting local
/home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
(not authoritative at backend).
Last full backup date: Wed Feb 10 16:51:46 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb 10 16:51:46 2010
Chain end time: Wed Feb 10 16:52:04 2010
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb 10 16:51:46 2010                 1
         Incremental         Wed Feb 10 16:52:04 2010                 1
-------------------------
No orphaned or incomplete backup sets found.

(Of course, one has to do full-backups from time to time in the first
place, when you only ever do incrementals, ALL signature-files are still
needed)

cu
Tim



_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Kenneth Loafman
Tim,

Good testing!

Please report this bug at https://bugs.launchpad.net/duplicity

...Thanks,
...Ken

Tim Riemenschneider wrote:

> Timothee Besset schrieb:
>> I'm finding that some systems that were backed up by duplicity are
>> running out of disk space after I upgraded the duplicity version. More
>> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity folder?
>>
>> I can't afford to have 20GB of disk occupied by my backup software. I
>> haven't found much information on the nature of that directory, except
>> that it's an addition in 0.6? Should it really be growing that big? Is
>> there a limit to how much it'll grow? Is it a case of backup biting it's
>> own tail like we had backups of /tmp trying to backup duplicity's own
>> temporary files at one point?
>>
>>  
> The problem seems to be, that duplicity does not cleanup signature-files
> when removing old backups, so the signatures occupy more and more space.
>
> To reproduce, I did some backups (locally), some with a forced full
> backup and some incementals. Afterwards the backup looks like this:
>
> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Wed Feb 10 16:51:46 2010
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>
> Found 2 secondary backup chains.
> Secondary chain 1 of 2:
> -------------------------
> Chain start time: Wed Feb 10 16:50:16 2010
> Chain end time: Wed Feb 10 16:50:44 2010
> Number of contained backup sets: 4
> Total number of contained volumes: 4
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:50:16 2010                 1
>          Incremental         Wed Feb 10 16:50:36 2010                 1
>          Incremental         Wed Feb 10 16:50:41 2010                 1
>          Incremental         Wed Feb 10 16:50:44 2010                 1
> -------------------------
>
> Secondary chain 2 of 2:
> -------------------------
> Chain start time: Wed Feb 10 16:51:01 2010
> Chain end time: Wed Feb 10 16:51:39 2010
> Number of contained backup sets: 4
> Total number of contained volumes: 4
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:01 2010                 1
>          Incremental         Wed Feb 10 16:51:36 2010                 1
>          Incremental         Wed Feb 10 16:51:38 2010                 1
>          Incremental         Wed Feb 10 16:51:39 2010                 1
> -------------------------
>
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Feb 10 16:51:46 2010
> Chain end time: Wed Feb 10 16:52:04 2010
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:46 2010                 1
>          Incremental         Wed Feb 10 16:52:04 2010                 1
> -------------------------
> No orphaned or incomplete backup sets found.
> $ ls /tmp/duptest/
> duplicity-full.20100210T155016Z.manifest.gpg
> duplicity-full.20100210T155016Z.vol1.difftar.gpg
> duplicity-full.20100210T155101Z.manifest.gpg
> duplicity-full.20100210T155101Z.vol1.difftar.gpg
> duplicity-full.20100210T155146Z.manifest.gpg
> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
> duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
> duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
> duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
> duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
> duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
> duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
> duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
> duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
> duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
> duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
> duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>
> Now I remove the old backups:
>
> $ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
> file:///tmp/duptest
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Wed Feb 10 16:51:46 2010
> Deleting backup sets at times:
> Wed Feb 10 16:50:16 2010
> Wed Feb 10 16:50:36 2010
> Wed Feb 10 16:50:41 2010
> Wed Feb 10 16:50:44 2010
> Wed Feb 10 16:51:01 2010
> Wed Feb 10 16:51:36 2010
> Wed Feb 10 16:51:38 2010
> Wed Feb 10 16:51:39 2010
> Warning, found the following local orphaned signature files:
> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
>
> Note the "locally orphaned files".
> When I look at the status:
>
> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> Synchronizing remote metadata to local cache...
> Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
> Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
> Last full backup date: Wed Feb 10 16:51:46 2010
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>
> Found 0 secondary backup chains.
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Feb 10 16:51:46 2010
> Chain end time: Wed Feb 10 16:52:04 2010
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:46 2010                 1
>          Incremental         Wed Feb 10 16:52:04 2010                 1
> -------------------------
> No orphaned or incomplete backup sets found.
>
> Note that duplicity did fetch the signatures of the 2 old (deleted!)
> full-backups
>
> When looking at the target, it becomes clear that "duplicity remove..."
> did only remove the data (*vol*difftar-files), not the obsoleted signatures:
>
> $ ls /tmp/duptest/
> duplicity-full.20100210T155146Z.manifest.gpg
> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>
> These files are also present (in decrypted form) in the cache-dir. And
> they stay there (or are refetched from the backup-target).
> Only workaround: delete the unneeded signature-files on the target, then
> duplicity also removes them from the cache (but be carefull to only
> remove old and unneeded files):
> $ cd /tmp/duptest/
> :/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> duplicity-new-signatures.20100210T1550*
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> duplicity-new-signatures.20100210T15513*
> :/tmp/duptest$ ls
> duplicity-full.20100210T155146Z.manifest.gpg
> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> :/tmp/duptest$ cd -
> ~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> Synchronizing remote metadata to local cache...
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
> (not authoritative at backend).
> Last full backup date: Wed Feb 10 16:51:46 2010
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>
> Found 0 secondary backup chains.
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Feb 10 16:51:46 2010
> Chain end time: Wed Feb 10 16:52:04 2010
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:46 2010                 1
>          Incremental         Wed Feb 10 16:52:04 2010                 1
> -------------------------
> No orphaned or incomplete backup sets found.
>
> (Of course, one has to do full-backups from time to time in the first
> place, when you only ever do incrementals, ALL signature-files are still
> needed)
>
> cu
> Tim
>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc (268 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Timothee Besset-2
In reply to this post by Tim Riemenschneider-2
Thank you for the various replies. I am passing --extra-clean to my
cleanup statements, which seems to take care of the problem. I have
landed the extra parameter in my management script [1], but it sounds to
me like this should be the default behavior?

If it turns out that I really have to do away with the cache and
sacrifice resuming interrupted backups then I also know how to turn that
off completely from those mails, so thanks.

TTimo

[1] dupinanny - http://github.com/TTimo/dupinanny

Tim Riemenschneider wrote:

> Timothee Besset schrieb:
>  
>> I'm finding that some systems that were backed up by duplicity are
>> running out of disk space after I upgraded the duplicity version. More
>> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity folder?
>>
>> I can't afford to have 20GB of disk occupied by my backup software. I
>> haven't found much information on the nature of that directory, except
>> that it's an addition in 0.6? Should it really be growing that big? Is
>> there a limit to how much it'll grow? Is it a case of backup biting it's
>> own tail like we had backups of /tmp trying to backup duplicity's own
>> temporary files at one point?
>>
>>  
>>    
> The problem seems to be, that duplicity does not cleanup signature-files
> when removing old backups, so the signatures occupy more and more space.
>
> To reproduce, I did some backups (locally), some with a forced full
> backup and some incementals. Afterwards the backup looks like this:
>
> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Wed Feb 10 16:51:46 2010
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>
> Found 2 secondary backup chains.
> Secondary chain 1 of 2:
> -------------------------
> Chain start time: Wed Feb 10 16:50:16 2010
> Chain end time: Wed Feb 10 16:50:44 2010
> Number of contained backup sets: 4
> Total number of contained volumes: 4
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:50:16 2010                 1
>          Incremental         Wed Feb 10 16:50:36 2010                 1
>          Incremental         Wed Feb 10 16:50:41 2010                 1
>          Incremental         Wed Feb 10 16:50:44 2010                 1
> -------------------------
>
> Secondary chain 2 of 2:
> -------------------------
> Chain start time: Wed Feb 10 16:51:01 2010
> Chain end time: Wed Feb 10 16:51:39 2010
> Number of contained backup sets: 4
> Total number of contained volumes: 4
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:01 2010                 1
>          Incremental         Wed Feb 10 16:51:36 2010                 1
>          Incremental         Wed Feb 10 16:51:38 2010                 1
>          Incremental         Wed Feb 10 16:51:39 2010                 1
> -------------------------
>
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Feb 10 16:51:46 2010
> Chain end time: Wed Feb 10 16:52:04 2010
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:46 2010                 1
>          Incremental         Wed Feb 10 16:52:04 2010                 1
> -------------------------
> No orphaned or incomplete backup sets found.
> $ ls /tmp/duptest/
> duplicity-full.20100210T155016Z.manifest.gpg
> duplicity-full.20100210T155016Z.vol1.difftar.gpg
> duplicity-full.20100210T155101Z.manifest.gpg
> duplicity-full.20100210T155101Z.vol1.difftar.gpg
> duplicity-full.20100210T155146Z.manifest.gpg
> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
> duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
> duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
> duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
> duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
> duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
> duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
> duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
> duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
> duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
> duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
> duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>
> Now I remove the old backups:
>
> $ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
> file:///tmp/duptest
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Wed Feb 10 16:51:46 2010
> Deleting backup sets at times:
> Wed Feb 10 16:50:16 2010
> Wed Feb 10 16:50:36 2010
> Wed Feb 10 16:50:41 2010
> Wed Feb 10 16:50:44 2010
> Wed Feb 10 16:51:01 2010
> Wed Feb 10 16:51:36 2010
> Wed Feb 10 16:51:38 2010
> Wed Feb 10 16:51:39 2010
> Warning, found the following local orphaned signature files:
> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
>
> Note the "locally orphaned files".
> When I look at the status:
>
> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> Synchronizing remote metadata to local cache...
> Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
> Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
> Last full backup date: Wed Feb 10 16:51:46 2010
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>
> Found 0 secondary backup chains.
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Feb 10 16:51:46 2010
> Chain end time: Wed Feb 10 16:52:04 2010
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:46 2010                 1
>          Incremental         Wed Feb 10 16:52:04 2010                 1
> -------------------------
> No orphaned or incomplete backup sets found.
>
> Note that duplicity did fetch the signatures of the 2 old (deleted!)
> full-backups
>
> When looking at the target, it becomes clear that "duplicity remove..."
> did only remove the data (*vol*difftar-files), not the obsoleted signatures:
>
> $ ls /tmp/duptest/
> duplicity-full.20100210T155146Z.manifest.gpg
> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>
> These files are also present (in decrypted form) in the cache-dir. And
> they stay there (or are refetched from the backup-target).
> Only workaround: delete the unneeded signature-files on the target, then
> duplicity also removes them from the cache (but be carefull to only
> remove old and unneeded files):
> $ cd /tmp/duptest/
> :/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> duplicity-new-signatures.20100210T1550*
> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> duplicity-new-signatures.20100210T15513*
> :/tmp/duptest$ ls
> duplicity-full.20100210T155146Z.manifest.gpg
> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> :/tmp/duptest$ cd -
> ~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> Synchronizing remote metadata to local cache...
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> (not authoritative at backend).
> Deleting local
> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
> (not authoritative at backend).
> Last full backup date: Wed Feb 10 16:51:46 2010
> Collection Status
> -----------------
> Connecting with backend: LocalBackend
> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>
> Found 0 secondary backup chains.
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Feb 10 16:51:46 2010
> Chain end time: Wed Feb 10 16:52:04 2010
> Number of contained backup sets: 2
> Total number of contained volumes: 2
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Feb 10 16:51:46 2010                 1
>          Incremental         Wed Feb 10 16:52:04 2010                 1
> -------------------------
> No orphaned or incomplete backup sets found.
>
> (Of course, one has to do full-backups from time to time in the first
> place, when you only ever do incrementals, ALL signature-files are still
> needed)
>
> cu
> Tim
>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>  



_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

edgar.soldin
In reply to this post by Kenneth Loafman
Yeah, well done .. please add a ticket at launchpad.. ede

On 10.02.2010 17:22, Kenneth Loafman wrote:

> Tim,
>
> Good testing!
>
> Please report this bug at https://bugs.launchpad.net/duplicity
>
> ...Thanks,
> ...Ken
>
> Tim Riemenschneider wrote:
>> Timothee Besset schrieb:
>>> I'm finding that some systems that were backed up by duplicity are
>>> running out of disk space after I upgraded the duplicity version. More
>>> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity folder?
>>>
>>> I can't afford to have 20GB of disk occupied by my backup software. I
>>> haven't found much information on the nature of that directory, except
>>> that it's an addition in 0.6? Should it really be growing that big? Is
>>> there a limit to how much it'll grow? Is it a case of backup biting it's
>>> own tail like we had backups of /tmp trying to backup duplicity's own
>>> temporary files at one point?
>>>
>>>
>> The problem seems to be, that duplicity does not cleanup signature-files
>> when removing old backups, so the signatures occupy more and more space.
>>
>> To reproduce, I did some backups (locally), some with a forced full
>> backup and some incementals. Afterwards the backup looks like this:
>>
>> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
>> Local and Remote metadata are synchronized, no sync needed.
>> Last full backup date: Wed Feb 10 16:51:46 2010
>> Collection Status
>> -----------------
>> Connecting with backend: LocalBackend
>> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>>
>> Found 2 secondary backup chains.
>> Secondary chain 1 of 2:
>> -------------------------
>> Chain start time: Wed Feb 10 16:50:16 2010
>> Chain end time: Wed Feb 10 16:50:44 2010
>> Number of contained backup sets: 4
>> Total number of contained volumes: 4
>>   Type of backup set:                            Time:      Num volumes:
>>                  Full         Wed Feb 10 16:50:16 2010                 1
>>           Incremental         Wed Feb 10 16:50:36 2010                 1
>>           Incremental         Wed Feb 10 16:50:41 2010                 1
>>           Incremental         Wed Feb 10 16:50:44 2010                 1
>> -------------------------
>>
>> Secondary chain 2 of 2:
>> -------------------------
>> Chain start time: Wed Feb 10 16:51:01 2010
>> Chain end time: Wed Feb 10 16:51:39 2010
>> Number of contained backup sets: 4
>> Total number of contained volumes: 4
>>   Type of backup set:                            Time:      Num volumes:
>>                  Full         Wed Feb 10 16:51:01 2010                 1
>>           Incremental         Wed Feb 10 16:51:36 2010                 1
>>           Incremental         Wed Feb 10 16:51:38 2010                 1
>>           Incremental         Wed Feb 10 16:51:39 2010                 1
>> -------------------------
>>
>>
>> Found primary backup chain with matching signature chain:
>> -------------------------
>> Chain start time: Wed Feb 10 16:51:46 2010
>> Chain end time: Wed Feb 10 16:52:04 2010
>> Number of contained backup sets: 2
>> Total number of contained volumes: 2
>>   Type of backup set:                            Time:      Num volumes:
>>                  Full         Wed Feb 10 16:51:46 2010                 1
>>           Incremental         Wed Feb 10 16:52:04 2010                 1
>> -------------------------
>> No orphaned or incomplete backup sets found.
>> $ ls /tmp/duptest/
>> duplicity-full.20100210T155016Z.manifest.gpg
>> duplicity-full.20100210T155016Z.vol1.difftar.gpg
>> duplicity-full.20100210T155101Z.manifest.gpg
>> duplicity-full.20100210T155101Z.vol1.difftar.gpg
>> duplicity-full.20100210T155146Z.manifest.gpg
>> duplicity-full.20100210T155146Z.vol1.difftar.gpg
>> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
>> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
>> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
>> duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
>> duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
>> duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
>> duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
>> duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
>> duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
>> duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
>> duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
>> duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
>> duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
>> duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
>> duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
>> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>>
>> Now I remove the old backups:
>>
>> $ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
>> file:///tmp/duptest
>> Local and Remote metadata are synchronized, no sync needed.
>> Last full backup date: Wed Feb 10 16:51:46 2010
>> Deleting backup sets at times:
>> Wed Feb 10 16:50:16 2010
>> Wed Feb 10 16:50:36 2010
>> Wed Feb 10 16:50:41 2010
>> Wed Feb 10 16:50:44 2010
>> Wed Feb 10 16:51:01 2010
>> Wed Feb 10 16:51:36 2010
>> Wed Feb 10 16:51:38 2010
>> Wed Feb 10 16:51:39 2010
>> Warning, found the following local orphaned signature files:
>> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
>> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
>> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
>> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
>> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
>>
>> Note the "locally orphaned files".
>> When I look at the status:
>>
>> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
>> Synchronizing remote metadata to local cache...
>> Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
>> Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
>> Last full backup date: Wed Feb 10 16:51:46 2010
>> Collection Status
>> -----------------
>> Connecting with backend: LocalBackend
>> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>>
>> Found 0 secondary backup chains.
>>
>> Found primary backup chain with matching signature chain:
>> -------------------------
>> Chain start time: Wed Feb 10 16:51:46 2010
>> Chain end time: Wed Feb 10 16:52:04 2010
>> Number of contained backup sets: 2
>> Total number of contained volumes: 2
>>   Type of backup set:                            Time:      Num volumes:
>>                  Full         Wed Feb 10 16:51:46 2010                 1
>>           Incremental         Wed Feb 10 16:52:04 2010                 1
>> -------------------------
>> No orphaned or incomplete backup sets found.
>>
>> Note that duplicity did fetch the signatures of the 2 old (deleted!)
>> full-backups
>>
>> When looking at the target, it becomes clear that "duplicity remove..."
>> did only remove the data (*vol*difftar-files), not the obsoleted signatures:
>>
>> $ ls /tmp/duptest/
>> duplicity-full.20100210T155146Z.manifest.gpg
>> duplicity-full.20100210T155146Z.vol1.difftar.gpg
>> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
>> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
>> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
>> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
>> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>>
>> These files are also present (in decrypted form) in the cache-dir. And
>> they stay there (or are refetched from the backup-target).
>> Only workaround: delete the unneeded signature-files on the target, then
>> duplicity also removes them from the cache (but be carefull to only
>> remove old and unneeded files):
>> $ cd /tmp/duptest/
>> :/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
>> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
>> duplicity-new-signatures.20100210T1550*
>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
>> duplicity-new-signatures.20100210T15513*
>> :/tmp/duptest$ ls
>> duplicity-full.20100210T155146Z.manifest.gpg
>> duplicity-full.20100210T155146Z.vol1.difftar.gpg
>> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
>> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>> :/tmp/duptest$ cd -
>> ~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
>> Synchronizing remote metadata to local cache...
>> Deleting local
>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
>> (not authoritative at backend).
>> Deleting local
>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
>> (not authoritative at backend).
>> Deleting local
>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
>> (not authoritative at backend).
>> Deleting local
>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
>> (not authoritative at backend).
>> Deleting local
>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
>> (not authoritative at backend).
>> Deleting local
>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
>> (not authoritative at backend).
>> Deleting local
>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
>> (not authoritative at backend).
>> Deleting local
>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
>> (not authoritative at backend).
>> Last full backup date: Wed Feb 10 16:51:46 2010
>> Collection Status
>> -----------------
>> Connecting with backend: LocalBackend
>> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>>
>> Found 0 secondary backup chains.
>>
>> Found primary backup chain with matching signature chain:
>> -------------------------
>> Chain start time: Wed Feb 10 16:51:46 2010
>> Chain end time: Wed Feb 10 16:52:04 2010
>> Number of contained backup sets: 2
>> Total number of contained volumes: 2
>>   Type of backup set:                            Time:      Num volumes:
>>                  Full         Wed Feb 10 16:51:46 2010                 1
>>           Incremental         Wed Feb 10 16:52:04 2010                 1
>> -------------------------
>> No orphaned or incomplete backup sets found.
>>
>> (Of course, one has to do full-backups from time to time in the first
>> place, when you only ever do incrementals, ALL signature-files are still
>> needed)
>>
>> cu
>> Tim
>>
>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> [hidden email]
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>
>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Olivier Berger
Hi.

Has a ticket indeed been filed for this ?

Is it fixed in latest release ?

Thanks in advance.

Regards,

Le mercredi 10 février 2010 à 18:35 +0100, [hidden email] a écrit :

> Yeah, well done .. please add a ticket at launchpad.. ede
>
> On 10.02.2010 17:22, Kenneth Loafman wrote:
> > Tim,
> >
> > Good testing!
> >
> > Please report this bug at https://bugs.launchpad.net/duplicity
> >
> > ...Thanks,
> > ...Ken
> >
> > Tim Riemenschneider wrote:
> >> Timothee Besset schrieb:
> >>> I'm finding that some systems that were backed up by duplicity are
> >>> running out of disk space after I upgraded the duplicity version. More
> >>> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity folder?
> >>>
> >>> I can't afford to have 20GB of disk occupied by my backup software. I
> >>> haven't found much information on the nature of that directory, except
> >>> that it's an addition in 0.6? Should it really be growing that big? Is
> >>> there a limit to how much it'll grow? Is it a case of backup biting it's
> >>> own tail like we had backups of /tmp trying to backup duplicity's own
> >>> temporary files at one point?
> >>>
> >>>
> >> The problem seems to be, that duplicity does not cleanup signature-files
> >> when removing old backups, so the signatures occupy more and more space.
> >>
> >> To reproduce, I did some backups (locally), some with a forced full
> >> backup and some incementals. Afterwards the backup looks like this:
> >>
> >> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> >> Local and Remote metadata are synchronized, no sync needed.
> >> Last full backup date: Wed Feb 10 16:51:46 2010
> >> Collection Status
> >> -----------------
> >> Connecting with backend: LocalBackend
> >> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
> >>
> >> Found 2 secondary backup chains.
> >> Secondary chain 1 of 2:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:50:16 2010
> >> Chain end time: Wed Feb 10 16:50:44 2010
> >> Number of contained backup sets: 4
> >> Total number of contained volumes: 4
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:50:16 2010                 1
> >>           Incremental         Wed Feb 10 16:50:36 2010                 1
> >>           Incremental         Wed Feb 10 16:50:41 2010                 1
> >>           Incremental         Wed Feb 10 16:50:44 2010                 1
> >> -------------------------
> >>
> >> Secondary chain 2 of 2:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:51:01 2010
> >> Chain end time: Wed Feb 10 16:51:39 2010
> >> Number of contained backup sets: 4
> >> Total number of contained volumes: 4
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:51:01 2010                 1
> >>           Incremental         Wed Feb 10 16:51:36 2010                 1
> >>           Incremental         Wed Feb 10 16:51:38 2010                 1
> >>           Incremental         Wed Feb 10 16:51:39 2010                 1
> >> -------------------------
> >>
> >>
> >> Found primary backup chain with matching signature chain:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:51:46 2010
> >> Chain end time: Wed Feb 10 16:52:04 2010
> >> Number of contained backup sets: 2
> >> Total number of contained volumes: 2
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:51:46 2010                 1
> >>           Incremental         Wed Feb 10 16:52:04 2010                 1
> >> -------------------------
> >> No orphaned or incomplete backup sets found.
> >> $ ls /tmp/duptest/
> >> duplicity-full.20100210T155016Z.manifest.gpg
> >> duplicity-full.20100210T155016Z.vol1.difftar.gpg
> >> duplicity-full.20100210T155101Z.manifest.gpg
> >> duplicity-full.20100210T155101Z.vol1.difftar.gpg
> >> duplicity-full.20100210T155146Z.manifest.gpg
> >> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> >> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> >> duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
> >> duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
> >> duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
> >> duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
> >> duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
> >> duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
> >> duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> >> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> >>
> >> Now I remove the old backups:
> >>
> >> $ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
> >> file:///tmp/duptest
> >> Local and Remote metadata are synchronized, no sync needed.
> >> Last full backup date: Wed Feb 10 16:51:46 2010
> >> Deleting backup sets at times:
> >> Wed Feb 10 16:50:16 2010
> >> Wed Feb 10 16:50:36 2010
> >> Wed Feb 10 16:50:41 2010
> >> Wed Feb 10 16:50:44 2010
> >> Wed Feb 10 16:51:01 2010
> >> Wed Feb 10 16:51:36 2010
> >> Wed Feb 10 16:51:38 2010
> >> Wed Feb 10 16:51:39 2010
> >> Warning, found the following local orphaned signature files:
> >> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> >> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
> >>
> >> Note the "locally orphaned files".
> >> When I look at the status:
> >>
> >> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> >> Synchronizing remote metadata to local cache...
> >> Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
> >> Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
> >> Last full backup date: Wed Feb 10 16:51:46 2010
> >> Collection Status
> >> -----------------
> >> Connecting with backend: LocalBackend
> >> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
> >>
> >> Found 0 secondary backup chains.
> >>
> >> Found primary backup chain with matching signature chain:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:51:46 2010
> >> Chain end time: Wed Feb 10 16:52:04 2010
> >> Number of contained backup sets: 2
> >> Total number of contained volumes: 2
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:51:46 2010                 1
> >>           Incremental         Wed Feb 10 16:52:04 2010                 1
> >> -------------------------
> >> No orphaned or incomplete backup sets found.
> >>
> >> Note that duplicity did fetch the signatures of the 2 old (deleted!)
> >> full-backups
> >>
> >> When looking at the target, it becomes clear that "duplicity remove..."
> >> did only remove the data (*vol*difftar-files), not the obsoleted signatures:
> >>
> >> $ ls /tmp/duptest/
> >> duplicity-full.20100210T155146Z.manifest.gpg
> >> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> >> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> >> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> >>
> >> These files are also present (in decrypted form) in the cache-dir. And
> >> they stay there (or are refetched from the backup-target).
> >> Only workaround: delete the unneeded signature-files on the target, then
> >> duplicity also removes them from the cache (but be carefull to only
> >> remove old and unneeded files):
> >> $ cd /tmp/duptest/
> >> :/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
> >> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T1550*
> >> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
> >> duplicity-new-signatures.20100210T15513*
> >> :/tmp/duptest$ ls
> >> duplicity-full.20100210T155146Z.manifest.gpg
> >> duplicity-full.20100210T155146Z.vol1.difftar.gpg
> >> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
> >> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
> >> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
> >> :/tmp/duptest$ cd -
> >> ~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
> >> Synchronizing remote metadata to local cache...
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
> >> (not authoritative at backend).
> >> Deleting local
> >> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
> >> (not authoritative at backend).
> >> Last full backup date: Wed Feb 10 16:51:46 2010
> >> Collection Status
> >> -----------------
> >> Connecting with backend: LocalBackend
> >> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
> >>
> >> Found 0 secondary backup chains.
> >>
> >> Found primary backup chain with matching signature chain:
> >> -------------------------
> >> Chain start time: Wed Feb 10 16:51:46 2010
> >> Chain end time: Wed Feb 10 16:52:04 2010
> >> Number of contained backup sets: 2
> >> Total number of contained volumes: 2
> >>   Type of backup set:                            Time:      Num volumes:
> >>                  Full         Wed Feb 10 16:51:46 2010                 1
> >>           Incremental         Wed Feb 10 16:52:04 2010                 1
> >> -------------------------
> >> No orphaned or incomplete backup sets found.
> >>
> >> (Of course, one has to do full-backups from time to time in the first
> >> place, when you only ever do incrementals, ALL signature-files are still
> >> needed)
> >>
> >> cu
> >> Tim
> >>
> >>
> >>
> >> _______________________________________________
> >> Duplicity-talk mailing list
> >> [hidden email]
> >> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >>
> >
> >
> >
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > [hidden email]
> > http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
>
> _______________________________________________
> Duplicity-talk mailing list
> [hidden email]
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk

--
Olivier BERGER <[hidden email]>
http://www-public.it-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8
Ingénieur Recherche - Dept INF
Institut TELECOM, SudParis (http://www.it-sudparis.eu/), Evry (France)



_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Kenneth Loafman
The OP never did file a bug report and its not been "fixed".

The large size is in the nature of the beast that is incremental backup.
 We try to keep backup sizes small, so we use small chunks for signature
comparison.  This produces more signatures on large files, and a minimal
backup.  Larger chunks would produce smaller signatures and a smaller
backup, so the only "fix" is to allow the user to select signature size
thresholds on their own.

I've not had much time to chase this, but the code that determines the
chunk size is simple, so it may not take much to make the change.

Plus, my suspicion is that these large metadata stores are a direct
result of long strings of incremental backups with no full backup at
regular intervals.  Since this is bad practice in general, I've not
really been inclined to fix duplicity to encourage it.

...Ken

Olivier Berger wrote:

> Hi.
>
> Has a ticket indeed been filed for this ?
>
> Is it fixed in latest release ?
>
> Thanks in advance.
>
> Regards,
>
> Le mercredi 10 février 2010 à 18:35 +0100, [hidden email] a écrit :
>> Yeah, well done .. please add a ticket at launchpad.. ede
>>
>> On 10.02.2010 17:22, Kenneth Loafman wrote:
>>> Tim,
>>>
>>> Good testing!
>>>
>>> Please report this bug at https://bugs.launchpad.net/duplicity
>>>
>>> ...Thanks,
>>> ...Ken
>>>
>>> Tim Riemenschneider wrote:
>>>> Timothee Besset schrieb:
>>>>> I'm finding that some systems that were backed up by duplicity are
>>>>> running out of disk space after I upgraded the duplicity version. More
>>>>> specifically, I am seeing 19GB worth of data in a ~/.cache/duplicity folder?
>>>>>
>>>>> I can't afford to have 20GB of disk occupied by my backup software. I
>>>>> haven't found much information on the nature of that directory, except
>>>>> that it's an addition in 0.6? Should it really be growing that big? Is
>>>>> there a limit to how much it'll grow? Is it a case of backup biting it's
>>>>> own tail like we had backups of /tmp trying to backup duplicity's own
>>>>> temporary files at one point?
>>>>>
>>>>>
>>>> The problem seems to be, that duplicity does not cleanup signature-files
>>>> when removing old backups, so the signatures occupy more and more space.
>>>>
>>>> To reproduce, I did some backups (locally), some with a forced full
>>>> backup and some incementals. Afterwards the backup looks like this:
>>>>
>>>> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
>>>> Local and Remote metadata are synchronized, no sync needed.
>>>> Last full backup date: Wed Feb 10 16:51:46 2010
>>>> Collection Status
>>>> -----------------
>>>> Connecting with backend: LocalBackend
>>>> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>>>>
>>>> Found 2 secondary backup chains.
>>>> Secondary chain 1 of 2:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:50:16 2010
>>>> Chain end time: Wed Feb 10 16:50:44 2010
>>>> Number of contained backup sets: 4
>>>> Total number of contained volumes: 4
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:50:16 2010                 1
>>>>           Incremental         Wed Feb 10 16:50:36 2010                 1
>>>>           Incremental         Wed Feb 10 16:50:41 2010                 1
>>>>           Incremental         Wed Feb 10 16:50:44 2010                 1
>>>> -------------------------
>>>>
>>>> Secondary chain 2 of 2:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:51:01 2010
>>>> Chain end time: Wed Feb 10 16:51:39 2010
>>>> Number of contained backup sets: 4
>>>> Total number of contained volumes: 4
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:51:01 2010                 1
>>>>           Incremental         Wed Feb 10 16:51:36 2010                 1
>>>>           Incremental         Wed Feb 10 16:51:38 2010                 1
>>>>           Incremental         Wed Feb 10 16:51:39 2010                 1
>>>> -------------------------
>>>>
>>>>
>>>> Found primary backup chain with matching signature chain:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:51:46 2010
>>>> Chain end time: Wed Feb 10 16:52:04 2010
>>>> Number of contained backup sets: 2
>>>> Total number of contained volumes: 2
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:51:46 2010                 1
>>>>           Incremental         Wed Feb 10 16:52:04 2010                 1
>>>> -------------------------
>>>> No orphaned or incomplete backup sets found.
>>>> $ ls /tmp/duptest/
>>>> duplicity-full.20100210T155016Z.manifest.gpg
>>>> duplicity-full.20100210T155016Z.vol1.difftar.gpg
>>>> duplicity-full.20100210T155101Z.manifest.gpg
>>>> duplicity-full.20100210T155101Z.vol1.difftar.gpg
>>>> duplicity-full.20100210T155146Z.manifest.gpg
>>>> duplicity-full.20100210T155146Z.vol1.difftar.gpg
>>>> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
>>>> duplicity-inc.20100210T155016Z.to.20100210T155036Z.manifest.gpg
>>>> duplicity-inc.20100210T155016Z.to.20100210T155036Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155036Z.to.20100210T155041Z.manifest.gpg
>>>> duplicity-inc.20100210T155036Z.to.20100210T155041Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155041Z.to.20100210T155044Z.manifest.gpg
>>>> duplicity-inc.20100210T155041Z.to.20100210T155044Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155101Z.to.20100210T155136Z.manifest.gpg
>>>> duplicity-inc.20100210T155101Z.to.20100210T155136Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155136Z.to.20100210T155138Z.manifest.gpg
>>>> duplicity-inc.20100210T155136Z.to.20100210T155138Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155138Z.to.20100210T155139Z.manifest.gpg
>>>> duplicity-inc.20100210T155138Z.to.20100210T155139Z.vol1.difftar.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
>>>> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>>>>
>>>> Now I remove the old backups:
>>>>
>>>> $ PASSPHRASE="foo" duplicity remove-all-but-n-full 1 --force
>>>> file:///tmp/duptest
>>>> Local and Remote metadata are synchronized, no sync needed.
>>>> Last full backup date: Wed Feb 10 16:51:46 2010
>>>> Deleting backup sets at times:
>>>> Wed Feb 10 16:50:16 2010
>>>> Wed Feb 10 16:50:36 2010
>>>> Wed Feb 10 16:50:41 2010
>>>> Wed Feb 10 16:50:44 2010
>>>> Wed Feb 10 16:51:01 2010
>>>> Wed Feb 10 16:51:36 2010
>>>> Wed Feb 10 16:51:38 2010
>>>> Wed Feb 10 16:51:39 2010
>>>> Warning, found the following local orphaned signature files:
>>>> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
>>>> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
>>>>
>>>> Note the "locally orphaned files".
>>>> When I look at the status:
>>>>
>>>> $ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
>>>> Synchronizing remote metadata to local cache...
>>>> Copying duplicity-full-signatures.20100210T155016Z.sigtar to local cache.
>>>> Copying duplicity-full-signatures.20100210T155101Z.sigtar to local cache.
>>>> Last full backup date: Wed Feb 10 16:51:46 2010
>>>> Collection Status
>>>> -----------------
>>>> Connecting with backend: LocalBackend
>>>> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>>>>
>>>> Found 0 secondary backup chains.
>>>>
>>>> Found primary backup chain with matching signature chain:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:51:46 2010
>>>> Chain end time: Wed Feb 10 16:52:04 2010
>>>> Number of contained backup sets: 2
>>>> Total number of contained volumes: 2
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:51:46 2010                 1
>>>>           Incremental         Wed Feb 10 16:52:04 2010                 1
>>>> -------------------------
>>>> No orphaned or incomplete backup sets found.
>>>>
>>>> Note that duplicity did fetch the signatures of the 2 old (deleted!)
>>>> full-backups
>>>>
>>>> When looking at the target, it becomes clear that "duplicity remove..."
>>>> did only remove the data (*vol*difftar-files), not the obsoleted signatures:
>>>>
>>>> $ ls /tmp/duptest/
>>>> duplicity-full.20100210T155146Z.manifest.gpg
>>>> duplicity-full.20100210T155146Z.vol1.difftar.gpg
>>>> duplicity-full-signatures.20100210T155016Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
>>>> duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>>>>
>>>> These files are also present (in decrypted form) in the cache-dir. And
>>>> they stay there (or are refetched from the backup-target).
>>>> Only workaround: delete the unneeded signature-files on the target, then
>>>> duplicity also removes them from the cache (but be carefull to only
>>>> remove old and unneeded files):
>>>> $ cd /tmp/duptest/
>>>> :/tmp/duptest$ rm duplicity-full-signatures.20100210T155016Z.sigtar.gpg
>>>> duplicity-full-signatures.20100210T155101Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T1550*
>>>> duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gpg
>>>> duplicity-new-signatures.20100210T15513*
>>>> :/tmp/duptest$ ls
>>>> duplicity-full.20100210T155146Z.manifest.gpg
>>>> duplicity-full.20100210T155146Z.vol1.difftar.gpg
>>>> duplicity-full-signatures.20100210T155146Z.sigtar.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.manifest.gpg
>>>> duplicity-inc.20100210T155146Z.to.20100210T155204Z.vol1.difftar.gpg
>>>> duplicity-new-signatures.20100210T155146Z.to.20100210T155204Z.sigtar.gpg
>>>> :/tmp/duptest$ cd -
>>>> ~$ PASSPHRASE="foo" duplicity collection-status file:///tmp/duptest
>>>> Synchronizing remote metadata to local cache...
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155016Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-full-signatures.20100210T155101Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155016Z.to.20100210T155036Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155036Z.to.20100210T155041Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155041Z.to.20100210T155044Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155101Z.to.20100210T155136Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155136Z.to.20100210T155138Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Deleting local
>>>> /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93/duplicity-new-signatures.20100210T155138Z.to.20100210T155139Z.sigtar.gz
>>>> (not authoritative at backend).
>>>> Last full backup date: Wed Feb 10 16:51:46 2010
>>>> Collection Status
>>>> -----------------
>>>> Connecting with backend: LocalBackend
>>>> Archive dir: /home/tim/.cache/duplicity/dfd6cdfea9f4dc13e6e57c08d55d2a93
>>>>
>>>> Found 0 secondary backup chains.
>>>>
>>>> Found primary backup chain with matching signature chain:
>>>> -------------------------
>>>> Chain start time: Wed Feb 10 16:51:46 2010
>>>> Chain end time: Wed Feb 10 16:52:04 2010
>>>> Number of contained backup sets: 2
>>>> Total number of contained volumes: 2
>>>>   Type of backup set:                            Time:      Num volumes:
>>>>                  Full         Wed Feb 10 16:51:46 2010                 1
>>>>           Incremental         Wed Feb 10 16:52:04 2010                 1
>>>> -------------------------
>>>> No orphaned or incomplete backup sets found.
>>>>
>>>> (Of course, one has to do full-backups from time to time in the first
>>>> place, when you only ever do incrementals, ALL signature-files are still
>>>> needed)
>>>>
>>>> cu
>>>> Tim
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Duplicity-talk mailing list
>>>> [hidden email]
>>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Duplicity-talk mailing list
>>> [hidden email]
>>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> [hidden email]
>> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc (268 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Tim Riemenschneider-2
In reply to this post by Olivier Berger
Olivier Berger schrieb:
> Hi.
>
> Has a ticket indeed been filed for this ?
>
> Is it fixed in latest release ?
>  
I reported that as #519948 some time ago.

Regards,
Tim

--
detco GbR
Tim Riemenschneider - Entwickler
Haferwende 27
28357 Bremen

Geschäftsführer: Sebastian Constantin, Jens Frischen
Phone:  0421 / 22 37 99 00
Fax:    0421 / 22 37 99 01
Mobil:  0179 / 21 27 285
Mail:   [hidden email]


Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und löschen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin enthaltenen Informationen sind nicht gestattet.


This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.



_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Tim Riemenschneider-2
In reply to this post by Kenneth Loafman
Kenneth Loafman schrieb:
> The OP never did file a bug report and its not been "fixed".
>
>  
The OP didn't, but I did ;-)
The bugid is https://bugs.launchpad.net/duplicity/+bug/519948

> The large size is in the nature of the beast that is incremental backup.
>  We try to keep backup sizes small, so we use small chunks for signature
> comparison.  This produces more signatures on large files, and a minimal
> backup.  Larger chunks would produce smaller signatures and a smaller
> backup, so the only "fix" is to allow the user to select signature size
> thresholds on their own.
>
> I've not had much time to chase this, but the code that determines the
> chunk size is simple, so it may not take much to make the change.
>
>  
As I said in my earlier post, that could be a problem of signatures of
old backups still lying around (at least that was the case with one of
my backups. After manually deleting older signatures (those, where no
difftars where around anymore) on the backend, the size of my
.cache-directory was reduced from 3 GB to 1 GB.
> Plus, my suspicion is that these large metadata stores are a direct
> result of long strings of incremental backups with no full backup at
> regular intervals.  Since this is bad practice in general, I've not
> really been inclined to fix duplicity to encourage it.
>
>  
When that is the case (with the OP), then you are right, that should not
be encouraged. But if it's old, unused signatures lying around, then the
size gets even bigger if you do full-backups more often!
(Ok, in the meantime I found the --extra-cleanup option, which deposed
of all the old signatures. But that is not the default)

cu
Tim

--
detco GbR
Tim Riemenschneider - Entwickler
Haferwende 27
28357 Bremen

Geschäftsführer: Sebastian Constantin, Jens Frischen
Phone:  0421 / 22 37 99 00
Fax:    0421 / 22 37 99 01
Mobil:  0179 / 21 27 285
Mail:   [hidden email]


Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und löschen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin enthaltenen Informationen sind nicht gestattet.


This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.



_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Kenneth Loafman
Tim Riemenschneider wrote:
> Kenneth Loafman schrieb:
>> The OP never did file a bug report and its not been "fixed".
>>
>>  
> The OP didn't, but I did ;-)
> The bugid is https://bugs.launchpad.net/duplicity/+bug/519948

Sorry, I missed that when I looked.

>> The large size is in the nature of the beast that is incremental backup.
>>  We try to keep backup sizes small, so we use small chunks for signature
>> comparison.  This produces more signatures on large files, and a minimal
>> backup.  Larger chunks would produce smaller signatures and a smaller
>> backup, so the only "fix" is to allow the user to select signature size
>> thresholds on their own.
>>
>> I've not had much time to chase this, but the code that determines the
>> chunk size is simple, so it may not take much to make the change.
>>
>>  
> As I said in my earlier post, that could be a problem of signatures of
> old backups still lying around (at least that was the case with one of
> my backups. After manually deleting older signatures (those, where no
> difftars where around anymore) on the backend, the size of my
> .cache-directory was reduced from 3 GB to 1 GB.
'duplicity cleanup' may be the culprit in this one.  I'll check that.

>> Plus, my suspicion is that these large metadata stores are a direct
>> result of long strings of incremental backups with no full backup at
>> regular intervals.  Since this is bad practice in general, I've not
>> really been inclined to fix duplicity to encourage it.
>>
>>  
> When that is the case (with the OP), then you are right, that should not
> be encouraged. But if it's old, unused signatures lying around, then the
> size gets even bigger if you do full-backups more often!
> (Ok, in the meantime I found the --extra-cleanup option, which deposed
> of all the old signatures. But that is not the default)
The extra-cleanup may have worked, but cleanup by itself should have
removed any old sigtars that were not associated with a current backup.

...Thanks,
...Ken


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc (268 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Michael Terry-4
On 6 March 2010 06:54, Kenneth Loafman <[hidden email]> wrote:
> The extra-cleanup may have worked, but cleanup by itself should have
> removed any old sigtars that were not associated with a current backup.

I think I wrote the patch that stopped us doing that automatically
unless --extra-clean was given.  The reason was, without old sigtars,
we can't list files in non-current backups.  If we can do that without
sigtars, then great.  But some way of listing old files will be
important to Deja Dup (and I think ede was interested too for duply).

Although, in those cases, it would also be nice not to take up so much
space.  Sigtars must be holding more than just file lists, eh?  Can we
'strip' them or something when they become non-current?
-mt


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

edgar.soldin

> I think I wrote the patch that stopped us doing that automatically
> unless --extra-clean was given.  The reason was, without old sigtars,
> we can't list files in non-current backups.  If we can do that without
> sigtars, then great.  But some way of listing old files will be
> important to Deja Dup (and I think ede was interested too for duply).

i am .. the big schedule is to list old versions and cherry pick the
ones to restore

btw. I am still looking for a size info on backed up files. Is it
possible to add this?

> Although, in those cases, it would also be nice not to take up so much
> space.  Sigtars must be holding more than just file lists, eh?  Can we
> 'strip' them or something when they become non-current?

my use case has daily backups, monthly fulls, volumesize is 50 MB, a
data size of 1,4GB and dates back 3 months .. the cache folder is 50MB,
i can live with that very well

in this case only few files change, mainly logfiles and such

... ede


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Kenneth Loafman
In reply to this post by Michael Terry-4
Michael Terry wrote:

> On 6 March 2010 06:54, Kenneth Loafman <[hidden email]> wrote:
>> The extra-cleanup may have worked, but cleanup by itself should have
>> removed any old sigtars that were not associated with a current backup.
>
> I think I wrote the patch that stopped us doing that automatically
> unless --extra-clean was given.  The reason was, without old sigtars,
> we can't list files in non-current backups.  If we can do that without
> sigtars, then great.  But some way of listing old files will be
> important to Deja Dup (and I think ede was interested too for duply).
>
> Although, in those cases, it would also be nice not to take up so much
> space.  Sigtars must be holding more than just file lists, eh?  Can we
> 'strip' them or something when they become non-current?
Sigtar files contain the signatures for the backups and they can be
quite large if your backup is large.  They should not be needed for
listing old content.  Maybe we need to rethink the listing mechanism.
All the data for listing should be available in the difftar files.

...Ken


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc (268 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

edgar.soldin
> Sigtar files contain the signatures for the backups and they can be
> quite large if your backup is large.  They should not be needed for
> listing old content.  Maybe we need to rethink the listing mechanism.
> All the data for listing should be available in the difftar files.

the signatures are needed to decide if a file changed since last backup,
right? .. ede


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

Kenneth Loafman
[hidden email] wrote:
>> Sigtar files contain the signatures for the backups and they can be
>> quite large if your backup is large.  They should not be needed for
>> listing old content.  Maybe we need to rethink the listing mechanism.
>> All the data for listing should be available in the difftar files.
>
> the signatures are needed to decide if a file changed since last backup,
> right? .. ede

Yes, but they are not needed for old backups.  The current full backup
makes them obsolete since you'll not need to add to an older backup.

...Ken



_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk

signature.asc (268 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Why do I have a 19GB ~/.cache/duplicity folder?

edgar.soldin
On 06.03.2010 16:32, Kenneth Loafman wrote:

> [hidden email] wrote:
>>> Sigtar files contain the signatures for the backups and they can be
>>> quite large if your backup is large.  They should not be needed for
>>> listing old content.  Maybe we need to rethink the listing mechanism.
>>> All the data for listing should be available in the difftar files.
>>
>> the signatures are needed to decide if a file changed since last backup,
>> right? .. ede
>
> Yes, but they are not needed for old backups.  The current full backup
> makes them obsolete since you'll not need to add to an older backup.
>
> ...Ken

makes perfect sense, thx ede


_______________________________________________
Duplicity-talk mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/duplicity-talk