Old backups are not removed

[duplicity-talk mailing list]

Hi,

I'm experiencing very similar problems as reported back in 2017 in the
thread starting at

https://lists.gnu.org/archive/html/duplicity-talk/2017-03/msg00014.html

Namely, my old backups are not deleted, and therefore the remote disk
gets full.

At the beginning of each month I remove all the backups and create a
full backup. Then, each day I create an incremental backup. The script
I use is something like this:

---------------------------------------
duplicity remove-older-than 3W --force file:////mnt/backup
duplicity / --full-if-older-than 4W --include-filelist /etc/duplicity/dirs.txt file:////mnt/backup
---------------------------------------

However, when the time comes it does not remove old backups:

---------------------------------------

$ duplicity remove-older-than 3W --force file:////mnt/backup
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Oct 17 00:05:59 2019
There are backup set(s) at time(s):
Wed Sep 18 08:54:46 2019
Thu Sep 19 00:06:26 2019
Fri Sep 20 00:07:16 2019
Mon Sep 23 08:52:00 2019
Tue Sep 24 00:07:06 2019
Wed Sep 25 00:06:01 2019
Thu Sep 26 00:09:06 2019
Which can't be deleted because newer sets depend on them.
No old backup sets found, nothing deleted.
---------------------------------------

Collection-status gives me the following:

---------------------------------------
$ duplicity collection-status file:///mnt/backup/
Warning, found incomplete backup sets, probably left from aborted session
Last full backup date: Wed Sep 18 08:54:46 2019
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.cache/duplicity/870198592d381b892337283420a78807

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Sep 18 08:54:46 2019
Chain end time: Wed Oct 16 00:09:30 2019
Number of contained backup sets: 27
Total number of contained volumes: 105
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Sep 18 08:54:46 2019                79
         Incremental         Thu Sep 19 00:06:26 2019                 1
         Incremental         Fri Sep 20 00:07:16 2019                 1
         Incremental         Mon Sep 23 08:52:00 2019                 1
         Incremental         Tue Sep 24 00:07:06 2019                 1
         Incremental         Wed Sep 25 00:06:01 2019                 1
         Incremental         Thu Sep 26 00:09:06 2019                 1
         Incremental         Fri Sep 27 00:07:06 2019                 1
         Incremental         Sat Sep 28 00:09:06 2019                 1
         Incremental         Sun Sep 29 00:06:00 2019                 1
         Incremental         Mon Sep 30 00:09:22 2019                 1
         Incremental         Tue Oct  1 00:05:35 2019                 1
         Incremental         Wed Oct  2 00:07:00 2019                 1
         Incremental         Thu Oct  3 00:09:03 2019                 1
         Incremental         Fri Oct  4 00:09:46 2019                 1
         Incremental         Sat Oct  5 00:10:22 2019                 1
         Incremental         Sun Oct  6 00:07:15 2019                 1
         Incremental         Mon Oct  7 00:10:44 2019                 1
         Incremental         Tue Oct  8 00:09:50 2019                 1
         Incremental         Wed Oct  9 00:09:43 2019                 1
         Incremental         Thu Oct 10 00:06:54 2019                 1
         Incremental         Fri Oct 11 00:07:10 2019                 1
         Incremental         Sat Oct 12 00:10:31 2019                 1
         Incremental         Sun Oct 13 00:07:16 2019                 1
         Incremental         Mon Oct 14 00:08:52 2019                 1
         Incremental         Tue Oct 15 00:05:52 2019                 1
         Incremental         Wed Oct 16 00:09:30 2019                 1
-------------------------
Also found 0 backup sets not part of any chain,
and 1 incomplete backup set.
These may be deleted by running duplicity with the "cleanup" command.
---------------------------------------

The "Warning, found incomplete backup sets, probably left from aborted
session" above is that the last backup couldn't finish because the disk was
full.

Any help will be much appreciated!

best.
                                aitor

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

[duplicity-talk mailing list]

hi Aitor,

comments inline below

On 17.10.2019 11:18, Aitor Soroa via Duplicity-talk wrote:

> Hi,
>
> I'm experiencing very similar problems as reported back in 2017 in the
> thread starting at
>
> https://lists.gnu.org/archive/html/duplicity-talk/2017-03/msg00014.html
>
> Namely, my old backups are not deleted, and therefore the remote disk
> gets full.
>
> At the beginning of each month I remove all the backups and create a
> full backup. Then, each day I create an incremental backup. The script
> I use is something like this:
>
> ---------------------------------------
> duplicity remove-older-than 3W --force file:////mnt/backup
> duplicity / --full-if-older-than 4W --include-filelist /etc/duplicity/dirs.txt file:////mnt/backup
> ---------------------------------------

generally a bad idea conceptually. you should only run remove if your backups succeeded first.

> However, when the time comes it does not remove old backups:
>
> ---------------------------------------
>
> $ duplicity remove-older-than 3W --force file:////mnt/backup
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Thu Oct 17 00:05:59 2019
> There are backup set(s) at time(s):
> Wed Sep 18 08:54:46 2019
> Thu Sep 19 00:06:26 2019
> Fri Sep 20 00:07:16 2019
> Mon Sep 23 08:52:00 2019
> Tue Sep 24 00:07:06 2019
> Wed Sep 25 00:06:01 2019
> Thu Sep 26 00:09:06 2019
> Which can't be deleted because newer sets depend on them.
> No old backup sets found, nothing deleted.
> ---------------------------------------
>
> Collection-status gives me the following:
>
> ---------------------------------------
> $ duplicity collection-status file:///mnt/backup/
> Warning, found incomplete backup sets, probably left from aborted session
> Last full backup date: Wed Sep 18 08:54:46 2019
> Collection Status
> -----------------
> Connecting with backend: BackendWrapper
> Archive dir: /root/.cache/duplicity/870198592d381b892337283420a78807
>
> Found 0 secondary backup chains.
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed Sep 18 08:54:46 2019
> Chain end time: Wed Oct 16 00:09:30 2019
> Number of contained backup sets: 27
> Total number of contained volumes: 105
>  Type of backup set:                            Time:      Num volumes:
>                 Full         Wed Sep 18 08:54:46 2019                79
>          Incremental         Thu Sep 19 00:06:26 2019                 1
>          Incremental         Fri Sep 20 00:07:16 2019                 1
>          Incremental         Mon Sep 23 08:52:00 2019                 1
>          Incremental         Tue Sep 24 00:07:06 2019                 1
>          Incremental         Wed Sep 25 00:06:01 2019                 1
>          Incremental         Thu Sep 26 00:09:06 2019                 1
>          Incremental         Fri Sep 27 00:07:06 2019                 1
>          Incremental         Sat Sep 28 00:09:06 2019                 1
>          Incremental         Sun Sep 29 00:06:00 2019                 1
>          Incremental         Mon Sep 30 00:09:22 2019                 1
>          Incremental         Tue Oct  1 00:05:35 2019                 1
>          Incremental         Wed Oct  2 00:07:00 2019                 1
>          Incremental         Thu Oct  3 00:09:03 2019                 1
>          Incremental         Fri Oct  4 00:09:46 2019                 1
>          Incremental         Sat Oct  5 00:10:22 2019                 1
>          Incremental         Sun Oct  6 00:07:15 2019                 1
>          Incremental         Mon Oct  7 00:10:44 2019                 1
>          Incremental         Tue Oct  8 00:09:50 2019                 1
>          Incremental         Wed Oct  9 00:09:43 2019                 1
>          Incremental         Thu Oct 10 00:06:54 2019                 1
>          Incremental         Fri Oct 11 00:07:10 2019                 1
>          Incremental         Sat Oct 12 00:10:31 2019                 1
>          Incremental         Sun Oct 13 00:07:16 2019                 1
>          Incremental         Mon Oct 14 00:08:52 2019                 1
>          Incremental         Tue Oct 15 00:05:52 2019                 1
>          Incremental         Wed Oct 16 00:09:30 2019                 1
> -------------------------
> Also found 0 backup sets not part of any chain,
> and 1 incomplete backup set.
> These may be deleted by running duplicity with the "cleanup" command.
> ---------------------------------------
>
> The "Warning, found incomplete backup sets, probably left from aborted
> session" above is that the last backup couldn't finish because the disk was
> full.
>
> Any help will be much appreciated!
>

looks good to me. as you keep a 4 week long chain duplicity cannot delete backups from 3 weeks ago and earlier as it still depends on them if you wanna restore eg. yesterdays backup.

in you current setup you will have to keep backing up successfully for at least 7 weeks to see removal. duplicity will start a second chain and only after that was run for 3 weeks successfully the older chain can be removed. thats why it's called a chain ;)

technical background is simple. on restore duplicity first restores the first state of the file in the chain (usually from full) and the recursively applies the changes up until the date requested.

..ede/duply.net


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk