using --remove-older-than takes a very long time?

Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

using --remove-older-than takes a very long time?

Derek Atkins-2
HI,

I've been running backups on my servers for a long time using
rdiff-backup.  My backup server has been offline for about a week and I
just put it back online yesterday.  The server does the following:

foreach host (list of machines to backup); do
  rdiff-backup host local-storage-for-host
  rdiff-backup --force --remove-older-than 1Y local-storage-for-host
end

This process starts at 1am.  I noticed today (it's not 8am) that the
remove process for the first server started at 4:09am and is still
running.  In other words, the "clear history" process is taking longer
than the backup did!

Is this expected behavior?

While I keep track of how long it takes to run the backup process for
each server, I didnt separate out the backup vs cleanup stages of the
process.  If cleanup is going to take so long, I might separate it out
into a different stage!  Although it would be nice if cleanup could be
sped up!

Is there any way to profile the code to see where it is spending its
time?

Thanks,

-derek

PS: I should mention this is with 1.2.8, not a more recent version.

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant

Reply | Threaded
Open this post in threaded view
|

Re: using --remove-older-than takes a very long time?

EricZolf
Hi Derek,

On 07/02/2020 14:46, Derek Atkins wrote:

> HI,
>
> I've been running backups on my servers for a long time using
> rdiff-backup.  My backup server has been offline for about a week and I
> just put it back online yesterday.  The server does the following:
>
> foreach host (list of machines to backup); do
>   rdiff-backup host local-storage-for-host
>   rdiff-backup --force --remove-older-than 1Y local-storage-for-host
> end
>
> This process starts at 1am.  I noticed today (it's not 8am) that the
> remove process for the first server started at 4:09am and is still
> running.  In other words, the "clear history" process is taking longer
> than the backup did!
>
> Is this expected behavior?

I don't think so but it probably depends on how many backups it has to
remove.

>
> While I keep track of how long it takes to run the backup process for
> each server, I didnt separate out the backup vs cleanup stages of the
> process.  If cleanup is going to take so long, I might separate it out
> into a different stage!  Although it would be nice if cleanup could be
> sped up!
>
> Is there any way to profile the code to see where it is spending its
> time?

A profiling as such is possible at python level but I would start with
just calling the removal option with `-v9`, it should give you/us a
first hint as the date/times for each action are logged.

KR, Eric

>
> Thanks,
>
> -derek
>
> PS: I should mention this is with 1.2.8, not a more recent version.

I wouldn't expect a big difference but you never know...

Reply | Threaded
Open this post in threaded view
|

Re: using --remove-older-than takes a very long time?

Derek Atkins-2
Hi,

On Fri, February 7, 2020 12:31 pm, Eric L. wrote:
> Hi Derek,
>
[snip]
> I don't think so but it probably depends on how many backups it has to
> remove.

Well, normally just one.   Today I suspect it is removing around 7.

>>
>> While I keep track of how long it takes to run the backup process for
>> each server, I didnt separate out the backup vs cleanup stages of the
>> process.  If cleanup is going to take so long, I might separate it out
>> into a different stage!  Although it would be nice if cleanup could be
>> sped up!
>>
>> Is there any way to profile the code to see where it is spending its
>> time?
>
> A profiling as such is possible at python level but I would start with
> just calling the removal option with `-v9`, it should give you/us a
> first hint as the date/times for each action are logged.

I can certainly change the code to do that tomorrow night; I think it's a
bit late to make that change now for today's run.

> KR, Eric

-derek
--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


Reply | Threaded
Open this post in threaded view
|

Re: using --remove-older-than takes a very long time?

Robert Nichols-2
On 2/7/20 11:42 AM, Derek Atkins wrote:
> Hi,
>
> On Fri, February 7, 2020 12:31 pm, Eric L. wrote:
>> A profiling as such is possible at python level but I would start with
>> just calling the removal option with `-v9`, it should give you/us a
>> first hint as the date/times for each action are logged.
>
> I can certainly change the code to do that tomorrow night; I think it's a
> bit late to make that change now for today's run.

You can use the "lsof" command or look in /proc/{PID}/fd/ to see what
files the process is handling. You might get some clue from that.

--
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.


Reply | Threaded
Open this post in threaded view
|

Re: using --remove-older-than takes a very long time?

Derek Atkins-2

On Fri, February 7, 2020 2:05 pm, Robert Nichols wrote:

> On 2/7/20 11:42 AM, Derek Atkins wrote:
>> Hi,
>>
>> On Fri, February 7, 2020 12:31 pm, Eric L. wrote:
>>> A profiling as such is possible at python level but I would start with
>>> just calling the removal option with `-v9`, it should give you/us a
>>> first hint as the date/times for each action are logged.
>>
>> I can certainly change the code to do that tomorrow night; I think it's
>> a
>> bit late to make that change now for today's run.
>
> You can use the "lsof" command or look in /proc/{PID}/fd/ to see what
> files the process is handling. You might get some clue from that.

The process has moved on to the next server now.  But I can try that
tomorrow.  Then again, tomorrow it will be running with -v9.

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant


Reply | Threaded
Open this post in threaded view
|

Re: using --remove-older-than takes a very long time?

Derek Atkins-2
In reply to this post by EricZolf
"Eric L." <[hidden email]> writes:

> I don't think so but it probably depends on how many backups it has to
> remove.

So that first run it had to remove 7.  Normally it's just removing 1 (as
it runs once per day).  However the difference in runtime between
removing 7 and removing 1 was definitely not 7x the runtime.  But it
definitely spends more time on the remove-old-backups than it does
backing up the day's changes.

[snip]
> A profiling as such is possible at python level but I would start with
> just calling the removal option with `-v9`, it should give you/us a
> first hint as the date/times for each action are logged.

I now have some multi-megabyte emails detailing the last few days of
processing (modulo some VM system crashes that happened over the
weekend).  There are dozens of operations per second (or more -- I
haven't looked closely), but still thousands of operations as it runs
for hours.  Frankly I'm not sure what I am looking at/for here.

I'm happy to share a log email (privately) if someone wants to look at
it with me?

-derek

--
       Derek Atkins                 617-623-3745
       [hidden email]             www.ihtfp.com
       Computer and Internet Security Consultant