OSError: [Errno 24] Too many open files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

OSError: [Errno 24] Too many open files

duplicity-talk mailing list
Ubuntu 16.04, and yes I'm still running 0.7.19.  Everything was fine until my cron scheduled backups stopped without warning on June 6th, as I just discovered.

Command line that looks something like:

PASSPHRASE="xxx" duplicity --log-file /home/me/duplicity.log --backend-retry-delay 60 --asynchronous-upload --name Remote --volsize 50 --full-if-older-than 6M --exclude '**.lock' /home/dev/mydir sftp://[hidden email]/Backup

Error below.  Any way around this until I get a new OS installed?

Thanks -

/usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: FutureWarning: CTR mode needs counter parameter, not IV
  self._cipher = factory.new(key, *args, **kwargs)
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed May 13 22:20:03 2020

ERROR 30 OSError
. Traceback (innermost last):
.   File "/usr/bin/duplicity", line 1581, in <module>
.     with_tempdir(main)
.   File "/usr/bin/duplicity", line 1567, in with_tempdir
.     fn()
.   File "/usr/bin/duplicity", line 1419, in main
.     do_backup(action)
.   File "/usr/bin/duplicity", line 1549, in do_backup
.     incremental_backup(sig_chain)
.   File "/usr/bin/duplicity", line 673, in incremental_backup
.     globals.backend)
.   File "/usr/bin/duplicity", line 435, in write_multivol
.     globals.volsize)
.   File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 378, in GPGWriteFile
.     file = GPGFile(True, path.Path(filename), profile)
.   File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 206, in __init__
.     'logger': self.logger_fp})
.   File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
.     create_fhs, attach_fhs)
.   File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 402, in _attach_fork_exec
.     pipe = os.pipe()
.  OSError: [Errno 24] Too many open files


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: OSError: [Errno 24] Too many open files

duplicity-talk mailing list
On 20.07.2020 07:16, Diagon via Duplicity-talk wrote:

> Ubuntu 16.04, and yes I'm still running 0.7.19.  Everything was fine until my cron scheduled backups stopped without warning on June 6th, as I just discovered.
>
> Command line that looks something like:
>
> PASSPHRASE="xxx" duplicity --log-file /home/me/duplicity.log --backend-retry-delay 60 --asynchronous-upload --name Remote --volsize 50 --full-if-older-than 6M --exclude '**.lock' /home/dev/mydir sftp://[hidden email]/Backup
>
> Error below.  Any way around this until I get a new OS installed?
>
> Thanks -
>
> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: FutureWarning: CTR mode needs counter parameter, not IV
SNIP
> .   File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
> .     create_fhs, attach_fhs)
> .   File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 402, in _attach_fork_exec
> .     pipe = os.pipe()
> .  OSError: [Errno 24] Too many open files

for now you can work around the issue by raising the open files limit via ulimit
  https://linuxhandbook.com/ulimit-command/

would you mind posting a 'duplicity collection-status ...' of your backup. i would guess you some really very very long chain in there. maybe it is time to do a new full?

..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: OSError: [Errno 24] Too many open files

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
Date: Mon, 20 Jul 2020 11:48:04 +0200

> On 20.07.2020 07:16, Diagon via Duplicity-talk wrote:
>> Ubuntu 16.04, and yes I'm still running 0.7.19. Everything was fine until my cron scheduled backups stopped without warning on June 6th, as I just discovered.
>>
>> Command line that looks something like:
>>
>> PASSPHRASE="xxx" duplicity --log-file /home/me/duplicity.log --backend-retry-delay 60 --asynchronous-upload --name Remote --volsize 50 --full-if-older-than 6M --exclude '**.lock' /home/dev/mydir sftp://[hidden email]/Backup
>>
>> Error below. Any way around this until I get a new OS installed?
>>
>> Thanks -
>>
>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: FutureWarning: CTR mode needs counter parameter, not IV
SNIP
>> . File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
> >. create_fhs, attach_fhs)
>> . File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 402, in _attach_fork_exec
>> . pipe = os.pipe()
>> . OSError: [Errno 24] Too many open files

> for now you can work around the issue by raising the open files limit via ulimit
> https://linuxhandbook.com/ulimit-command/

Is that on the server or on the client that I have to raise that limit?

> would you mind posting a 'duplicity collection-status ...' of your backup. i would guess you some really very very long chain in there. maybe it is time to do a new full?

I'm backing up this directory every 20 minutes, but almost all of the time it's only one file changing.  So it looks like the following:

$ duplicity collection-status sftp://[hidden email]/Backup
/usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: FutureWarning: CTR mode needs counter parameter, not IV
  self._cipher = factory.new(key, *args, **kwargs)
  Last full backup date: Wed May 13 22:20:03 2020
  Collection Status
  -----------------
  Connecting with backend: BackendWrapper
  Archive dir: /home/me/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxx

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed May 13 22:20:03 2020
Chain end time: Mon Jun 15 17:40:06 2020
Number of contained backup sets: 1007
Total number of contained volumes: 1052
Type of backup set:                            Time:      Num volumes:
Full                Wed May 13 22:20:03 2020                46
Incremental         Thu May 14 03:40:04 2020                 1

<That last line, with different date/times and Num volumes = 1, repeats 1006 times>

> ..ede/duply.net

/D

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: OSError: [Errno 24] Too many open files

duplicity-talk mailing list
On 21.07.2020 10:46, Diagon via Duplicity-talk wrote:

> Date: Mon, 20 Jul 2020 11:48:04 +0200
>
>> On 20.07.2020 07:16, Diagon via Duplicity-talk wrote:
>>> Ubuntu 16.04, and yes I'm still running 0.7.19. Everything was fine until my cron scheduled backups stopped without warning on June 6th, as I just discovered.
>>>
>>> Command line that looks something like:
>>>
>>> PASSPHRASE="xxx" duplicity --log-file /home/me/duplicity.log --backend-retry-delay 60 --asynchronous-upload --name Remote --volsize 50 --full-if-older-than 6M --exclude '**.lock' /home/dev/mydir sftp://[hidden email]/Backup
>>>
>>> Error below. Any way around this until I get a new OS installed?
>>>
>>> Thanks -
>>>
>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: FutureWarning: CTR mode needs counter parameter, not IV
> SNIP
>>> . File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
>>> . create_fhs, attach_fhs)
>>> . File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 402, in _attach_fork_exec
>>> . pipe = os.pipe()
>>> . OSError: [Errno 24] Too many open files
>
>> for now you can work around the issue by raising the open files limit via ulimit
>> https://linuxhandbook.com/ulimit-command/
>
> Is that on the server or on the client that I have to raise that limit?

on the box duplicity is running on.

>> would you mind posting a 'duplicity collection-status ...' of your backup. i would guess you some really very very long chain in there. maybe it is time to do a new full?
>
> I'm backing up this directory every 20 minutes, but almost all of the time it's only one file changing.  So it looks like the following:
>
> $ duplicity collection-status sftp://[hidden email]/Backup
> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: FutureWarning: CTR mode needs counter parameter, not IV
>   self._cipher = factory.new(key, *args, **kwargs)
>   Last full backup date: Wed May 13 22:20:03 2020
>   Collection Status
>   -----------------
>   Connecting with backend: BackendWrapper
>   Archive dir: /home/me/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxx
>
> Found 0 secondary backup chains.
>
> Found primary backup chain with matching signature chain:
> -------------------------
> Chain start time: Wed May 13 22:20:03 2020
> Chain end time: Mon Jun 15 17:40:06 2020
> Number of contained backup sets: 1007
> Total number of contained volumes: 1052
> Type of backup set:                            Time:      Num volumes:
> Full                Wed May 13 22:20:03 2020                46
> Incremental         Thu May 14 03:40:04 2020                 1
>
> <That last line, with different date/times and Num volumes = 1, repeats 1006 times>

how do you manage to do 1000+ incrementals between 14 May and today (21 July)?

1000+ incrementals is a long chain. consider doing a new full. and generally doing them monthly or so.

..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk