gpg: too many files open

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

gpg: too many files open

duplicity-talk mailing list
I am getting this on restoring. Looking at /proc I find
66 gpg processes, and each of them has about 260 open pipes.

So that opened about 17000 pipes.

Any ideas?

GPG_KEY='...'
GPG_PW='...'
GPG_OPTS="--pinentry-mode loopback" # - needed for GPG 2.1+ _and_

Start duply v2.0.2, time is 2019-02-14 22:15:22.
Using profile '/etc/duply/skull'.
Using installed duplicity version 0.7.18.2, python 2.7.15rc1, gpg 2.2.4 (Home: /root/.gnupg), awk 'GNU Awk 4.1.4, API: 1.1 (GNU MPFR 4.0.1, GNU MP 6.1.2)', grep 'grep (GNU grep) 3.1', bash '4.4.19(1)-
release (x86_64-pc-linux-gnu)'.
Autoset found secret key of first GPG_KEY entry 'D3AB10749D60BE45' for signing.
Checking TEMP_DIR '/tmp' is a folder and writable (OK)
Test - Encrypt to 'D3AB10749D60BE45' & Sign with 'D3AB10749D60BE45' (OK)
Test - Decrypt (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.7394.1550178922_*'(OK)

--- Start running command FETCH at 22:15:23.390 ---
Lokale und entfernte Metadaten sind bereits synchron, kein Abgleich benötigt.
Letzte vollständige Sicherung: Tue Jan 22 16:02:36 2019
Traceback (innermost last):
  File "/usr/local/bin/duplicity", line 1560, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1546, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1398, in main
    do_backup(action)
  File "/usr/local/bin/duplicity", line 1477, in do_backup
    restore(col_stats)
  File "/usr/local/bin/duplicity", line 733, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 558, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 521, in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 389, in yield_tuples
    setrorps(overflow, elems)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 378, in setrorps
    elems[i] = iter_list[i].next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 107, in filter_path_iter
    for path in path_iter:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 121, in difftar2path_iter
    tarinfo_list = [tar_iter.next()]
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 339, in next
    self.set_tarfile()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/patchdir.py", line 333, in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/usr/local/bin/duplicity", line 770, in get_fileobj_iter
    manifest.volume_info_dict[vol_num])
  File "/usr/local/bin/duplicity", line 813, in restore_get_enc_fileobj
    fileobj = tdp.filtered_open_with_delete("rb")
  File "/usr/local/lib/python2.7/dist-packages/duplicity/dup_temp.py", line 120, in filtered_open_with_delete
    fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
  File "/usr/local/lib/python2.7/dist-packages/duplicity/path.py", line 779, in filtered_open
    return gpg.GPGFile(False, self, gpg_profile)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 225, in __init__
    'logger': self.logger_fp})
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
    create_fhs, attach_fhs)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 402, in _attach_fork_exec
    pipe = os.pipe()
 OSError: [Errno 24] Too many open files

--
Wolfgang


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg: too many files open

duplicity-talk mailing list
On Do, 2019-02-14 at 22:29 +0100, Wolfgang Rohdewald via Duplicity-talk wrote:
> I am getting this on restoring. Looking at /proc I find
> 66 gpg processes, and each of them has about 260 open pipes.
>
> So that opened about 17000 pipes.
>
> Any ideas?

since I urgently needed the data, I drastically increased ulimit -n.

I believe the problem is that I have a LOT of small incremental
backups (one every work hour), 392 incrementals since last full backup.

And I believe duplicity calls GnuPG.run() for each and every inc
backup and releases all of them only when ending the process.
I added some debug output and got:

Verarbeitung des lokalen Manifest /root/.cache/duplicity/duply_skull/duplicity-inc.20190125T200203Z.to.20190125T210204Z.manifest (166)
Manifest-Volumen 1 gefunden
1 Volumen in Manifest gefunden
(mktemp) temporäre Datei /tmp/duplicity-a4mLfv-tempdir/mktemp-1ksqTV-59 wird registriert
('GnuPG.run gets create_fhs', ['stdout', 'passphrase'])
('len(create_fhs):', 2)
Verarbeitung des lokalen Manifest /root/.cache/duplicity/duply_skull/duplicity-inc.20190125T210204Z.to.20190125T220204Z.manifest (166)
Manifest-Volumen 1 gefunden
1 Volumen in Manifest gefunden
(mktemp) temporäre Datei /tmp/duplicity-a4mLfv-tempdir/mktemp-FI6AnK-60 wird registriert
('GnuPG.run gets create_fhs', ['stdout', 'passphrase'])
('len(create_fhs):', 2)



--
Wolfgang


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg: too many files open

duplicity-talk mailing list
On Thu, 14 Feb 2019, Wolfgang Rohdewald via Duplicity-talk wrote:

> On Do, 2019-02-14 at 22:29 +0100, Wolfgang Rohdewald via Duplicity-talk wrote:
>> I am getting this on restoring. Looking at /proc I find
>> 66 gpg processes, and each of them has about 260 open pipes.
>>
>> So that opened about 17000 pipes.
>>
>> Any ideas?
>
> since I urgently needed the data, I drastically increased ulimit -n.
>
> I believe the problem is that I have a LOT of small incremental
> backups (one every work hour), 392 incrementals since last full backup.
>
> And I believe duplicity calls GnuPG.run() for each and every inc
> backup and releases all of them only when ending the process.
> I added some debug output and got:

Yeah, I think this is expected behavior.  When restoring, duplicity wants
to read through (and decrypt) all the incrementals simultaneously.  The
idea is that since the diffs to the files are stored in sorted order, by
doing this you get to apply all the diffs to get the final version of a
single file before moving on to the next.

The alternative would be to do a full restore of one incremental, and then
patch all the files to get the next one, and so on.  One disadvantage
would be that you would need enough disk space for the largest of all the
incrementals, instead of just for the final one.

The solution is as you found: if you have a very long chain of
incrementals, you'll have to arrange for a higher file limit when you
restore.

--
Nate Eldredge
[hidden email]


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg: too many files open

duplicity-talk mailing list
A couple of answers...

You need to up the ulimit on the system, or

You need to upgrade to the latest version.  This bug has been fixed for a long time.


On Thu, Feb 14, 2019 at 4:45 PM Nate Eldredge via Duplicity-talk <[hidden email]> wrote:
On Thu, 14 Feb 2019, Wolfgang Rohdewald via Duplicity-talk wrote:

> On Do, 2019-02-14 at 22:29 +0100, Wolfgang Rohdewald via Duplicity-talk wrote:
>> I am getting this on restoring. Looking at /proc I find
>> 66 gpg processes, and each of them has about 260 open pipes.
>>
>> So that opened about 17000 pipes.
>>
>> Any ideas?
>
> since I urgently needed the data, I drastically increased ulimit -n.
>
> I believe the problem is that I have a LOT of small incremental
> backups (one every work hour), 392 incrementals since last full backup.
>
> And I believe duplicity calls GnuPG.run() for each and every inc
> backup and releases all of them only when ending the process.
> I added some debug output and got:

Yeah, I think this is expected behavior.  When restoring, duplicity wants
to read through (and decrypt) all the incrementals simultaneously.  The
idea is that since the diffs to the files are stored in sorted order, by
doing this you get to apply all the diffs to get the final version of a
single file before moving on to the next.

The alternative would be to do a full restore of one incremental, and then
patch all the files to get the next one, and so on.  One disadvantage
would be that you would need enough disk space for the largest of all the
incrementals, instead of just for the final one.

The solution is as you found: if you have a very long chain of
incrementals, you'll have to arrange for a higher file limit when you
restore.

--
Nate Eldredge
[hidden email]


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg: too many files open

duplicity-talk mailing list
On Do, 2019-02-14 at 16:55 -0600, Kenneth Loafman via Duplicity-talk wrote:
> You need to upgrade to the latest version.  This bug has been fixed for a long time.

Huh? You can see in my first mail I am running the latest duplicity version 0.7.18.2

--
Wolfgang



_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg: too many files open

duplicity-talk mailing list
On 15.02.2019 01:23, Wolfgang Rohdewald via Duplicity-talk wrote:
> On Do, 2019-02-14 at 16:55 -0600, Kenneth Loafman via Duplicity-talk wrote:
>> You need to upgrade to the latest version.  This bug has been fixed for a long time.
>
> Huh? You can see in my first mail I am running the latest duplicity version 0.7.18.2
>

hey Wolfgang,

hmm.. you are, but can you post your version of
 /usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py
.
there's a slight chance that it is old despite duplicity binary being new. how did/do you update your duplicity?

..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg: too many files open

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On 14.02.2019 23:44, Nate Eldredge via Duplicity-talk wrote:
> I am getting this on restoring. Looking at /proc I find
> 66 gpg processes, and each of them has about 260 open pipes.
>
> So that opened about 17000 pipes.

yupp.. that does not look healthy. ..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg: too many files open

duplicity-talk mailing list
In reply to this post by duplicity-talk mailing list
On Fr, 2019-02-15 at 11:29 +0100, edgar.soldin--- via Duplicity-talk wrote:

> On 15.02.2019 01:23, Wolfgang Rohdewald via Duplicity-talk wrote:
> > On Do, 2019-02-14 at 16:55 -0600, Kenneth Loafman via Duplicity-talk wrote:
> > > You need to upgrade to the latest version.  This bug has been fixed for a long time.
> >
> > Huh? You can see in my first mail I am running the latest duplicity version 0.7.18.2
> >
>
> hey Wolfgang,
>
> hmm.. you are, but can you post your version of
>  /usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py
> .
> there's a slight chance that it is old despite duplicity binary being new. how did/do you update your duplicity?

python setup.py install

It was the latest unchanged source code. And it is the executed one, the debug output
actually appeared.

I could try to create a reproducible test case, if you want me to.

wget https://code.launchpad.net/duplicity/0.7-series/0.7.18.2/+download/duplicity-0.7.18.2.tar.gz
tar xf duplicity-0.7.18.2.tar.gz  
cd duplicity-0.7.18.2/

LANG=C diff -r duplicity /usr/local/lib/python2.7/dist-packages/duplicity 2>&1 | fgrep -v .pyc

Only in /usr/local/lib/python2.7/dist-packages/duplicity: _librsync.so
Only in duplicity: _librsyncmodule.c
Only in duplicity/backends: README


--
Wolfgang


_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Reply | Threaded
Open this post in threaded view
|

Re: gpg: too many files open

duplicity-talk mailing list
On 15.02.2019 12:24, Wolfgang Rohdewald wrote:

> On Fr, 2019-02-15 at 11:29 +0100, edgar.soldin--- via Duplicity-talk wrote:
>> On 15.02.2019 01:23, Wolfgang Rohdewald via Duplicity-talk wrote:
>>> On Do, 2019-02-14 at 16:55 -0600, Kenneth Loafman via Duplicity-talk wrote:
>>>> You need to upgrade to the latest version.  This bug has been fixed for a long time.
>>>
>>> Huh? You can see in my first mail I am running the latest duplicity version 0.7.18.2
>>>
>>
>> hey Wolfgang,
>>
>> hmm.. you are, but can you post your version of
>>  /usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py
>> .
>> there's a slight chance that it is old despite duplicity binary being new. how did/do you update your duplicity?
>
> python setup.py install
>
> It was the latest unchanged source code. And it is the executed one, the debug output
> actually appeared.
>
> I could try to create a reproducible test case, if you want me to.

that would be most helpful.

and also please open a bug on launchpad, so it does not get lost.

> wget https://code.launchpad.net/duplicity/0.7-series/0.7.18.2/+download/duplicity-0.7.18.2.tar.gz
> tar xf duplicity-0.7.18.2.tar.gz
> cd duplicity-0.7.18.2/
>
> LANG=C diff -r duplicity /usr/local/lib/python2.7/dist-packages/duplicity 2>&1 | fgrep -v .pyc
>
> Only in /usr/local/lib/python2.7/dist-packages/duplicity: _librsync.so
> Only in duplicity: _librsyncmodule.c
> Only in duplicity/backends: README

looks good to me. you seem to be up to date. ..ede/duply.net




_______________________________________________
Duplicity-talk mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/duplicity-talk