Excessive use of /var/lib/sks/DB/log.*

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Excessive use of /var/lib/sks/DB/log.*

Gunnar Wolf
Hello,

Since a couple of days ago, I have noticed an incredible growth in
space usage (so much it has killed my server twice, hitting partition
limits). Since February 3rd, I have over 4000 files with binary-only
contents (could find no meaningful information in them) called
/var/lib/sks/DB/log.<increasing-11-digit-number> - I have just
checked, and it *seems* removing them causes no ill effects.

But... TTBOMK, I have not modified anything in my configuration. I
removed them knowing I might be doing something stupid, in the know
that BDB does not like external processes touching its turf, and was
happy not to lose my keyserver - but I was ready to rebuild it.

I haven't been monitoring this list as much as I should. Is there
anything I should be on the look for? I removed files dated ≤ Feb 3,
so there is still a lot of information if somebody wants to debug the
issue.

Should I just point a "logrotate" or such to the directory? What did I
do before that I'm not correctly doing now?

Thanks!


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Excessive use of /var/lib/sks/DB/log.*

Todd Fleisher
This sounds like you are missing the recommended DB_CONFIG values to prevent your server from holding into those log files when an issue is encountered. As I recall, the fix is to start over from scratch and rebuild after first putting that file in place. It is covered in the list archives and I believe the source and packages ship with an example file that had the needed options.

Sent from the Fleishphone

> On Feb 6, 2019, at 19:13, Gunnar Wolf <[hidden email]> wrote:
>
> Hello,
>
> Since a couple of days ago, I have noticed an incredible growth in
> space usage (so much it has killed my server twice, hitting partition
> limits). Since February 3rd, I have over 4000 files with binary-only
> contents (could find no meaningful information in them) called
> /var/lib/sks/DB/log.<increasing-11-digit-number> - I have just
> checked, and it *seems* removing them causes no ill effects.
>
> But... TTBOMK, I have not modified anything in my configuration. I
> removed them knowing I might be doing something stupid, in the know
> that BDB does not like external processes touching its turf, and was
> happy not to lose my keyserver - but I was ready to rebuild it.
>
> I haven't been monitoring this list as much as I should. Is there
> anything I should be on the look for? I removed files dated ≤ Feb 3,
> so there is still a lot of information if somebody wants to debug the
> issue.
>
> Should I just point a "logrotate" or such to the directory? What did I
> do before that I'm not correctly doing now?
>
> Thanks!
>
> _______________________________________________
> Sks-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/sks-devel


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Excessive use of /var/lib/sks/DB/log.*

Daniel Kahn Gillmor-7
On Wed 2019-02-06 20:27:28 -0800, Todd Fleisher wrote:
> This sounds like you are missing the recommended DB_CONFIG values to
> prevent your server from holding into those log files when an issue is
> encountered. As I recall, the fix is to start over from scratch and
> rebuild after first putting that file in place. It is covered in the
> list archives and I believe the source and packages ship with an
> example file that had the needed options.

If you're using a debian system, please compare
/usr/share/doc/sks/sampleConfig/DB_CONFIG with
/var/lib/sks/DB/DB_CONFIG -- if your files differ, i'd be happy to help
you figure out whether the problematic behavior you're seeing could be
attributable to those differences.

If they don't differ (if you're seeing the problematic behavior using
the sample DB_CONFIG, and *especially* if you fix your problem by
deviating from the shipped sample), i'd like to know that too.

    --dkg

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (233 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Excessive use of /var/lib/sks/DB/log.*

Gunnar Wolf-2
Hey dkg!

> If you're using a debian system, please compare
> /usr/share/doc/sks/sampleConfig/DB_CONFIG with
> /var/lib/sks/DB/DB_CONFIG -- if your files differ, i'd be happy to help
> you figure out whether the problematic behavior you're seeing could be
> attributable to those differences.
>
> If they don't differ (if you're seeing the problematic behavior using
> the sample DB_CONFIG, and *especially* if you fix your problem by
> deviating from the shipped sample), i'd like to know that too.

With hindsight, that was my mistake - When I rebuilt my server
about a month ago (after a DB corruption), I decided to keep my
installation and just delete /var/lib/sks/DB/*, rebuilding from
dumps. Of course, I blew the DB_CONFIG (which IMO has no business
there!)

In fact... I think that The Debian Way™ would be to have it in
/etc/sks, with a message on top clearly stating it should be linked
from /var/lib/sks/DB (as we Debian people are often too lazy to look
up configuration details in our software and expect everything to be
in /etc) 😉

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Excessive use of /var/lib/sks/DB/log.*

John Zaitseff
Hi, all,

> In fact... I think that The Debian Way™ would be to have [the
> DB_CONFIG file] in /etc/sks, with a message on top clearly stating
> it should be linked from /var/lib/sks/DB (as we Debian people are
> often too lazy to look up configuration details in our software
> and expect everything to be in /etc) 😉

That is indeed how I set up my own system: /etc/sks/DB_CONFIG is the
actual config file, and /var/lib/sks/DB/DB_CONFIG and
/var/lib/sks/PTree/DB_CONFIG are symlinks to it.

> > If you're using a debian system, please compare
> > /usr/share/doc/sks/sampleConfig/DB_CONFIG with
> > /var/lib/sks/DB/DB_CONFIG

I overwrote my DB_CONFIG file back in September 2018.  I changed

  set_lock_timeout      1000
  set_txn_timeout       1000

to

  set_lock_timeout      10000000
  set_txn_timeout       5000000

I did not notice any negative effects, but, by the same token, I was
still getting "add_keys_merge failed: Eventloop.SigAlarm" and "Key
addition failed: Eventloop.SigAlarm" in my log files.  Changing
/etc/sks/sksconf to include the following lines has completely
stopped those events from occurring (I made the change five days
ago):

  pagesize:          32
  ptree_pagesize:    16
  command_timeout:   600
  max_recover:       150

I fear, however, that increasing the timeouts simply pushes the
problem slightly further down the track...

Yours truly,

John Zaitseff

--
John Zaitseff                   ,--_|\    The ZAP Group
Telephone: +61 2 9643 7737     /      \   Sydney, Australia
Email: [hidden email]   \_,--._*   https://www.zap.org.au/
                                     v

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Excessive use of /var/lib/sks/DB/log.*

Daniel Kahn Gillmor-7
In reply to this post by Gunnar Wolf-2
On Fri 2019-02-08 16:48:16 -0600, Gunnar Wolf wrote:
> Hey dkg!

hi Gunnar! :)

> With hindsight, that was my mistake - When I rebuilt my server
> about a month ago (after a DB corruption), I decided to keep my
> installation and just delete /var/lib/sks/DB/*, rebuilding from
> dumps. Of course, I blew the DB_CONFIG (which IMO has no business
> there!)

oof, agreed.  mixing config and data is bad practice.

> In fact... I think that The Debian Way™ would be to have it in
> /etc/sks, with a message on top clearly stating it should be linked
> from /var/lib/sks/DB (as we Debian people are often too lazy to look
> up configuration details in our software and expect everything to be
> in /etc) 😉

Assuming that bdb will read from a symlinked DB_CONFIG, You'd still need
to have recreated the /var/lib/sks/DB/DB_CONFIG symlink after blowing
away the whole DB directory.  Or perhaps we could automate that creation
if it doesn't exist?

Alternately, we could have an automated check that a DB_CONFIG file at
least exists, and produce a noisy warning someplace in the packaging or
systemd unit files if it does not.

I'd welcome any such patches to the debian packaging if anyone has any
concrete suggestions.

this kind of finicky stuff is atrocious and should not be the
responsibility of the sysadmin, it should all Just Work™ by default. :/

               --dkg

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (233 bytes) Download Attachment