On-demand SKS Statistics Generation?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

On-demand SKS Statistics Generation?

John Marshall-2
Is it possible to force generation of statistics by sending a signal to
a running DB server?  At the moment the only options I know about are to
shutdown the server and re-start with '-inital_stat'; or to wait for the
daily statistics generation at 'stat_hour'.

There are times (e.g. after changing peering configuration) when it
would be good to be able to generate new statistics without having to
restart the keyserver.

--
John Marshall

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

attachment0 (203 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Jason Harris
On Thu, Jul 02, 2009 at 08:09:24AM +1000, John Marshall wrote:
> Is it possible to force generation of statistics by sending a signal to
> a running DB server?  At the moment the only options I know about are to
> shutdown the server and re-start with '-inital_stat'; or to wait for the
> daily statistics generation at 'stat_hour'.
>
> There are times (e.g. after changing peering configuration) when it
> would be good to be able to generate new statistics without having to
> restart the keyserver.

It should be.  Look at sync_db_on_sig() in dbserver.ml, add
calculate_stats_page().  The code just below it seems to catch SIGUSR1.

--
Jason Harris           |  NIC:  JH329, PGP:  This _is_ PGP-signed, isn't it?
[hidden email] _|_ web:  http://keyserver.kjsl.com/~jharris/
          Got photons?   (TM), (C) 2004

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

attachment0 (322 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

John Marshall-2
On Wed, 01 Jul 2009, 18:51 -0400, Jason wrote:
> On Thu, Jul 02, 2009 at 08:09:24AM +1000, John Marshall wrote:
> > Is it possible to force generation of statistics by sending a signal to
> > a running DB server?  At the moment the only options I know about are to
> > shutdown the server and re-start with '-inital_stat'; or to wait for the
> > daily statistics generation at 'stat_hour'.
>
> It should be.  Look at sync_db_on_sig() in dbserver.ml, add
> calculate_stats_page().  The code just below it seems to catch SIGUSR1.

I can't find it in 1.1.0, so I guess you're looking at something later.
Good to know it's coming.  Thanks!

--
John Marshall

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

attachment0 (203 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Jason Harris
On Thu, Jul 02, 2009 at 02:21:28PM +1000, John Marshall wrote:
> On Wed, 01 Jul 2009, 18:51 -0400, Jason wrote:

> > It should be.  Look at sync_db_on_sig() in dbserver.ml, add
> > calculate_stats_page().  The code just below it seems to catch SIGUSR1.
>
> I can't find it in 1.1.0, so I guess you're looking at something later.
> Good to know it's coming.  Thanks!

Both mercurial (/usr/ports/devel/mercurial) repositories have the
SIGUSR1 code, which doesn't currently redo the stats, and both repos
are currently sync'd at changeset 46c6aaac31da:

  %hg clone http://hg.minskyprimus.net/sks/trunk/

or:

  %hg clone http://www.kim-minh.com/hg/sks

or:

  %wget http://www.kim-minh.com/hg/sks/archive/46c6aaac31da.tar.gz
  %esha1sum 46c6aaac31da.tar.gz
  ad450726165909d0f0aa5fc08748cc6bf051d39d 272456 46c6aaac31da.tar.gz

--
Jason Harris           |  NIC:  JH329, PGP:  This _is_ PGP-signed, isn't it?
[hidden email] _|_ web:  http://keyserver.kjsl.com/~jharris/
          Got photons?   (TM), (C) 2004

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

attachment0 (322 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Peter Pramberger
In reply to this post by Jason Harris
Jason schrieb am 02.07.2009 00:51:
> It should be.  Look at sync_db_on_sig() in dbserver.ml, add
> calculate_stats_page().  The code just below it seems to catch SIGUSR1.

We shouldn't mix this expensive operation with the database sync, but -USR2 is
still available for use; patch attached (applies to HEAD).

However the assignment of the signal handler still gives an error, can someone
with more OCaml insight help?


  let () = Sys.set_signal Sys.sigusr2
          (Sys.Signal_handle (fun _ -> calculate_stats_page ()))

  File "dbserver.ml", line 629, characters 32-55:
  Error: This expression has type 'a list
         but an expression was expected of type unit


Br,
Peter



diff -up sks-1.1.0/common.ml.orig sks-1.1.0/common.ml
--- sks-1.1.0/common.ml.orig 2009-07-02 16:48:40.000000000 +0200
+++ sks-1.1.0/common.ml 2009-07-02 16:53:43.000000000 +0200
@@ -131,7 +131,6 @@
 let () = Sys.set_signal Sys.sigterm (Sys.Signal_handle handle_interrupt)
 let () = Sys.set_signal Sys.sigint (Sys.Signal_handle handle_interrupt)
 let () = Sys.set_signal Sys.sigpipe Sys.Signal_ignore
-let () = Sys.set_signal Sys.sigusr2 Sys.Signal_ignore
 let () = Sys.set_signal Sys.sighup
    (Sys.Signal_handle (fun _ -> reopen_logfile ()))
 
diff -up sks-1.1.0/dbserver.ml.orig sks-1.1.0/dbserver.ml
--- sks-1.1.0/dbserver.ml.orig 2009-07-02 16:48:40.000000000 +0200
+++ sks-1.1.0/dbserver.ml 2009-07-02 16:49:08.000000000 +0200
@@ -625,6 +625,9 @@
   let () = Sys.set_signal Sys.sigusr1
   (Sys.Signal_handle (fun _ -> sync_db_on_sig ()))
 
+  let () = Sys.set_signal Sys.sigusr2
+  (Sys.Signal_handle (fun _ -> calculate_stats_page ()))
+
   (***********************************************************************)
 
   let run () =
diff -up sks-1.1.0/reconserver.ml.orig sks-1.1.0/reconserver.ml
--- sks-1.1.0/reconserver.ml.orig 2009-07-02 16:48:40.000000000 +0200
+++ sks-1.1.0/reconserver.ml 2009-07-02 16:52:47.000000000 +0200
@@ -313,6 +313,7 @@
   (***************************************************************)
 
   let () = Sys.set_signal Sys.sigusr1 Sys.Signal_ignore
+  let () = Sys.set_signal Sys.sigusr2 Sys.Signal_ignore
 
   (***********************************************************************)
 

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Peter Pramberger
Peter Pramberger schrieb am 02.07.2009 19:48:
> We shouldn't mix this expensive operation with the database sync, but -USR2 is
> still available for use; patch attached (applies to HEAD).
>
> However the assignment of the signal handler still gives an error, can someone
> with more OCaml insight help?

Fixed version, however this still calculates the stats inside the signal
handler. I tried to use Eventloop.Event() to trigger the calculation, but no
success:

  let calc_stats_on_sig () =
    let now = Unix.gettimeofday () in
    ignore (Eventloop.Event (now, Eventloop.Callback calculate_stats_page))

Any ideas?


Br,
Peter



diff -up sks-1.1.0/common.ml.orig sks-1.1.0/common.ml
--- sks-1.1.0/common.ml.orig 2009-07-02 16:48:40.000000000 +0200
+++ sks-1.1.0/common.ml 2009-07-02 16:53:43.000000000 +0200
@@ -131,7 +131,6 @@
 let () = Sys.set_signal Sys.sigterm (Sys.Signal_handle handle_interrupt)
 let () = Sys.set_signal Sys.sigint (Sys.Signal_handle handle_interrupt)
 let () = Sys.set_signal Sys.sigpipe Sys.Signal_ignore
-let () = Sys.set_signal Sys.sigusr2 Sys.Signal_ignore
 let () = Sys.set_signal Sys.sighup
    (Sys.Signal_handle (fun _ -> reopen_logfile ()))
 
diff -up sks-1.1.0/dbserver.ml.orig sks-1.1.0/dbserver.ml
--- sks-1.1.0/dbserver.ml.orig 2009-07-02 16:48:40.000000000 +0200
+++ sks-1.1.0/dbserver.ml 2009-07-02 21:55:23.000000000 +0200
@@ -622,9 +622,15 @@
     sync ();
     checkpoint ()
 
+  let calc_stats_on_sig () =
+    ignore (calculate_stats_page ())
+
   let () = Sys.set_signal Sys.sigusr1
   (Sys.Signal_handle (fun _ -> sync_db_on_sig ()))
 
+  let () = Sys.set_signal Sys.sigusr2
+  (Sys.Signal_handle (fun _ -> calc_stats_on_sig ()))
+
   (***********************************************************************)
 
   let run () =
diff -up sks-1.1.0/reconserver.ml.orig sks-1.1.0/reconserver.ml
--- sks-1.1.0/reconserver.ml.orig 2009-07-02 16:48:40.000000000 +0200
+++ sks-1.1.0/reconserver.ml 2009-07-02 16:52:47.000000000 +0200
@@ -313,6 +313,7 @@
   (***************************************************************)
 
   let () = Sys.set_signal Sys.sigusr1 Sys.Signal_ignore
+  let () = Sys.set_signal Sys.sigusr2 Sys.Signal_ignore
 
   (***********************************************************************)
 

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Kim Minh Kaplan
Peter Pramberger:

> Fixed version, however this still calculates the stats inside the signal
> handler. I tried to use Eventloop.Event() to trigger the calculation, but no
> success:
>
>   let calc_stats_on_sig () =
>     let now = Unix.gettimeofday () in
>     ignore (Eventloop.Event (now, Eventloop.Callback calculate_stats_page))
>
> Any ideas?

Eventloop does not give you access to the event queue.  Attached is a
kludge to do that.  I do *not* think this is a nice solution.  Better
would be to add signal handling semantic to eventloop.ml.

Kim Minh.


diff -r a86b082a3067 dbserver.ml
--- a/dbserver.ml Sun Jun 07 15:39:48 2009 +0200
+++ b/dbserver.ml Fri Jul 03 10:14:01 2009 +0200
@@ -632,6 +632,11 @@
   let () = Sys.set_signal Sys.sigusr1
   (Sys.Signal_handle (fun _ -> sync_db_on_sig ()))
 
+  let () = Sys.set_signal Sys.sigusr2
+      (Sys.Signal_handle (fun _ ->
+ Eventloop.add_events Eventloop.heap
+  [Eventloop.Event(0.0, Eventloop.Callback calculate_stats_page)]))
+
   (***********************************************************************)
 
   let run () =
diff -r a86b082a3067 eventloop.ml
--- a/eventloop.ml Sun Jun 07 15:39:48 2009 +0200
+++ b/eventloop.ml Fri Jul 03 10:14:01 2009 +0200
@@ -227,8 +227,9 @@
 (***************************************************************)
 (***************************************************************)
 
+let heap = Heap.empty (<) 20
+
 let evloop events socklist =
-  let heap = Heap.empty (<) 20 in
   add_events heap events;
   try
     while true do

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Dinko Korunic
On Fri, Jul 03, 2009 at 08:21:31AM +0000, Kim Minh Kaplan wrote:
> Eventloop does not give you access to the event queue.  Attached is a
> kludge to do that.  I do *not* think this is a nice solution.  Better
> would be to add signal handling semantic to eventloop.ml.

Sorry to bother, but do we have a HG repo containing *current* version
with all these patches incorporated? As far as I have seen, we have two
identical HG repos and they both have only non-devel releases without
several of patches that were released on this list?


Kind regards,
D.

--
NAME:Dinko.kreator.Korunic   DISCLAIMER:Standard.disclaimer.applies
ICQ:16965294        JAB:[hidden email]        PGP:0xEA160D0B
HOME:http://dkorunic.net    QUOTE:Eat.right.stay.fit.and.die.anyway


_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Kim Minh Kaplan
Dinko Korunic:

> Sorry to bother, but do we have a HG repo containing *current* version
> with all these patches incorporated? As far as I have seen, we have two
> identical HG repos and they both have only non-devel releases without
> several of patches that were released on this list?

Please note that Yaron Minsky's repository is *the* reference
repository.  Now that my patches have been included in his repository,
the only reason I publicize mine is so that people who do not want to
bother with Mercurial can download an archived (tar.gz) version easily.

Kim Minh.


_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Dinko Korunic
On Fri, Jul 03, 2009 at 06:07:19PM +0000, Kim Minh Kaplan wrote:
> Please note that Yaron Minsky's repository is *the* reference
> repository.  Now that my patches have been included in his repository,
> the only reason I publicize mine is so that people who do not want to
> bother with Mercurial can download an archived (tar.gz) version easily.

Last time I have checked, not all patches were included -- for instance
that "don't error every few minutes on empty mailsync" patch still isn't
applied in Minsky's repo?


Kind regards,
D.

--
NAME:Dinko.kreator.Korunic   DISCLAIMER:Standard.disclaimer.applies
ICQ:16965294        JAB:[hidden email]        PGP:0xEA160D0B
HOME:http://dkorunic.net    QUOTE:Eat.right.stay.fit.and.die.anyway


_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Kim Minh Kaplan
Dinko Korunic writes:

> On Fri, Jul 03, 2009 at 06:07:19PM +0000, Kim Minh Kaplan wrote:
>> Now that my patches have been included in his repository,
>> the only reason I publicize mine is so that people who do not want to
>> bother with Mercurial can download an archived (tar.gz) version easily.
>
> Last time I have checked, not all patches were included -- for instance
> that "don't error every few minutes on empty mailsync" patch still isn't
> applied in Minsky's repo?

He explained that this was the wrong solution┬╣.  The error message is
there on purpose and the right solution is to add some mail partner in
your mailsync file.

Kim Minh.
┬╣ http://lists.gnu.org/archive/html/sks-devel/2009-05/msg00002.html


_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: On-demand SKS Statistics Generation?

Peter Pramberger
In reply to this post by Kim Minh Kaplan
Kim Minh Kaplan schrieb am 03.07.2009 10:21:
> Eventloop does not give you access to the event queue.  Attached is a
> kludge to do that.  I do *not* think this is a nice solution.  Better
> would be to add signal handling semantic to eventloop.ml.

Sorry, I somehow missed your reply. Yes, this one works, thanks.


Br,
Peter




_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel