Query UI which offers all servers in the SKS mesh

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Query UI which offers all servers in the SKS mesh

Phil Pennock-17
The server query interface at http://sks.spodhuis.org/ now automatically
adds a new list drop-down, if JavaScript is enabled in your browser,
letting you choose any server in the SKS mesh to query against.  No
JavaScript, no list of alternative servers, and the UI degrades down
cleanly.

Note that the hostnames used are those configured into the sks instances
as their own hostname, so not all are valid in public DNS; "antix",
"basket" and "booboo" come to mind.  The default server is the one
running on the host the JS was loaded from.

This relies on my mesh spidering service to deliver the list of
hostnames as JSON.  If the service has recently restarted then the
complete mesh won't be available, so instead a fallback list will be
used, which is just the hosts in my membership file.  [503 leads to
second AJAX query]  If the results are all valid public DNS, then this
is the likely cause.

Some minor Python additions to the mesh daemon and a little JavaScript
and this was fairly easy, but I'm certainly not a professional JS
programmer and it probably shows.

Copyright status: I don't know the various CC options and am not in the
mood to explore legalese right now, so if you want to copy the
JS/HTML, go ahead, but just let me know in a private mail please,
especially if you fix the AJAX callback URL to be my generated list,
instead of something on your server.

Thoughts, comments, feedback?

Regards,
-Phil

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

attachment0 (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Query UI which offers all servers in the SKS mesh

Sebastian Urbach
On Tue, 28 Dec 2010 01:11:59 -0500
Phil Pennock <[hidden email]> wrote:

Hi Phil,

> The server query interface at http://sks.spodhuis.org/ now automatically
> adds a new list drop-down, if JavaScript is enabled in your browser,
> letting you choose any server in the SKS mesh to query against.  No
> JavaScript, no list of alternative servers, and the UI degrades down
> cleanly.

Im impressed, nice Job Phil ! Thats definitely a great feature, so
nobody have to hop from one key-server to another to check out if his
or her key is known publicly on all servers. And it seems also nice for
the group-spirit within the sks-network :-)

> Note that the hostnames used are those configured into the sks instances
> as their own hostname, so not all are valid in public DNS; "antix",
> "basket" and "booboo" come to mind.  The default server is the one
> running on the host the JS was loaded from.

Nothing to complain, everything in order.

> This relies on my mesh spidering service to deliver the list of
> hostnames as JSON.  If the service has recently restarted then the
> complete mesh won't be available, so instead a fallback list will be
> used, which is just the hosts in my membership file.  [503 leads to
> second AJAX query]  If the results are all valid public DNS, then this
> is the likely cause.

If i may contribute an idea, wouldn't it be better to use the latest
available list as a backup and not just your membership file when the
service needs to be restarted ?

> Some minor Python additions to the mesh daemon and a little JavaScript
> and this was fairly easy, but I'm certainly not a professional JS
> programmer and it probably shows.

It works and it's a great feature, that's what matters to me.

> Copyright status: I don't know the various CC options and am not in the
> mood to explore legalese right now, so if you want to copy the
> JS/HTML, go ahead, but just let me know in a private mail please,
> especially if you fix the AJAX callback URL to be my generated list,
> instead of something on your server.

I will definitely try it :-) Still need a private mail ?

--

Mit freundlichen Gruessen / yours sincerely

Sebastian Urbach

----------------------------------------------------------
One word of truth shall outweigh the whole world
----------------------------------------------------------
Alexander Issajewitsch Solschenizyn (1918 - 2008)
Mathematiker, Schriftsteller und Nobelpreisträger

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Query UI which offers all servers in the SKS mesh

Phil Pennock-17
In reply to this post by Phil Pennock-17
On 2010-12-28 at 01:11 -0500, Phil Pennock wrote:
> Note that the hostnames used are those configured into the sks instances
> as their own hostname, so not all are valid in public DNS; "antix",
> "basket" and "booboo" come to mind.  The default server is the one
> running on the host the JS was loaded from.

Update: this was silly, it now only includes those which both resolve in
DNS and had a 'version' on the last spider iteration.  The spider
defaults to walking once an hour, so this should be fairly up-to-date at
any given time.

-Phil

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

attachment0 (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Query UI which offers all servers in the SKS mesh

Phil Pennock-17
In reply to this post by Sebastian Urbach
On 2010-12-28 at 08:49 +0100, Sebastian Urbach wrote:
> If i may contribute an idea, wouldn't it be better to use the latest
> available list as a backup and not just your membership file when the
> service needs to be restarted ?

That would involve preserving more state, which I've never yet done as
anything other than a debugging aid, because it means the state format
needs to be preserved yet still have enough data to be ready to use on
reload, even across new versions which might expect other data to be
present.  That's not hard, but is constraining and so I've not done it
on this toy service.  I absolutely would do so on a "production"
service.

Doing things the way I did means that I can just extract the data which
is already present, which means serving up data that's in RAM already.

> I will definitely try it :-) Still need a private mail ?

That counts, so no.  :)

If the service ever gets so loaded down that I notice, I'll just shove
in Referer: header checks and only whitelist those who let me know ahead
of time.

-Phil

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

attachment0 (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Query UI which offers all servers in the SKS mesh

Phil Pennock-17
In reply to this post by Phil Pennock-17
On 2010-12-28 at 01:11 -0500, Phil Pennock wrote:
> Some minor Python additions to the mesh daemon and a little JavaScript
> and this was fairly easy, but I'm certainly not a professional JS
> programmer and it probably shows.

I've shoved the latest version of the Python up onto:
  http://people.spodhuis.org/phil.pennock/software/
together with a PGP signature on that.  "sks_peers.py-v258"

Note that the code is grotty and organically grown and with crufty
corners.  I'm not proud of it, but it works.  Needs Python 2.6 and
various modules to work, some of which have ... "unstable" APIs.
pygraph >= 1.7.0 is needed and if you're much more recent than that,
things have probably broken again.  *sigh*

-Phil

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

attachment0 (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

New search options question (Was: Query UI which offers ...)

Gabor Kiss
In reply to this post by Phil Pennock-17
> The server query interface at http://sks.spodhuis.org/ now automatically
> adds a new list drop-down, if JavaScript is enabled in your browser,
> letting you choose any server in the SKS mesh to query against.  No
> JavaScript, no list of alternative servers, and the UI degrades down
> cleanly.

I checked sks.spodhuis.org and I'm interested in search options
- "Restrict to Exact matches",
- "Clean results disable" and
- "Machine-readable"
rather than JS. :-)

These are new to me and I guess they are version 1.1.1 specific.
I wonder what the first two do? I could not see any difference
in the results.
Browsing source does not help me, I'm dummy to OCAML. :-)

And where did you get your index.html from?
Source tarball nor contains it neither instructs you how to create.
Had you a vision? :)))

Cheers

Gabor
--
Most maszik ki a majom a vizbol.

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: New search options question (Was: Query UI which offers ...)

Phil Pennock-17
On 2010-12-28 at 09:42 +0100, Gabor Kiss wrote:

> > The server query interface at http://sks.spodhuis.org/ now automatically
> > adds a new list drop-down, if JavaScript is enabled in your browser,
> > letting you choose any server in the SKS mesh to query against.  No
> > JavaScript, no list of alternative servers, and the UI degrades down
> > cleanly.
>
> I checked sks.spodhuis.org and I'm interested in search options
> - "Restrict to Exact matches",
> - "Clean results disable" and
> - "Machine-readable"
> rather than JS. :-)
>
> These are new to me and I guess they are version 1.1.1 specific.
> I wonder what the first two do? I could not see any difference
> in the results.
> Browsing source does not help me, I'm dummy to OCAML. :-)
>
> And where did you get your index.html from?
> Source tarball nor contains it neither instructs you how to create.
> Had you a vision? :)))

None of those options are new in 1.1.1.  I found them by browsing the
SKS source-code, back around the time that I set up my keyserver.

I wrote the web-page from scratch, after looking at some other pages but
not being able to find copyright info and deciding that I'd rather write
something clean in modern HTML and explore the source to make sure I got
*all* the options.

The unhelpful answer is that the first two items will set "exact" to
true and "clean" to false in the generated "request"-type object.  That
there was a knob was enough for me to want to twist it to see what
happens.  :)

More helpfully: if the "clean" CGI param is set false, then a filter
will not be applied to the keys when trying to find results; that filter
drops bad sigs and, I *think* drops missing self-sigs, but I'm not sure.

I don't think that the "exact" option is actually used.  It sets a field
but that's never referenced.  It might be used at some point.

The specification for the various fields used in the PKS protocol is at:
  http://www.mit.edu/afs/net.mit.edu/project/pks/thesis/paper/thesis.html

Heh.  "Today's key servers manage over twenty thousand public keys";
compare to the almost 2.9 million keys in an SKS keyserver at the time
that I write this.  :)

-Phil

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: New search options question (Was: Query UI which offers ...)

Gabor Kiss
> More helpfully: if the "clean" CGI param is set false, then a filter
> will not be applied to the keys when trying to find results; that filter
> drops bad sigs and, I *think* drops missing self-sigs, but I'm not sure.
>
> I don't think that the "exact" option is actually used.  It sets a field
> but that's never referenced.  It might be used at some point.

Thanks for the reply. :-)

Gabor

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: New search options question

Kim Minh Kaplan
In reply to this post by Phil Pennock-17
Phil Pennock:

>   http://www.mit.edu/afs/net.mit.edu/project/pks/thesis/paper/thesis.html

Another resource you might find useful is the expired draft: The OpenPGP
HTTP Keyserver Protocol (HKP)
https://datatracker.ietf.org/doc/draft-shaw-openpgp-hkp/
--
Kim Minh

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: New search options question

Christoph Anton Mitterer-2
On Tue, 2010-12-28 at 20:56 +0000, Kim Minh Kaplan wrote:
> https://datatracker.ietf.org/doc/draft-shaw-openpgp-hkp/
SKS does not "conform" to it in several places,... I've opened issues
for the respective cases at google code some time ago.


Cheers,
Chris.

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: New search options question

Phil Pennock-17
On 2010-12-29 at 22:44 +0100, Christoph Anton Mitterer wrote:
> On Tue, 2010-12-28 at 20:56 +0000, Kim Minh Kaplan wrote:
> > https://datatracker.ietf.org/doc/draft-shaw-openpgp-hkp/
> SKS does not "conform" to it in several places,... I've opened issues
> for the respective cases at google code some time ago.

[ Before continuing, I'll emphasize here that this is merely personal
  opinion and my connection to SKS is that I use it, have offered
  patches and wrote a little documentation, I'm not a maintainer and
  speak in no official capacity. ]

It's a draft and somewhat dated.  The author's own software no longer
conforms to it in some areas (eg, the SRV service name to use; GnuPG now
uses two of the DNS-SD variants instead).

Most of the issues are of the x-<name> vs <name> variety, which in my
opinion is nit-picking of the highest order when the reference is a -00
version personal submission expired draft.  It's *normal* for people to
ignore such x-restrictions on naming at such an early stage, while
people get experience with the protocol and refine the draft.

options=mr being ignored with op=get is certainly a valid complaint.

It only makes sense for options=nm to raise an error if the server does
perform any modifications.  I don't believe that SKS does, so it's by
definition always complying with such a request.  Thus an error would be
incorrect.

MIME-types -- the built-in web-server is supposed to be a very minimal
implementation.  I strongly suggest that if you want anything
non-trivial served, you do so with a proper threaded/forking server, to
be able to sustain load?  Options are for SKS on port 11371 and a
default document that just does a redirect to port 80 serving, or for
SKS behind a proxying server on port 80 with the proxy providing SSL,
etc, and *only* passing on ^/pks/.* to SKS.

-Phil

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: New search options question

Christoph Anton Mitterer-2
Hey...


On Wed, 2010-12-29 at 19:39 -0500, Phil Pennock wrote:
> It's a draft and somewhat dated.
Of course...
(btw: IMHO it regrettably that we have no real standard document for
HKP)


>   The author's own software no longer
> conforms to it in some areas (eg, the SRV service name to use; GnuPG now
> uses two of the DNS-SD variants instead).
I haven't tracked gnupg in that manner recently,.. but would consider it
sad if it no longer supports the SRV way,... which is IMHO much more
"plain" DNS than any zeroconf stuff.


> Most of the issues are of the x-<name> vs <name> variety, which in my
> opinion is nit-picking of the highest order when the reference is a -00
> version personal submission expired draft.  It's *normal* for people to
> ignore such x-restrictions on naming at such an early stage, while
> people get experience with the protocol and refine the draft.
Well it might sound nit-picking,.. but things like these are what cause
many standardisation problems in other fields, as people think "well
it's ok not to follow the standard here for now",... but later it turns
out that things got so widespread and one has to keep legacy stuff
forever (just take the http content encoding headers as example).


> It only makes sense for options=nm to raise an error if the server does
> perform any modifications.  I don't believe that SKS does, so it's by
> definition always complying with such a request.  Thus an error would be
> incorrect.
If it really doesn't, then it's ok to silently ignore it, I guess. But
one should not forget this, in case code is ever added that _does_
modify keys.


> MIME-types -- the built-in web-server is supposed to be a very minimal
> implementation.  I strongly suggest that if you want anything
> non-trivial served, you do so with a proper threaded/forking server, to
> be able to sustain load?  Options are for SKS on port 11371 and a
> default document that just does a redirect to port 80 serving, or for
> SKS behind a proxying server on port 80 with the proxy providing SSL,
> etc, and *only* passing on ^/pks/.* to SKS.
That's what I plan to do,... put it behind an Apache, an forcibly set
the headers there.

I mean the request for this was rather a would-be-nice-to-have.


Cheers,
Chris.

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: New search options question

Phil Pennock-17
On 2010-12-30 at 02:52 +0100, Christoph Anton Mitterer wrote:
> On Wed, 2010-12-29 at 19:39 -0500, Phil Pennock wrote:
> >   The author's own software no longer
> > conforms to it in some areas (eg, the SRV service name to use; GnuPG now
> > uses two of the DNS-SD variants instead).
> I haven't tracked gnupg in that manner recently,.. but would consider it
> sad if it no longer supports the SRV way,... which is IMHO much more
> "plain" DNS than any zeroconf stuff.

I was unclear: DNS-SD also uses SRV records, but uses different Service
names (as in _Service._Protocol.example.org) and GnuPG has switched to
those.

Thus:
% dig +short -t srv _hkp._tcp.spodhuis.org
10 10 11371 sks.spodhuis.org.
% dig +short -t srv _pgpkey-http._tcp.spodhuis.org
10 10 11371 sks.spodhuis.org.
% dig +short -t srv _pgpkey-https._tcp.spodhuis.org
0 0 0 .

Regards,
-Phil

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: New search options question

Mike O'Connor-5
Hi All

So does this mean I should be setting up these dns records for
keyserver.oeg.com.au ?

Mike

On 30/12/10 1:18 PM, Phil Pennock wrote:

> On 2010-12-30 at 02:52 +0100, Christoph Anton Mitterer wrote:
>> On Wed, 2010-12-29 at 19:39 -0500, Phil Pennock wrote:
>>>   The author's own software no longer
>>> conforms to it in some areas (eg, the SRV service name to use; GnuPG now
>>> uses two of the DNS-SD variants instead).
>> I haven't tracked gnupg in that manner recently,.. but would consider it
>> sad if it no longer supports the SRV way,... which is IMHO much more
>> "plain" DNS than any zeroconf stuff.
> I was unclear: DNS-SD also uses SRV records, but uses different Service
> names (as in _Service._Protocol.example.org) and GnuPG has switched to
> those.
>
> Thus:
> % dig +short -t srv _hkp._tcp.spodhuis.org
> 10 10 11371 sks.spodhuis.org.
> % dig +short -t srv _pgpkey-http._tcp.spodhuis.org
> 10 10 11371 sks.spodhuis.org.
> % dig +short -t srv _pgpkey-https._tcp.spodhuis.org
> 0 0 0 .
>
> Regards,
> -Phil
>
> _______________________________________________
> Sks-devel mailing list
> [hidden email]
> http://lists.nongnu.org/mailman/listinfo/sks-devel


_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: New search options question

Daniel Kahn Gillmor-7
On 12/30/2010 04:38 PM, Mike O'Connor wrote:
> So does this mean I should be setting up these dns records for
> keyserver.oeg.com.au ?

I don't think that's necessary for individual keyservers (though i don't
think it would hurt either).  IIUC, the DNS-SD records are mostly used
by pools to distribute and weight their members.

        --dkg


_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (918 bytes) Download Attachment