Keyserver Network Down?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Keyserver Network Down?

Matthew Walster-2
Hello all,

I was getting alerts for large amounts of IO Wait on my server, I restarted it and now I'm getting a lot of failures in syslog from sks, either recon client timeouts / connection refused, or the key fetcher receiving 502 (Bad Gateway) from servers.

The keyserver status page seems broken also: https://sks-keyservers.net/status/

Is there some kind of mass breakage occurring with people's sks installs at the moment?

Matthew Walster
(sysop: keyserver.waffle.sexy)

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Keyserver Network Down?

Kristian Fiskerstrand-6
On 06/19/2018 10:53 PM, Matthew Walster wrote:
> The keyserver status page seems broken also:
> https://sks-keyservers.net/status/

This was an intermittent failure, should be back up now.. Needed to
shift around some primaries to bootstrap the crawler.

--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
Aurum est Potestas
Gold is power


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Keyserver Network Down?

Kristian Fiskerstrand-6
On 06/19/2018 11:09 PM, Kristian Fiskerstrand wrote:
> On 06/19/2018 10:53 PM, Matthew Walster wrote:
>> The keyserver status page seems broken also:
>> https://sks-keyservers.net/status/
>
> This was an intermittent failure, should be back up now.. Needed to
> shift around some primaries to bootstrap the crawler.
>

That said, looks to be very high activity towards my cluster atm, which
was why it timed out on my own server initially during last search,
seems more than 37k hosts requesting keyblocks just from my server
today, so might have to spin up a couple more nodes in the cluster.

--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"If you don't drive your business, you will be driven out of business"
(B. C. Forbes)


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Keyserver Network Down?

Kristian Fiskerstrand-6
On 06/19/2018 11:17 PM, Kristian Fiskerstrand wrote:

> On 06/19/2018 11:09 PM, Kristian Fiskerstrand wrote:
>> On 06/19/2018 10:53 PM, Matthew Walster wrote:
>>> The keyserver status page seems broken also:
>>> https://sks-keyservers.net/status/
>>
>> This was an intermittent failure, should be back up now.. Needed to
>> shift around some primaries to bootstrap the crawler.
>>
>
> That said, looks to be very high activity towards my cluster atm, which
> was why it timed out on my own server initially during last search,
> seems more than 37k hosts requesting keyblocks just from my server
> today, so might have to spin up a couple more nodes in the cluster.
>
Seems to be a very high request for mongodb release key, so forcing
caching on the front-end helps relaxing SKS quite a bit, see

https://www.nginx.com/blog/nginx-caching-guide/
https://www.digitalocean.com/community/tutorials/understanding-nginx-http-proxying-load-balancing-buffering-and-caching

some hints
   proxy_cache backcache;
   proxy_ignore_headers Cache-Control "Expires";
   proxy_cache_valid any 30m;


--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
Fabricando fit faber
Practice makes perfect


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Keyserver Network Down?

Phil Pennock-17
On 2018-06-20 at 00:35 +0200, Kristian Fiskerstrand wrote:
> Seems to be a very high request for mongodb release key, so forcing
> caching on the front-end helps relaxing SKS quite a bit, see

Last Friday I investigated and while adjusting my logging to capture the
HTTP verb/method, accidentally logged the entire request for a duration
of 1 minute 33 seconds.  In that time, of the 324 requests logged, two
two most frequent were:

  142  MongoDB 3.4 Release Signing Key <[hidden email]>
   10  MongoDB 3.2 Release Signing Key <[hidden email]>

That's almost half the traffic for keys related to one project.

This tied into an eyeball-visible (might be illusion, I don't have
insight analysis tools set up) swing of presented hostnames towards
`keys.gnupg.net` (the common CNAME) instead of the normal pool name.

I walked away from computers for the weekend, for various reasons, but
realizing how much I was spending out of my own pocket now just to
support some company doing stupid things with key distribution factored
into it.  I think there's still some value to running a keyserver so I'm
not ready to give up yet, but I came this --->.<--- close on Friday.

-Phil

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel