reverse proxies and the pool

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

reverse proxies and the pool

Kristian Fiskerstrand-6
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

With a great number of the SKS servers already in the pool now
supporting a reverse proxy[a] does it make sense to make this a
hard-requirement for inclusion in the pool in order to increase
availability?

Ideally, if network traffic should increase, it could be interesting
to setup a new subpool (to replace the current HA - High Availability
pool) that only include load-balanced setups with multiple SKS servers
behind a single reverse proxy.

What are your thoughts about such a move?

[a] please follow the Peer recommendations and allow every Host:
connection on 11371 to go through to SKS, otherwise it will break e.g.
keys.gnupg.net.

- --
- ----------------------------
Kristian Fiskerstrand
Blog: http://blog.sumptuouscapital.com
Twitter: @krifisk
- ----------------------------
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
- ----------------------------
"The power of accurate observation is commonly called cynicism by
those who have not got it."
George Bernard Shaw
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.1.0-beta255 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJSbra4AAoJEAt/i2Dj7frjC1AP/0UZvXtstMbwxrkueKoR1HXw
NFLqc2QLrVygHBtZgPSqHpweNInihympvCLWl5TnsyYilQTIpMdUy9fXbi8X3u9J
MS8+T9a3ujREda1PPGLn1+H7QIGfq9zDkZCoCPnkzGF9D3O07glKxFjQVwZEfPnQ
mfC9ogHGzM8IpYpPk48d73nXMDefRUApe4YpppV8aI0T9JeCVf8IcQShsdeBsgzF
lfgdMCo+o6N/0jmxkmXTXBXPtWUOOaFT0iJA67khhy5foQPKxRWquF7j8VpWxcY2
zex16lziHDjuYsjGutAszxVycaXQ9U6h3aFaOvYu2hJ7UpIGh3mNx2HoJw+M1p7n
UhRHcndfOq4ilk8+ieQ7bYIgRYiwGQA+RLC3s5cN0H1NF1tWJOFLe119R8irx1Wq
vuwXPN/zkeAcvL4fML3noaBlsuQ9pmmkgKJsun9jBgYljLJkypepiY6IbQEoHAkl
kRjKGh/Jtvwtg/U6WgkNhH1NNNV7PNdX1cAFPy/otACGStLjURWy/G8UA7i7hNl0
jE1PW1X8JBRjGBKSDMDH3TfIohUUfmab73AJsrGwv+4rVupTJM6eEUxzp/piFjVC
HJTJ1ObNaLXqd75Wm31D1yIVp2pt/lBjD9DeyyfJVwlHDLZ77qupVr4I5EdWfNiD
l5XO4Na+4SO2wChAESFd
=65y6
-----END PGP SIGNATURE-----

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Daniel Kahn Gillmor-7
On 10/28/2013 03:10 PM, Kristian Fiskerstrand wrote:
> With a great number of the SKS servers already in the pool now
> supporting a reverse proxy[a] does it make sense to make this a
> hard-requirement for inclusion in the pool in order to increase
> availability?

I like this idea.  We have a lot of servers that implement this practice
(with different reverse proxies).  currently, if a query to the pool
happens to hit one of the non-reverse-proxied servers that is currently
stuck serving a slow client, it makes it look like "the pool is broken".
  We'd avoid those problems by having the pool just refer to those
systems with higher-availability setups.  Non-proxied servers would
continue to work, and to gossip with the rest of the network as
expected, but they would not be subject to the load coming from the
clients who use the pool.

So I would be happy to see this change.

For those operating sks keyservers without a reverse proxy, but who want
to start using one, please see the examples and documentation at:

https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering#!http-performance

Regards,

        --dkg

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Kiss Gabor (Bitman)
In reply to this post by Kristian Fiskerstrand-6
> With a great number of the SKS servers already in the pool now
> supporting a reverse proxy[a] does it make sense to make this a
> hard-requirement for inclusion in the pool in order to increase
> availability?

1 vote against it. (Sorry if I seem to be ungrateful. :)

> Ideally, if network traffic should increase, it could be interesting
> to setup a new subpool (to replace the current HA - High Availability
> pool) that only include load-balanced setups with multiple SKS servers
> behind a single reverse proxy.
>
> What are your thoughts about such a move?

I already explicated that the main vulnerability of key servers is
not a temporary network overload at socket level. Guys at No Such Agency
once decide to flood the servers with one hundred million fake keys
with ardent help of several governments of Near, Middle and Far East.

Gabor

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Daniel Kahn Gillmor-7
On 10/28/2013 03:25 PM, Gabor Kiss wrote:

> 1 vote against it. (Sorry if I seem to be ungrateful. :)

can you explain more why?

>> Ideally, if network traffic should increase, it could be interesting
>> to setup a new subpool (to replace the current HA - High Availability
>> pool) that only include load-balanced setups with multiple SKS servers
>> behind a single reverse proxy.
>>
>> What are your thoughts about such a move?
>
> I already explicated that the main vulnerability of key servers is
> not a temporary network overload at socket level. Guys at No Such Agency
> once decide to flood the servers with one hundred million fake keys
> with ardent help of several governments of Near, Middle and Far East.

I share your concerns (though maybe without such geographic
specificity), but i'm not sure how they're relevant to the question
being asked.

Is an argument against restricting pool.sks-keyservers.net to
reverse-proxied servers?  or as an argument against creating a new
high-availability subpool of servers that actually run their own
internally load-balanced setups?

I'm not sure how this argument works in either context.  What specific
threat is mitigated by leaving servers that are trivially-DoSable in the
default pool?

        --dkg

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Kiss Gabor (Bitman)
Dear Daniel,

> > 1 vote against it. (Sorry if I seem to be ungrateful. :)
>
> can you explain more why?

These efforts with HA pool reminds me bikeshedding.
Wasting time with unremarkable things.

> I'm not sure how this argument works in either context.  What specific threat
> is mitigated by leaving servers that are trivially-DoSable in the default
> pool?

Strictly speaking you are right.
I just don't agree with this step because I truly think its benefits
are illusory and it is devaluation of efforts of enthusiastic
volunteers who make sacrifies for the public.

(Please excuse me for less diplomatic words. English is not my
natural language.)

I really regret my question about reverse proxies giving Kristian
the idea about shutting out 40% of servers from the pool. :-(

Gabor
--
No smoke, no drugs, no vindoze.

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Phil Pennock-17
On 2013-10-28 at 21:53 +0100, Gabor Kiss wrote:
> These efforts with HA pool reminds me bikeshedding.
> Wasting time with unremarkable things.

If there are three big problems, tackling them one by one while not
tackling the hardest _yet_ is not bikeshedding: it's improving the state
of affairs in manageable chunks, working slowly to gain consensus in a
community project.

Refusing to solve any of the problems because there's one really big
problem outstanding just leaves us in a worse state of affairs.  "The
perfect is the enemy of the good".  Folks working to solve this does not
take away energy from anyone working on solving the really big problem,
does not work at cross-purposes to it and generally is constructive.

> I just don't agree with this step because I truly think its benefits
> are illusory and it is devaluation of efforts of enthusiastic
> volunteers who make sacrifies for the public.

The benefits are a more reliably fast response from PGP keyservers by
the general public using the normal names (eg, "keys.gnupg.net" which is
a CNAME to "pool.sks-keyservers.net".)  This is not illusory: it reduces
frustration and helps PGP usage become more automatic, rather than
something to be disabled because it's causing problems and keeping the
email from getting out.

Anyone can volunteer to run a keyserver, and they can use whatever
configuration they like (as long as they can find peers).  Nobody is
proposing to stop peering with a non-proxied server.  You can use your
own keyserver, and you can encourage your friends to use your keyserver,
and you should do so to help preserve communicant privacy.

At question here is _one_ of the pool-maintainers, who runs the pools
which are "normal", working to change the pool which various people use
as a default, until such time as they know someone they trust to
preserve their privacy and so can choose to use a specific keyserver.

Anyone can run a pool if they disagree with Kristian's decisions;
Kristian's developed a good reputation and is generally trusted, so
folks in other projects go so far as to pick his pool as the default.

Kristian's PHP keyserver-pool software is open sourced:

  https://code.google.com/p/sks-keyservers-pool/

My own keyserver-pool software (Golang, some scripting for DNS) is open
source:

  https://github.com/philpennock/sks_spider

So that's two available code-bases, in a choice of languages, for anyone
to run a pool and compete in an open market for mindshare, by running
the best service which they think people will like.

I, for one, think that it's good that there is a population of
keyservers with different policies which different people can layer
pools over the top of, for the public good.  I think it's good that
there's no cabal forcing certain behaviour, just some guidance which
most people follow because it makes life easier.  Any forced
restrictions are purely technical (eg, stopping peering with software
which is so old that peering is just broken).

I think that the goal of making the default serving pool be as fast and
responsive as possible, without one slow client affecting every other
user, is a good one.  I think that switching the default to HA should
happen at some point, the only question is "when".

If we have enough reverse-proxy servers to provide decent latency to
every part of the world, then "any time after now" seems to be a
technically sound choice.  So, it seems to me that asking for input,
and if that has a rough consensus of "yes" then providing advance
notice, then making the change, is a very open and clear way to proceed
in how Kristian runs the volunteer service which _he_ runs.

To the extent that anyone other than Kristian has a vote as anything
more than a courtesy: I vote yes, it would be good for the main pool to
be HA-only, with a sub-pool for non-ha perhaps, and I think that a one
month lead-time would be very generous, giving people who want to stay
in the default pool but who haven't deployed a reverse proxy yet plenty
of time to do so.

-Phil

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

attachment0 (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Kristian Fiskerstrand-6
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 10/28/2013 11:00 PM, Phil Pennock wrote:
> On 2013-10-28 at 21:53 +0100, Gabor Kiss wrote:
>> These efforts with HA pool reminds me bikeshedding. Wasting time
>> with unremarkable things.
>
> If there are three big problems, tackling them one by one while
> not tackling the hardest _yet_ is not bikeshedding: it's improving
> the state of affairs in manageable chunks, working slowly to gain
> consensus in a community project.

Thank you Phil for a (as usual) very good overview of the situation.

...

>
> I think that the goal of making the default serving pool be as fast
> and responsive as possible, without one slow client affecting every
> other user, is a good one.  I think that switching the default to
> HA should happen at some point, the only question is "when".
>
> If we have enough reverse-proxy servers to provide decent latency
> to every part of the world, then "any time after now" seems to be
> a technically sound choice.

Current snapshot shows that 45 of 76 servers in the active pool are
identified as being behind a reverse proxy, being roughly 60%. This
includes nearly all of the servers that are included in the
geographical pools based on a more calculated approach[0]. In
comparison we only had some 30-odd servers directly qualifying when I
first started looking into setting the minimum requirement of the pool
to 1.1.3[1], at the time of the actual switch another 10-15 operators
had upgraded, and I believe the pool results are better for it today.

So, it seems to me that asking for input,

> and if that has a rough consensus of "yes" then providing advance
> notice, then making the change, is a very open and clear way to
> proceed in how Kristian runs the volunteer service which _he_
> runs.
>
> To the extent that anyone other than Kristian has a vote as
> anything more than a courtesy: I vote yes, it would be good for the
> main pool to be HA-only, with a sub-pool for non-ha perhaps, and I
> think that a one month lead-time would be very generous, giving
> people who want to stay in the default pool but who haven't
> deployed a reverse proxy yet plenty of time to do so.
>

Indeed, a months time for implementing a reverse proxy should be
sufficient for most that has a strong desire to stay in the active
pool. And I'd like to further emphasize that even if anyones server
isn't in the active pool in the front, facing clients, it is still
valuable for the pools ability to stay stable and synchronize.
Steering traffic towards the servers that are most responsive and
reliable is however the primary purpose of the pool.

PS! For those that have noticed a blue indicator on that status
page[2], this is a preliminary setup for a potential new HA pool in
the future for load-balanced servers in front of multiple SKS
instances. I do however expect the HA pool to continue in the same
manner as today for a while longer before that change happens. If
anyone is interested in my own load-balanced setup using nginx I've
written up a blog post on [3].

References:
[0] http://kfwebs.com/sks-keyservers-SRV.pdf
[1] http://lists.nongnu.org/archive/html/sks-devel/2012-06/msg00043.html
[2] http://sks-keyservers.net/status/
[3] http://blog.sumptuouscapital.com/2013/10/load-balancing-sks/
- --
- ----------------------------
Kristian Fiskerstrand
Blog: http://blog.sumptuouscapital.com
Twitter: @krifisk
- ----------------------------
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
- ----------------------------
"Be a yardstick of quality. Some people aren't used to an environment
where excellence is expected."
(Steve Jobs)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.1.0-beta255 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJSbuYfAAoJEAt/i2Dj7frjfTsP/16odgd0TdNoqTbbUFgxOs/A
kqWy8TW3FnMb2hiYj3ylBNAxnhQTr99da+qUmzNdCvvF6arwJqtJF8uSNanbnxN0
YEYmPglZ/33F8hDKkitYlzHSDUeY/azx3z5oPiLL4BWdnBbLSkhnJDjvoKiL/BKS
y5n+oHiZeVmXbVJB98bxqgnoxCJCwceGRkbYzALacDBdRn3K6+UA6VDvJNIn0AYj
qjhXVI44bEQfKLCBAnAzbhVhkJ3xxBYLbd/wIwtj4Qd8t8YeAOm2GRSk7LoIJsBL
qpbTMBpBMqR4dpK8kk+DIIarL58xiPg87Fa7o/Jg7N1wlWBuHTRLoqpWzeibPq2K
xULtZ9MfyzTYBOrRDUMsw4T8jgjWOHrXIV+stbcoF84x0O41YkrZRn9/HQXdRfQe
DIJriL7g98AG4Uk74JVQ5kItk24w4LNkkBmmd2Ubp3p//qBm4HVk90EE2vuPI8RO
SpuycObyIp5K1h2tLW7nm9lRtRk+hIqz+hUsQKIVZDZsz1Ebb4RHNxhUh/a2qY89
dEPm2qKGU7FRuygYYmYiIG1JvqMmeh/wEd7/UIxyHrPmDV9nyH6motOg/xV+WHcE
i5pqmXVuXm0rlR5prWaKRNJ3TZSFNhccM3Ks7k2OSGqk+Z9gROBxSfb9lIcKG2iH
XHUFd1qTbejuAJrSCTvG
=qi4f
-----END PGP SIGNATURE-----

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Andy Ruddock-2
In reply to this post by Kristian Fiskerstrand-6
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Kristian Fiskerstrand wrote:

> Hi,
>
> With a great number of the SKS servers already in the pool now
> supporting a reverse proxy[a] does it make sense to make this a
> hard-requirement for inclusion in the pool in order to increase
> availability?
>
> Ideally, if network traffic should increase, it could be
> interesting to setup a new subpool (to replace the current HA -
> High Availability pool) that only include load-balanced setups with
> multiple SKS servers behind a single reverse proxy.
>
> What are your thoughts about such a move?
>
> [a] please follow the Peer recommendations and allow every Host:
> connection on 11371 to go through to SKS, otherwise it will break
> e.g. keys.gnupg.net.

Whatever the decision, could you provide documentation for
configuration of such a reverse proxy for both Apache and Nginx?

- --
Andy Ruddock
- ------------
[hidden email] (OpenPGP Key ID 0xB0324245)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCgAGBQJScU5yAAoJECqtbbewMkJFStQP/icQmbBcpG9jVyg2RhfhLpGO
L0zSjAGQ7xJbjJlr6EklTxqyHTnAVDJSU21fFHDBcFDzoFxMQvRYXyfVNmTLPtQW
FGru6don1Phgo1W3opk3i4TUG3aVotEA65xANWABdcJ6XO6QiJNHmBJ/d0nr7cII
N5461Az/NkEBkS4OgCA8Bn6Bj+neuwXB6RNKsho0pRDq2/wkNShLAx/hSUyRF8+1
xlcmYCrjjSXf1FYCMvNCSvN6SWsKlHnro5AOUvb3aQZW/cl+9kr3POK0htga2kaT
co4mRZjosZh95Mf9S5JNybWCxzo17XqU14jtD6PhnwKAomENGW8Xbbc4066uWlT8
9LudM0POqeaEF1yzvFgliyI84GiEsGOHyE9OnihbMF5LM5i+K/4OIsaziCg1fkTC
u047oaP+bXR+hl79BM2NKkIoKMa5UHONFsNIcr2vG+60yWyEFQIVKT4T7n307k2e
Qt8+1Kds387tqVsUm8gYNjBmULvI9tQyZBPRELR4nt4XTCAYXQ74kOnNq8U9dRIe
T/WBVJVN/zTxcZHH46jP4ykYO+z+pyI2/TCVD6He83Ga5VP9QCeEFAwtvKyWCMvs
GFD2WjtYV18h+V9LEXCH/v7xuvQnocQblMx/2acYC9WmXvnqY5vFbUDkRK/ROyjw
t8OzdU2hrRGvLoAbthfG
=p7/+
-----END PGP SIGNATURE-----

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Daniel Kahn Gillmor-7
On 10/30/2013 02:22 PM, Andy Ruddock wrote:

> Whatever the decision, could you provide documentation for
> configuration of such a reverse proxy for both Apache and Nginx?

As i mentioned earlier up-thread:

For those operating sks keyservers without a reverse proxy, but who want
to start using one, please see the examples and documentation at:

https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering#!http-performance

Regards,

        --dkg

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Kiss Gabor (Bitman)
In reply to this post by Andy Ruddock-2
> Whatever the decision, could you provide documentation for
> configuration of such a reverse proxy for both Apache and Nginx?

What I miss is a set of diagnostic procedures/recipes that could
help an operator to figure out if his server fits various requirements.

Like this was on Monday:

| Virtualhost-related, no match found
|
| address@hidden ~ $ curl -H'Host: p80.pool.sks-keyservers.net' "http://keys.niif.hu/pks/lookup?op=stats";
| <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
| <html><head>
| <title>404 Not Found</title>
| </head><body>

Gabor

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Kristian Fiskerstrand-6
In reply to this post by Andy Ruddock-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 10/30/2013 07:22 PM, Andy Ruddock wrote:
> Kristian Fiskerstrand wrote:
>> Hi,
>
>>

..



>> [a] please follow the Peer recommendations and allow every Host:
>>  connection on 11371 to go through to SKS, otherwise it will
>> break e.g. keys.gnupg.net.
>
> Whatever the decision, could you provide documentation for
> configuration of such a reverse proxy for both Apache and Nginx?
>
>

You'll find this in the wiki[0], and for load balanced nginx I've also
written a post on [1].

References:
[0] https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering
[1] http://blog.sumptuouscapital.com/2013/10/load-balancing-sks/

- --
- ----------------------------
Kristian Fiskerstrand
Blog: http://blog.sumptuouscapital.com
Twitter: @krifisk
- ----------------------------
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
- ----------------------------
Testis unus, testis nullus
A single witness is no witness
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.1.0-beta255 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJScVFSAAoJEAt/i2Dj7frjGVYP/3wBjkK+H03mWaYmRFCBz1jH
1lz+7bW5FQMfdUtP3XAQ2Bi7zW7aoos+8v6hDAPJCPf47cBVZGK+hVJD2zRLxwZZ
y0870sAW+73eMGeVkSuehZhbnk+Rsh1P+0MtM1PwnauqeFNQ20AXTsH481F6pXdp
NkYu4CiABp9w1YQgIroQ/wZTO/Q/oq0dOihaBSeaby3X+ks572BmkDX1ZqLjqZel
d9NijlGbeUPYcPsYUoOZCuVoJwTEfIwDwWHq+AVLLNcDCar8Rs0MgQ0zFqAxfF7j
6q/0brlI2DwrZfPaSrcDgQ+ideVqiISFsEvo42nX8yuPbSnJ2DR6oTI1VMmaFArS
xAvJktjcc9YUMS33B9YZpPHLVry7pQPYbeTxK3yCjJjASPNeMEbqCcpNz+wJAZYq
8rkQpyPVzb+v+ROVkttRb4zYuKtAYl+m+0Jl8/N+COAyc/9T7FTiGROH8ETsYZ7R
fOLP5eDfUbSv0yM0CBmd/2SXukseHNol/Na+u2BYWJ09Uf0EIYx/nmS9go3+JQcY
rfM7+94sVR9Q5NiyWinpCK/YVZYOXhFzP6UOVrz35FE59P/bANCF6kRsilHiYjQN
5FdZCmZPxtLeOt60LJjcZj8k58PnJwNtFSTVVzUZPZLZmGmh3K0uxmW4cF2JfAPH
VKmYBBD8S4rb6l70uidk
=fjPw
-----END PGP SIGNATURE-----

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Kristian Fiskerstrand-6
In reply to this post by Kiss Gabor (Bitman)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 10/30/2013 07:31 PM, Gabor Kiss wrote:

>> Whatever the decision, could you provide documentation for
>> configuration of such a reverse proxy for both Apache and Nginx?
>
> What I miss is a set of diagnostic procedures/recipes that could
> help an operator to figure out if his server fits various
> requirements.
>
> Like this was on Monday:
>
> | Virtualhost-related, no match found

Note the wiki[0] clearly stating, for apache config:
"## do *not* set NameVirtualHost on this host:port combination!
## For :11371, we use IP/port virtual-hosting, not names, accepting
## any pool name."


References:
[0] https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering


- --
- ----------------------------
Kristian Fiskerstrand
Blog: http://blog.sumptuouscapital.com
Twitter: @krifisk
- ----------------------------
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
- ----------------------------
Testis unus, testis nullus
A single witness is no witness
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.1.0-beta255 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJScVGyAAoJEAt/i2Dj7frjgGwQAIGvFLOCnq6kT05b6SSWdwu5
Ze62Jyr99v0b0XE2RQJ0Pu6M/EMsVbP2B//rh+IwJQkosBU7K8FUkJgAFtJXHMtl
Ej40vz8fwqbyviT9TctKZs4UEb5PwZ14YzKVVKT0UOiHhd5A8EQTVkdqBSxfRBwA
FFHF3Jkd9RbLYgaZiqehkjFY3ycyPtBTDlQJJ5m+/pCGoy7CLnpSVbDziPL+zVB2
Np5aN5JdMzzfe09yv6UTwx0ZuqYTMjX1zY5UZfHGzUIvGW3hs8QqG77KecmBDoAL
eHOWH7qobtdGAcxUXtqFs7ljJUrPa8wYHcZbH9kK1BS/pgJTsvPMN5eBflJPCjMi
BiXQjOVRRPInTAN+dAysGoDfp7gKdxwihvi+mprRG6mtwnykZEdpB+boTEI3CYXC
7Lp5n8okX6mhqlF2a4l1yhN4/uTeaCHZP1iqUM0XL3yTJLlOvcZTAruvUP5eTToT
c+lsosAfUISL0iAUN1lIz7Pz41zaSrq4AeRH1pvjI/ufvQzhfX79G0MdkXInGT7R
KYbdk3Dwq8Rk5Whhg90UiZsfO27oYoYFahVzN2zLoEb2Zf4asXQMsMg0ge+C2/v/
qNAvs450FT4t6AkD+n+vuA9uuN4X9ZprNjCPies148w58wxvFtK2COEnsFn0jJG5
q1X1i2zARWnMJywas4H1
=4S8m
-----END PGP SIGNATURE-----

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Phil Pennock-17
In reply to this post by Kristian Fiskerstrand-6
-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

On 2013-10-30 at 19:34 +0100, Kristian Fiskerstrand wrote:
> You'll find this in the wiki[0], and for load balanced nginx I've also
> written a post on [1].
>
> References:
> [0] https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering
> [1] http://blog.sumptuouscapital.com/2013/10/load-balancing-sks/

Hrm, say you have two instances, one only peering with the other, as
you describe.  Then during reconciliation of the two, when they're
talking to each other, they're likely to be busy at the _same_ time, so
load balancing isn't buying you much then.  It seems more that you'll
want to be biasing towards the internal-only one, so that what you're
buying protection against is the external peering occupying your
gateway.

In fact, it looks more like you want either:

 (1) at least two instances which are not gateway instances and not
     peering with each other; reverse proxy upstreams pointing to just
     those two
 (2) the same three instances total, all able to peer with each other,
     and the reverse proxy talking to all three (but with the gateway
     de-preferenced)

Otherwise, while what you describe is strictly better than a single
instance, I'm not sure its as much better as might immediately be
inferred without more analysis.

- -Phil
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAlJxhuoACgkQQDBDFTkDY398PwCfYwDyljXNRGnOUjxmhUr373Da
DF8AnR3IWZX38CPumUtF6SMuw5zJGTMa
=AfhV
-----END PGP SIGNATURE-----

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Todd Lyons-2
In reply to this post by Kiss Gabor (Bitman)
On Wed, Oct 30, 2013 at 11:31 AM, Gabor Kiss <[hidden email]> wrote:

>
> > Whatever the decision, could you provide documentation for
> > configuration of such a reverse proxy for both Apache and Nginx?
>
> What I miss is a set of diagnostic procedures/recipes that could
> help an operator to figure out if his server fits various requirements.
>
> Like this was on Monday:
>
> | Virtualhost-related, no match found
> |
> | address@hidden ~ $ curl -H'Host: p80.pool.sks-keyservers.net' "http://keys.niif.hu/pks/lookup?op=stats";
> | <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> | <html><head>
> | <title>404 Not Found</title>
> | </head><body>

Yes, that was a very nice statement, and when I ran it, it revealed
that I had a misconfiguration on my system too.  The #httpd channel
gave me one AWESOME command that immediately indicated how my system
was configured:

# httpd -S
VirtualHost configuration:
[2001:470:d:367::555]:80 sks.mrball.net (/etc/httpd/conf.d/sks.conf:23)
[2001:470:d:367::555]:443 sks.mrball.net (/etc/httpd/conf.d/sks.conf:63)
208.89.139.251:80      sks.mrball.net (/etc/httpd/conf.d/sks.conf:23)
208.89.139.251:443     sks.mrball.net (/etc/httpd/conf.d/sks.conf:40)
wildcard NameVirtualHosts and _default_ servers:
*:11371                sks.mrball.net (/etc/httpd/conf.d/sks.conf:8)
_default_:443          mail.mrball.net (/etc/httpd/conf.d/ssl.conf:74)
*:80                   is a NameVirtualHost
         default server www.mrball.net (/etc/httpd/conf.d/00-vhosts.conf:61)
         port 80 namevhost www.mrball.net (/etc/httpd/conf.d/00-vhosts.conf:61)
         port 80 namevhost downloads.mrball.net
(/etc/httpd/conf.d/00-vhosts.conf:69)
         port 80 namevhost bluefish.mrball.net
(/etc/httpd/conf.d/00-vhosts.conf:80)
         port 80 namevhost eximbuild.mrball.net
(/etc/httpd/conf.d/eximbuild.conf:1)
Syntax OK

Originally I had the keyserver stuff listening on the *:80 and *:443
NameVHost instead of a separate Listen directive and IP:80 / IP:443.
I do find it interesting that the *:11371 is listed as a
NameVirtualHost, but it still accepts any Host header that comes in
(probably because I use Listen IP:11371 multiple times instead of Port
11371).

It may be that my system needs more tweaking though.  It's working for
everything that I test with (all Host headers I send at it), and I
have green lights on the status page.

...Todd
--
SOPA: Any attempt to [use legal means to] reverse technological
advances is doomed.  --Leo Leporte

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Kristian Fiskerstrand-6
In reply to this post by Kristian Fiskerstrand-6
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 10/28/2013 11:33 PM, Kristian Fiskerstrand wrote:

>
>
> Current snapshot shows that 45 of 76 servers in the active pool
> are identified as being behind a reverse proxy, being roughly 60%.
> This includes nearly all of the servers that are included in the
> geographical pools based on a more calculated approach[0]. In
> comparison we only had some 30-odd servers directly qualifying when
> I first started looking into setting the minimum requirement of the
> pool to 1.1.3[1], at the time of the actual switch another 10-15
> operators had upgraded, and I believe the pool results are better
> for it today.
>

Just a heads up that we now have 51 servers behind reverse proxies, so
I've decided to implement the change to require a reverse proxy for
the main pool some time this weekend.

If anyone without such a configuration wants to set it up, information
is found in the wiki[0].

References:
[0] https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering

- --
- ----------------------------
Kristian Fiskerstrand
Blog: http://blog.sumptuouscapital.com
Twitter: @krifisk
- ----------------------------
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
- ----------------------------
Corruptissima re publica plurimæ leges
The greater the degeneration of the republic, the more of its laws
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJShlQvAAoJEAt/i2Dj7frj4hMQAJF9/2x2geSFye2EdL+FdA5u
jfEUaxx3MGrTmC3EalkvQhULljoDzqmNL7hPFfPHt0A9XVgD+bPGs+Ku/DIKZfdi
60NTNyocDk0VnWW4EJLSZVmFcP8UXBIb1h9Pr5p2sJN67f+u6j+SUFUb74YvXKUc
jUARvmRAQ76koT1qDsr1dKGaBIGL7SJb5iaEMtY1/13LEtcLgAMZVcRGbffbkNLt
1IbNo62mnvZcaTiq+H8th7zxV6DHWmrtll7PRAhJfV0yZ8a2p3aVPZJAr+iTa8B8
GYoD7DUaRj0SYRKokCFLFLVcC/oEjMjs7TiYST3XCjmf3hiXOLQ++Er0LKDweN8w
+C5CksZnhtkkPIng3BlU9NxNBMyYlJDf1BVq5+bw1pt20PZex/uktKSHBM7FUqgf
uAtcjKbd0U80VTdSpms0JpVvnUQaMH8Xq3//ii1706gLmocn52CowT9gQMjs8Mah
+r4GIc8ZAj4mVgwznB9O5wP5n5AfP5QMVolUse6iSRcvA0rwG/gArxlcNO+ux70+
wZE6DgVnNDVh06kJHKMw1VPZ33ouFUVC49KVqpdv8ZLEK8D2I8QP524oGcQIq5OK
fTJYwkm80tCJbmkyisKI+Md12TKCtQqo251jfL2eyFB3GrBYlK/GZPrQSEJMHP58
svRaSlbl8LCWP8dBGeN5
=lp01
-----END PGP SIGNATURE-----

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Todd Lyons-2
In reply to this post by Kristian Fiskerstrand-6
On Mon, Oct 28, 2013 at 3:33 PM, Kristian Fiskerstrand

> PS! For those that have noticed a blue indicator on that status
> page[2], this is a preliminary setup for a potential new HA pool in
> the future for load-balanced servers in front of multiple SKS
> instances. I do however expect the HA pool to continue in the same

Just curious, how do you detect this?  I just noticed the blue for the
first time today.

> manner as today for a while longer before that change happens. If
> anyone is interested in my own load-balanced setup using nginx I've
> written up a blog post on [3].

I've always wondered if I could just run multiple instances of sks db
on the localhost with different ports and load balance amongst them. I
never tried it though because I'm afraid of corrupting the database.
The sks db instances don't need rw access, just ro.  That would allow
for simple load balancing configurations, good for throughput, but not
for HA.

...Todd
--
SOPA: Any attempt to [use legal means to] reverse technological
advances is doomed.  --Leo Leporte

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: reverse proxies and the pool

Kristian Fiskerstrand-6
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 11/15/2013 10:24 PM, Todd Lyons wrote:

> On Mon, Oct 28, 2013 at 3:33 PM, Kristian Fiskerstrand
>
>> PS! For those that have noticed a blue indicator on that status
>> page[2], this is a preliminary setup for a potential new HA pool
>> in the future for load-balanced servers in front of multiple SKS
>> instances. I do however expect the HA pool to continue in the
>> same
>
> Just curious, how do you detect this?  I just noticed the blue for
> the first time today.

I don't, at the moment it is manually specified. However, in the long
term, multiple requests and checking for nodename difference for a
singular host would be an option. This would probably be a matter for
a maintenance script running far less frequently than the hourly
updates of the pool though.

>
>> manner as today for a while longer before that change happens.
>> If anyone is interested in my own load-balanced setup using nginx
>> I've written up a blog post on [3].
>
> I've always wondered if I could just run multiple instances of sks
> db on the localhost with different ports and load balance amongst
> them. I never tried it though because I'm afraid of corrupting the
> database. The sks db instances don't need rw access, just ro.  That
> would allow for simple load balancing configurations, good for
> throughput, but not for HA.

For now I'm only including nodes on different servers (although that
can of course be VMs)

- --
- ----------------------------
Kristian Fiskerstrand
Blog: http://blog.sumptuouscapital.com
Twitter: @krifisk
- ----------------------------
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
- ----------------------------
Testis unus, testis nullus
A single witness is no witness
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJShpIoAAoJEAt/i2Dj7frjfMMP+wTe2E17tHgggZrprvzGeEdK
cmLS/lH68ES/iUuNt5Qb4bT0t1XCvMx3Mp2KMwO+FBW/+A+D075JfxuyQu4in6V4
aOSiNBRv2cC65cu48UQYGaeUe8ld94yAQrIVzwplT7/f+2hOSCM1x7fkKSlY47XM
dmR06touv+kmcNuftdvIMz1EKWimFjLLzbzr7TbyBkHGxhQ4mROSBQfwKp19PMqk
lfVy0/afvOmBGXoAJDj+TnJQ+F3YZaGzskvsvZ0tfyHeoJYwr+8Rdr2hzScFyUdU
FerSUzaZy+DLSa00BnEd8BDvVAvRfPnhNEvW/KmO3l9n+2IMDLiD4w3g2nye5B99
hTnOyPGmrLscK7fieUhcQ4KowKLB6kWezu4t/MXrOIUMGYygaqFW+HAcOttkKcCD
fKYyzHIVaVrrTSaWmoLTYdR/w4OUxAmjpuVG3exQcOQWmFYEXSyUk+VQiMPqUsvc
JL1d3zYvWbQTRlctuN4yZL0Yuxpvu/OlbFGVRHsH+FV75qnQcKsZ9mQkHs5AV092
/VT8rkURgWoMNkni8OJ+CUifnDO2UeFVzGOsK9D5pZfEI/b3V7TCcicCjOc7ERAC
V/d1QtCWTQbQYSg7jZpTeSCwIZBBzHkqaLNXtMQ3vZBVco8Cr6tn+xMy0+Ke/6Rp
WwN+kBiyJL2Xq9+5H1bC
=ZMZu
-----END PGP SIGNATURE-----

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel