Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Andrew Nagy
All,

Looking to figure out a solution here. A Maintainer on the Ubuntu Key server informed me about discussion of the following keys 0x69D2EAD9 and 0xB33B4659 here: https://lists.nongnu.org/archive/html/sks-devel/2019-01/msg00003.html

Unfortunately the email address [hidden email] is just a black hole and so the email that was sent there from Brent Saner was lost forever.

I currently run the FreePBX project which uses the GPG network to sign modules. Unfortunately due to: https://bitbucket.org/skskeyserver/sks-keyserver/issues/57/anyone-can-make-any-pgp-key-unimportable

Someone poisoned our master key that we use to sign all other keys. This has caused issues on the sks network for a while. However since January we've noticed more and more sks servers are now just timing out and not returning back our requests for 0xB33B4659. I assume that is probably because of the message thread from January.

The way FreePBX software works is that it checks nightly against a list of key servers to redownload 0x69D2EAD9 and 0xB33B4659 and re-verify. However it appears that for many of you the bandwidth this causes is much too high. Internally we need to recreate our master key without the poison but I am afraid it will just as easily be re-poisoned again. Also even if we put a new key out you will notice traffic increase from those keys over time and well and we will be back to the bandwidth issue.

Perhaps we should be using GPG locally instead of through the GPG key network. Let me know what you guys think,

Thank you

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Jeremy T. Bouse
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 3/20/2019 2:42 PM, Andrew Nagy wrote:

> All,
>
> Looking to figure out a solution here. A Maintainer on the Ubuntu
> Key server informed me about discussion of the following keys
> 0x69D2EAD9 and 0xB33B4659 here:
> https://lists.nongnu.org/archive/html/sks-devel/2019-01/msg00003.html
>
>
>
>
>
Unfortunately the email address [hidden email]
> <mailto:[hidden email]> is just a black hole and so the email
> that was sent there from Brent Saner was lost forever.
>
> I currently run the FreePBX project which uses the GPG network to
> sign modules. Unfortunately due to:
> https://bitbucket.org/skskeyserver/sks-keyserver/issues/57/anyone-can-
make-any-pgp-key-unimportable
>
>
>
>
>
Someone poisoned our master key that we use to sign all other

> keys. This has caused issues on the sks network for a while.
> However since January we've noticed more and more sks servers are
> now just timing out and not returning back our requests for
> 0xB33B4659. I assume that is probably because of the message
> thread from January.
>
> The way FreePBX software works is that it checks nightly against a
> list of key servers to redownload 0x69D2EAD9 and 0xB33B4659 and
> re-verify. However it appears that for many of you the bandwidth
> this causes is much too high. Internally we need to recreate our
> master key without the poison but I am afraid it will just as
> easily be re-poisoned again. Also even if we put a new key out you
> will notice traffic increase from those keys over time and well and
> we will be back to the bandwidth issue.
>
> Perhaps we should be using GPG locally instead of through the GPG
> key network. Let me know what you guys think,
>
> Thank you
>

        I don't speak for all SKS server operators, but I do have a
configuration block in my NGINX configuration that specifically
identifies those keys and simply returns a 444 status code.

        I've been trying to get a handle on the instability of my server
which has been running the CPU at 100% at times so I enabled an access
log rule to idenitfy the query strings and upstream times but not the
requesting IP address... According to my status page my server only
spent 61.924% of yesterday in the pool or roughly 14.86 hours, during
that same period of time my server saw 12436 requests for those 2 keys
for an average of 836.86 requests per hour or nearly 14 per minute.
During most times that wouldn't be a lot but when the system is under
load that volume adds up and is non-trivial.

        One opption the FreePBX team could do is self-publish their key using
WKD or PKA. WKD you would store the file on your own web server
that no one else could touch (presumably if setup properly anyway) and
PKA you would publish within DNS records. GPG has the ability to
retrieve from both methods without needing to use a keyserver.
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEEakJ0F+CHS9VzhSFg6lYpTv4TPXUFAlySpigACgkQ6lYpTv4T
PXU3aAwAve2kSUqxiXkg74zoO+l0lL1sD1cPiok4i6i9+D+nre+g4awR/sJdcVal
yGmIgYJa4HpuXKrCBUD9oX+n4HDjjekJpHdbqYOUYI5mx6+T5YKPLtP0hUmhe4Jv
P4hSnX4tppsQADJo4Ms4txHkmHqPis3Khjnr92+nGG7Xq98tHgOmu67jjoNmTsAG
0ADZ1+Lkd9V7UTBMy7+jkbqrda7/v65+3YgYfyoSQIYg7DCRp6Rg+jwGyrzRD/Vh
ZRRu+1McPrw2dKMksRabZHm8efkyDdpkFldvR9+bDR7EC70axZ+zWb8iKTIr7qLY
huiu9x/wVvTxw3mpVCN1Ii5Vj9qe3SfuiVYtHws8vqmaoOf/h9QGLIK+KcPgf0sj
Y8jKz7obltc29RyyfqblDtEJXNOrKD5OpSLptOUpa/6sm+F6MRT0DQKEIsHq4BRR
nCi2aLldat0ojZwkFTqNzD+M6mCBSI3FgCPzGlSITVRiLDdatO/R+wmy1hAgT3Hv
k5GG4CBt
=en3K
-----END PGP SIGNATURE-----

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

brent s.
On 3/20/19 4:44 PM, Jeremy T. Bouse wrote:>

> I don't speak for all SKS server operators, but I do have a
> configuration block in my NGINX configuration that specifically
> identifies those keys and simply returns a 444 status code.
>
> I've been trying to get a handle on the instability of my server
> which has been running the CPU at 100% at times so I enabled an access
> log rule to idenitfy the query strings and upstream times but not the
> requesting IP address... According to my status page my server only
> spent 61.924% of yesterday in the pool or roughly 14.86 hours, during
> that same period of time my server saw 12436 requests for those 2 keys
> for an average of 836.86 requests per hour or nearly 14 per minute.
> During most times that wouldn't be a lot but when the system is under
> load that volume adds up and is non-trivial.
>
> One opption the FreePBX team could do is self-publish their key using
> WKD or PKA. WKD you would store the file on your own web server
> that no one else could touch (presumably if setup properly anyway) and
> PKA you would publish within DNS records. GPG has the ability to
> retrieve from both methods without needing to use a keyserver.
>

I'd second this, as it's the "most right" solution (he says, still not
having set up WKD/PKA for his own personal infra yet. I really need to
get on that...).

An alternative is to set up your own SKS that does not allow submissions
(I'd imagine you could just 403 the upload path/args except for a few
authorized addresses since HKP more or less is just HTTP), keep that as
your modules' key repository, and create a fresh master key that signs
those pubkeys, and publish that new master to the WoT/SKS pool. That
doesn't do a terribly good job of preventing something like this in the
future, of course, but does insulate you from the load caused by
installations fetching the keys; core could be distributed with the
master pubkey, even, and you'd then have a method of checking module
signatures right from the get-go.

You can also just host the master pubkey as an exported key somewhere,
and fetch that directly.

I think the massive load is mostly caused by the querying of the master
key against the SKS pool more than anything.

--
brent saner
https://square-r00t.net/
GPG info: https://square-r00t.net/gpg-info


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (916 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Shengjing Zhu
In reply to this post by Andrew Nagy
On Thu, Mar 21, 2019 at 2:42 AM Andrew Nagy <[hidden email]> wrote:

>
> All,
>
> Looking to figure out a solution here. A Maintainer on the Ubuntu Key server informed me about discussion of the following keys 0x69D2EAD9 and 0xB33B4659 here: https://lists.nongnu.org/archive/html/sks-devel/2019-01/msg00003.html
>
> Unfortunately the email address [hidden email] is just a black hole and so the email that was sent there from Brent Saner was lost forever.
>
> I currently run the FreePBX project which uses the GPG network to sign modules. Unfortunately due to: https://bitbucket.org/skskeyserver/sks-keyserver/issues/57/anyone-can-make-any-pgp-key-unimportable
>
> Someone poisoned our master key that we use to sign all other keys. This has caused issues on the sks network for a while. However since January we've noticed more and more sks servers are now just timing out and not returning back our requests for 0xB33B4659. I assume that is probably because of the message thread from January.
>

Actually I've not blacklist your keys. But these two keys are not only
pulled too frequently, they are tooo large. I assume it's not your
fault, but someone poisons them.

0x1013D73FECAC918A0A25823986CE877469D2EAD9 stored in the key server is 38M.

As to no response for these requests, probably the key servers are just down :(

Please just don't rely on the key server network for such task, you
can use https://wiki.gnupg.org/WKD

--
Regards,
Shengjing Zhu

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Keys size repartition (was Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659)

Kim Minh Kaplan-2
In reply to this post by Andrew Nagy
Hello,

I did some stats on the keys in my dataset as of 2019-03-22. The list
may find these interesting so here it is.

  number  | %-age   | keys size             | size for all  | %-age of
  of keys | of keys | in octets             | these keys    | total size
----------+---------+-----------------------+---------------+-----------
      270     0.00    >= 10 and < 100                24,530    0.00
  730,085    13.37    >= 100 and < 1,000        512,038,375    4.00
4,645,289    85.09    >= 1,000 and < 10,000   8,340,048,313   65.15
   77,761     1.42    >= 10^4 and < 10^5      1,845,831,322   14.41
    5,816     0.11    >= 10^5 and < 10^6      1,134,650,188    8.86
       89     0.00    >= 10^6 and < 10^7        178,791,550    1.39
       14     0.00    >= 10^7 and < 10^8        538,791,133    4.20
        2     0.00    >= 10^8                   250,575,213    1.95
---------                                   ---------------
5,459,326 total number of keys               12,800,750,624 total size for all keys

Maximum key size was 127,749,570 octets.

Kim Minh.

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Keys size repartition (was Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659)

Kim Minh Kaplan-2


Jason Harris writes:

>
>> Kim Minh Kaplan wrote:
>>
>> Jason Harris writes:
>>
>>> Cool!  What tool/code did you use?
>>
>> Homegrown tools. I use a simple C program to read KDB/keys and create
>a
>> single PGP file for each key. I can then easily use traditionnal
>tools
>> to get some interesting numbers. Mainly find, sort, sed, uniq, bc.
>
>Interesting.  I could use a keydump like that to more fully populate a
>hockeypuck server without enabling gossiping.

As much as I try to come up with a solution, I can't see how to distribute such a dump efficiently. We are speaking millions of files here.

>Currently, hockeypuck
>chokes on several SKS keydump files and apparently needs to gossip to
>fill in the gaps.  I now see you posted a link to the program to fetch
>a single key on 2019-01-19, thanks for that!

I'll try and find some time to clean-up and post the program to dump the whole database.

>We just need more people hacking hockeypuck...

SKS has the same problem. And we also need a new policy for accepting data in the network. All these abuses are killing it.

Kim Minh.

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel