Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Shengjing Zhu
Hi,

While I rescued my key server back this night, I found the unusual
traffic for key 0x69D2EAD9 and 0xB33B4659. It caused load to my server
when it tried to sync up with the network.

Request counted in 2h:

   178 0xB33B4659
    186 0x69D2EAD9
    290 0x2016349F5BC6F49340FCCAF99F9169F4B33B4659
    336 0x1013D73FECAC918A0A25823986CE877469D2EAD9

Requests come from pool.sks-keyservers.net. Compare to the server
number behind the pool,  I think these requests are quite unusual.
Does anyone know what happens to these two keys?

--
Regards,
Shengjing Zhu

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

brent s.
On 1/12/19 2:15 PM, Shengjing Zhu wrote:

> Hi,
>
> While I rescued my key server back this night, I found the unusual
> traffic for key 0x69D2EAD9 and 0xB33B4659. It caused load to my server
> when it tried to sync up with the network.
>
> Request counted in 2h:
>
>    178 0xB33B4659
>     186 0x69D2EAD9
>     290 0x2016349F5BC6F49340FCCAF99F9169F4B33B4659
>     336 0x1013D73FECAC918A0A25823986CE877469D2EAD9
>
> Requests come from pool.sks-keyservers.net. Compare to the server
> number behind the pool,  I think these requests are quite unusual.
> Does anyone know what happens to these two keys?
>
they're for FreePBX and have caused at least one other issue:

https://lists.gnu.org/archive/html/sks-devel/2018-07/msg00077.html

based on this:

https://www.dslreports.com/forum/r30661088-PBX-FreePBX-for-the-Raspberry-Pi~start=810

it would SEEM they're part of the FreePBX installation process, but it's
possible that something from normal operation even fetches the key
operationally and frequently.

i see three possible situations:

0.) a recent update was made to FreePBX that fetches the key, even if it
exists in the keyring or a key refresh is called (very likely)
1.) a random attack targeting you specifically is ocurring and they just
randomly picked that key ID (a little likely, but not very)
2.) the key has been compromised and is being used as part of a botnet
for some purpose (extremely unlikely)

i'll see if i can find out from the freepbx source/the project devs.

will reply when i have further info.


meanwhile, can you let us know if those requests are all coming from the
same IP or allocation block?

--
brent saner
https://square-r00t.net/
GPG info: https://square-r00t.net/gpg-info


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (916 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Shengjing Zhu
Sorry for top replying. I'm using mobile phone.

Requests are coming from different network, at least hundreds IP.

And it seems my server(pgp.ustc.edu.cn) is down again... I'll check it when I got home. If it's caused by the two keys.. I may blacklist them...

brent s. <[hidden email]> 于 2019年1月13日周日 04:45写道:
On 1/12/19 2:15 PM, Shengjing Zhu wrote:
> Hi,
>
> While I rescued my key server back this night, I found the unusual
> traffic for key 0x69D2EAD9 and 0xB33B4659. It caused load to my server
> when it tried to sync up with the network.
>
> Request counted in 2h:
>
>    178 0xB33B4659
>     186 0x69D2EAD9
>     290 0x2016349F5BC6F49340FCCAF99F9169F4B33B4659
>     336 0x1013D73FECAC918A0A25823986CE877469D2EAD9
>
> Requests come from pool.sks-keyservers.net. Compare to the server
> number behind the pool,  I think these requests are quite unusual.
> Does anyone know what happens to these two keys?
>

they're for FreePBX and have caused at least one other issue:

https://lists.gnu.org/archive/html/sks-devel/2018-07/msg00077.html

based on this:

https://www.dslreports.com/forum/r30661088-PBX-FreePBX-for-the-Raspberry-Pi~start=810

it would SEEM they're part of the FreePBX installation process, but it's
possible that something from normal operation even fetches the key
operationally and frequently.

i see three possible situations:

0.) a recent update was made to FreePBX that fetches the key, even if it
exists in the keyring or a key refresh is called (very likely)
1.) a random attack targeting you specifically is ocurring and they just
randomly picked that key ID (a little likely, but not very)
2.) the key has been compromised and is being used as part of a botnet
for some purpose (extremely unlikely)

i'll see if i can find out from the freepbx source/the project devs.

will reply when i have further info.


meanwhile, can you let us know if those requests are all coming from the
same IP or allocation block?

--
brent saner
https://square-r00t.net/
GPG info: https://square-r00t.net/gpg-info

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

brent s.
On 1/13/19 12:15 AM, Shengjing Zhu wrote:
> Sorry for top replying. I'm using mobile phone.
>
> Requests are coming from different network, at least hundreds IP.
>
> And it seems my server(pgp.ustc.edu.cn <http://pgp.ustc.edu.cn>) is down
> again... I'll check it when I got home. If it's caused by the two keys..
> I may blacklist them...
>

i've asked on FreePBX's channel and emailed the organization directly
(via their key's UID info) but have not yet gotten a response from
either. it IS the weekend, so it may be a bit...

meanwhile you may want to firewall off your HKP port(s) (recon port
should still be fine to keep open, but someone correct me if not) and
disable the forwarding for HKPS (if you have it).


--
brent saner
https://square-r00t.net/
GPG info: https://square-r00t.net/gpg-info


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (916 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Gabor Kiss
In reply to this post by Shengjing Zhu
> Request counted in 2h:
>
>    178 0xB33B4659
>     186 0x69D2EAD9
>     290 0x2016349F5BC6F49340FCCAF99F9169F4B33B4659
>     336 0x1013D73FECAC918A0A25823986CE877469D2EAD9

I checked my logs. 15% of the recent 18k requests were related to these keys.
They belong to:

FreePBX Module Signing (This is the master key to sign FreePBX Modules) <[hidden email]>
FreePBX Mirror 1 (Module Signing - 2014/2015) <[hidden email]>

I guess there was some software upgrade on ten thousands of Asterix nodes.
Looks normal.

Gabor

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

brent s.
On 1/13/19 1:49 AM, Gabor Kiss wrote:

>> Request counted in 2h:
>>
>>    178 0xB33B4659
>>     186 0x69D2EAD9
>>     290 0x2016349F5BC6F49340FCCAF99F9169F4B33B4659
>>     336 0x1013D73FECAC918A0A25823986CE877469D2EAD9
>
> I checked my logs. 15% of the recent 18k requests were related to these keys.
> They belong to:
>
> FreePBX Module Signing (This is the master key to sign FreePBX Modules) <[hidden email]>
> FreePBX Mirror 1 (Module Signing - 2014/2015) <[hidden email]>
>
> I guess there was some software upgrade on ten thousands of Asterix nodes.
> Looks normal.
>
> Gabor
last stable release was may 2018, so i'm not sure on that personally.
i'd expect a lot MORE if that were the case.[0] it's... a really popular
piece of software. i even used it when i managed some VoIP systems.

it's just at the amount where it's inordinately high, but low enough to
make me not think it was something like a new release.



[0] "With over 1 MILLION production systems worldwide and 20,000 new
systems installed monthly, ..."
https://www.freepbx.org/

--
brent saner
https://square-r00t.net/
GPG info: https://square-r00t.net/gpg-info


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (916 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Kristian Fiskerstrand-6
In reply to this post by Shengjing Zhu
On 1/12/19 8:15 PM, Shengjing Zhu wrote:
>  I think these requests are quite unusual.
> Does anyone know what happens to these two keys?

Just to add a comment on this, adding a cache on the load-balancer is
really a nice way to slow down hits on the underlying SKS nodes, I keep
cache for 10 minutes in nginx, which really makes life more pleasant.

--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Action is the foundational key to all success"
(Pablo Picasso)


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Martin Dobrev
Hi,

My observations so far show that both keys generate  2+ TB/month traffic on average for all my clustered nodes. I'm running nginx + Varnish in-memory cache tuned at 5 minutes TTL which gives plenty of CPU cycles for the never-ending EventLoop alarm loops. The latter cause load-average spikes of up to 10 with just 4 Docker containers running on a 12 core system.
Don't get me wrong. The throttling penalty is something I'd swallow-up as long as we keep the network running. 

Regards,
Martin 

keyserver.dobrev.eu | pgp.dobrev.it 

-------- Original message --------
From: Kristian Fiskerstrand <[hidden email]>
Date: 30/01/2019 20:18 (GMT+00:00)
To: Shengjing Zhu <[hidden email]>, [hidden email]
Subject: Re: [Sks-devel] Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

On 1/12/19 8:15 PM, Shengjing Zhu wrote:
>  I think these requests are quite unusual.
> Does anyone know what happens to these two keys?

Just to add a comment on this, adding a cache on the load-balancer is
really a nice way to slow down hits on the underlying SKS nodes, I keep
cache for 10 minutes in nginx, which really makes life more pleasant.

--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Action is the foundational key to all success"
(Pablo Picasso)


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Rolf Wuerdemann
Hi,

Don't get me wrong, but within three days I've got 450G traffic
which can be assigned to sks by 99.9%. Estimated to 30 days this
means 4.5T (which is in good agreement of your 2+T/Key for these
two poison keys).

With this amount of traffic and the possibility to get
more of this keys (thus more traffic) every moment, I think it's
only a question of time until the network with the current
implementation will vanish. Traffic increased roughly a factor of
300 (15G->4.5T) within twelve months, nodes within the network
decreased by a factor of two at least for the same time.

So: where to go and how?

Just my 2ct,

    rowue

Am 2019-01-30 22:09, schrieb Martin Dobrev:

> Hi,
>
> My observations so far show that both keys generate  2+ TB/month
> traffic on average for all my clustered nodes. I'm running nginx +
> Varnish in-memory cache tuned at 5 minutes TTL which gives plenty of
> CPU cycles for the never-ending EventLoop alarm loops. The latter
> cause load-average spikes of up to 10 with just 4 Docker containers
> running on a 12 core system.
> Don't get me wrong. The throttling penalty is something I'd swallow-up
> as long as we keep the network running.
>
> Regards,
> Martin
>
> keyserver.dobrev.eu | pgp.dobrev.it
>
> -------- Original message --------
> From: Kristian Fiskerstrand
> <[hidden email]>
> Date: 30/01/2019 20:18 (GMT+00:00)
> To: Shengjing Zhu <[hidden email]>, [hidden email]
> Subject: Re: [Sks-devel] Unusual traffic for key 0x69D2EAD9 and
> 0xB33B4659
>
> On 1/12/19 8:15 PM, Shengjing Zhu wrote:
>>  I think these requests are quite unusual.
>> Does anyone know what happens to these two keys?
>
> Just to add a comment on this, adding a cache on the load-balancer is
> really a nice way to slow down hits on the underlying SKS nodes, I
> keep
> cache for 10 minutes in nginx, which really makes life more pleasant.
>
> --
> ----------------------------
> Kristian Fiskerstrand
> Blog: https://blog.sumptuouscapital.com
> Twitter: @krifisk
> ----------------------------
> Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
> fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
> ----------------------------
> "Action is the foundational key to all success"
> (Pablo Picasso)
> _______________________________________________
> Sks-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/sks-devel

--
Security is an illusion - Datasecurity twice
Rolf Würdemann  -  [hidden email]  -  DL9ROW
GnuPG fingerprint:    EEDC BEA9 EFEA 54A9 E1A9  2D54 69CC 9F31 6C64 206A
xmpp: [hidden email] E1189573 6B4A150C A0C2BF5A 5553F865 0B9CBF7A
       [hidden email] 64CBBB68 0A3514A4 026FC1E7 5328CE87 AEE2185F

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Martin Dobrev

Hi,

I've spent last week trying to optimize configuration as much as possible. Following advise from a previous mail I've added:

command_timeout: 600
wserver_timeout: 30
max_recover: 150

to my sksconf and it seems this fixed majority of the EventLoop failures. I've added DB_CONFIG in KDB/PTree folders to get rid of DB archive logs that were causing plenty of IO load too.

My clusters are now happily responding to queries and load-average is bellow one. Traffic wise things look better too, ~20GB/day.


Kind regards,
Martin Dobrev

P.S. Adding/changing DB_CONFIG might cause an error in the databases that you can easily fix by running

db_recover -e -v -h <path to SKS>/{KDB,PTree}


On 04/02/2019 09:49, Rolf Wuerdemann wrote:
Hi,

Don't get me wrong, but within three days I've got 450G traffic
which can be assigned to sks by 99.9%. Estimated to 30 days this
means 4.5T (which is in good agreement of your 2+T/Key for these
two poison keys).

With this amount of traffic and the possibility to get
more of this keys (thus more traffic) every moment, I think it's
only a question of time until the network with the current
implementation will vanish. Traffic increased roughly a factor of
300 (15G->4.5T) within twelve months, nodes within the network
decreased by a factor of two at least for the same time.

So: where to go and how?

Just my 2ct,

   rowue

Am 2019-01-30 22:09, schrieb Martin Dobrev:
Hi,

My observations so far show that both keys generate  2+ TB/month
traffic on average for all my clustered nodes. I'm running nginx +
Varnish in-memory cache tuned at 5 minutes TTL which gives plenty of
CPU cycles for the never-ending EventLoop alarm loops. The latter
cause load-average spikes of up to 10 with just 4 Docker containers
running on a 12 core system.
Don't get me wrong. The throttling penalty is something I'd swallow-up
as long as we keep the network running.

Regards,
Martin

keyserver.dobrev.eu | pgp.dobrev.it

-------- Original message --------
From: Kristian Fiskerstrand
[hidden email]
Date: 30/01/2019 20:18 (GMT+00:00)
To: Shengjing Zhu [hidden email], [hidden email]
Subject: Re: [Sks-devel] Unusual traffic for key 0x69D2EAD9 and
0xB33B4659

On 1/12/19 8:15 PM, Shengjing Zhu wrote:
 I think these requests are quite unusual.
Does anyone know what happens to these two keys?

Just to add a comment on this, adding a cache on the load-balancer is
really a nice way to slow down hits on the underlying SKS nodes, I
keep
cache for 10 minutes in nginx, which really makes life more pleasant.

--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Action is the foundational key to all success"
(Pablo Picasso)
_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

pEpkey.asc (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Rolf Wuerdemann
With your suggestions:

load average below 1
Traffic: ~150G/day

Best,

    Rolf

Am 2019-02-04 12:52, schrieb Martin Dobrev:

> Hi,
>
> I've spent last week trying to optimize configuration as much as
> possible. Following advise from a previous mail I've added:
>
>> command_timeout: 600
>> wserver_timeout: 30
>> max_recover: 150
>
> to my sksconf and it seems this fixed majority of the EventLoop
> failures. I've added DB_CONFIG in KDB/PTree folders to get rid of DB
> archive logs that were causing plenty of IO load too.
>
> My clusters are now happily responding to queries and load-average is
> bellow one. Traffic wise things look better too, ~20GB/day.
>
> Kind regards,
> Martin Dobrev
>
> P.S. Adding/changing DB_CONFIG might cause an error in the databases
> that you can easily fix by running
>
> db_recover -e -v -h <path to SKS>/{KDB,PTree}
>
> On 04/02/2019 09:49, Rolf Wuerdemann wrote:
>
>> Hi,
>>
>> Don't get me wrong, but within three days I've got 450G traffic
>> which can be assigned to sks by 99.9%. Estimated to 30 days this
>> means 4.5T (which is in good agreement of your 2+T/Key for these
>> two poison keys).
>>
>> With this amount of traffic and the possibility to get
>> more of this keys (thus more traffic) every moment, I think it's
>> only a question of time until the network with the current
>> implementation will vanish. Traffic increased roughly a factor of
>> 300 (15G->4.5T) within twelve months, nodes within the network
>> decreased by a factor of two at least for the same time.
>>
>> So: where to go and how?
>>
>> Just my 2ct,
>>
>> rowue
>>
>> Am 2019-01-30 22:09, schrieb Martin Dobrev:
>> Hi,
>>
>> My observations so far show that both keys generate  2+ TB/month
>> traffic on average for all my clustered nodes. I'm running nginx +
>> Varnish in-memory cache tuned at 5 minutes TTL which gives plenty of
>>
>> CPU cycles for the never-ending EventLoop alarm loops. The latter
>> cause load-average spikes of up to 10 with just 4 Docker containers
>> running on a 12 core system.
>> Don't get me wrong. The throttling penalty is something I'd
>> swallow-up
>> as long as we keep the network running.
>>
>> Regards,
>> Martin
>>
>> keyserver.dobrev.eu | pgp.dobrev.it
>>
>> -------- Original message --------
>> From: Kristian Fiskerstrand
>> <[hidden email]>
>> Date: 30/01/2019 20:18 (GMT+00:00)
>> To: Shengjing Zhu <[hidden email]>, [hidden email]
>> Subject: Re: [Sks-devel] Unusual traffic for key 0x69D2EAD9 and
>> 0xB33B4659
>>
>> On 1/12/19 8:15 PM, Shengjing Zhu wrote:
>> I think these requests are quite unusual.
>> Does anyone know what happens to these two keys?
>>
>> Just to add a comment on this, adding a cache on the load-balancer
>> is
>> really a nice way to slow down hits on the underlying SKS nodes, I
>> keep
>> cache for 10 minutes in nginx, which really makes life more
>> pleasant.
>>
>> --
>> ----------------------------
>> Kristian Fiskerstrand
>> Blog: https://blog.sumptuouscapital.com
>> Twitter: @krifisk
>> ----------------------------
>> Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
>> fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
>> ----------------------------
>> "Action is the foundational key to all success"
>> (Pablo Picasso)
>> _______________________________________________
>> Sks-devel mailing list
>> [hidden email]
>> https://lists.nongnu.org/mailman/listinfo/sks-devel
> _______________________________________________
> Sks-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/sks-devel

--
Security is an illusion - Datasecurity twice
Rolf Würdemann  -  [hidden email]  -  DL9ROW
GnuPG fingerprint:    EEDC BEA9 EFEA 54A9 E1A9  2D54 69CC 9F31 6C64 206A
xmpp: [hidden email] E1189573 6B4A150C A0C2BF5A 5553F865 0B9CBF7A
       [hidden email] 64CBBB68 0A3514A4 026FC1E7 5328CE87 AEE2185F

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Todd Fleisher
I also applied these configuration options earlier today to all the servers in 1 of my pools that was experiencing high IO load and repeated SigAlarms:

command_timeout: 600
wserver_timeout: 30
max_recover: 150

And since then, everything has been quiet:

IO on the main node that gossips externally: https://i.imgur.com/ERgz0Xo.jpg

IO from another node in the same pool that gossips internally with the above node: https://i.imgur.com/wsaxrJ5.jpg

Hopefully this can help other operators keep things in better shape for the time being.

-T


On Feb 6, 2019, at 3:22 AM, Rolf Wuerdemann <[hidden email]> wrote:

With your suggestions:

load average below 1
Traffic: ~150G/day

Best,

  Rolf

Am 2019-02-04 12:52, schrieb Martin Dobrev:
Hi,
I've spent last week trying to optimize configuration as much as
possible. Following advise from a previous mail I've added:
command_timeout: 600
wserver_timeout: 30
max_recover: 150
to my sksconf and it seems this fixed majority of the EventLoop
failures. I've added DB_CONFIG in KDB/PTree folders to get rid of DB
archive logs that were causing plenty of IO load too.
My clusters are now happily responding to queries and load-average is
bellow one. Traffic wise things look better too, ~20GB/day.
Kind regards,
Martin Dobrev
P.S. Adding/changing DB_CONFIG might cause an error in the databases
that you can easily fix by running
db_recover -e -v -h <path to SKS>/{KDB,PTree}
On 04/02/2019 09:49, Rolf Wuerdemann wrote:
Hi,
Don't get me wrong, but within three days I've got 450G traffic
which can be assigned to sks by 99.9%. Estimated to 30 days this
means 4.5T (which is in good agreement of your 2+T/Key for these
two poison keys).
With this amount of traffic and the possibility to get
more of this keys (thus more traffic) every moment, I think it's
only a question of time until the network with the current
implementation will vanish. Traffic increased roughly a factor of
300 (15G->4.5T) within twelve months, nodes within the network
decreased by a factor of two at least for the same time.
So: where to go and how?
Just my 2ct,
rowue
Am 2019-01-30 22:09, schrieb Martin Dobrev:
Hi,
My observations so far show that both keys generate  2+ TB/month
traffic on average for all my clustered nodes. I'm running nginx +
Varnish in-memory cache tuned at 5 minutes TTL which gives plenty of
CPU cycles for the never-ending EventLoop alarm loops. The latter
cause load-average spikes of up to 10 with just 4 Docker containers
running on a 12 core system.
Don't get me wrong. The throttling penalty is something I'd
swallow-up
as long as we keep the network running.
Regards,
Martin
keyserver.dobrev.eu | pgp.dobrev.it
-------- Original message --------
From: Kristian Fiskerstrand
<[hidden email]>
Date: 30/01/2019 20:18 (GMT+00:00)
To: Shengjing Zhu <[hidden email]>, [hidden email]
Subject: Re: [Sks-devel] Unusual traffic for key 0x69D2EAD9 and
0xB33B4659
On 1/12/19 8:15 PM, Shengjing Zhu wrote:
I think these requests are quite unusual.
Does anyone know what happens to these two keys?
Just to add a comment on this, adding a cache on the load-balancer
is
really a nice way to slow down hits on the underlying SKS nodes, I
keep
cache for 10 minutes in nginx, which really makes life more
pleasant.
--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at <a href="hkp://pool.sks-keyservers.net" class="">hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Action is the foundational key to all success"
(Pablo Picasso)
_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

--
Security is an illusion - Datasecurity twice
Rolf Würdemann  -  [hidden email]  -  DL9ROW
GnuPG fingerprint:    EEDC BEA9 EFEA 54A9 E1A9  2D54 69CC 9F31 6C64 206A
xmpp: [hidden email] E1189573 6B4A150C A0C2BF5A 5553F865 0B9CBF7A
     [hidden email] 64CBBB68 0A3514A4 026FC1E7 5328CE87 AEE2185F

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Unusual traffic for key 0x69D2EAD9 and 0xB33B4659

Todd Fleisher
To follow up on this, after making the below changes while my main disk IO went down, my load average went up, memory usage went through the roof & swapping ensued. I increased the amount of memory assigned to each of my main nodes (those that gossip with the outside world) and it seems to be holding steady so far: https://imgur.com/a/b0S4Ui2

I believe the nodes may have also been having issues gossiping as I saw outbound network traffic flatline during the same time periods: https://imgur.com/a/IEoLboM

-T

On Feb 6, 2019, at 4:21 PM, Todd Fleisher <[hidden email]> wrote:

Signed PGP part
I also applied these configuration options earlier today to all the servers in 1 of my pools that was experiencing high IO load and repeated SigAlarms:

command_timeout: 600
wserver_timeout: 30
max_recover: 150

And since then, everything has been quiet:

IO on the main node that gossips externally: https://i.imgur.com/ERgz0Xo.jpg

IO from another node in the same pool that gossips internally with the above node: https://i.imgur.com/wsaxrJ5.jpg

Hopefully this can help other operators keep things in better shape for the time being.

-T


On Feb 6, 2019, at 3:22 AM, Rolf Wuerdemann <[hidden email]> wrote:

With your suggestions:

load average below 1
Traffic: ~150G/day

Best,

  Rolf

Am 2019-02-04 12:52, schrieb Martin Dobrev:
Hi,
I've spent last week trying to optimize configuration as much as
possible. Following advise from a previous mail I've added:
command_timeout: 600
wserver_timeout: 30
max_recover: 150
to my sksconf and it seems this fixed majority of the EventLoop
failures. I've added DB_CONFIG in KDB/PTree folders to get rid of DB
archive logs that were causing plenty of IO load too.
My clusters are now happily responding to queries and load-average is
bellow one. Traffic wise things look better too, ~20GB/day.
Kind regards,
Martin Dobrev
P.S. Adding/changing DB_CONFIG might cause an error in the databases
that you can easily fix by running
db_recover -e -v -h <path to SKS>/{KDB,PTree}
On 04/02/2019 09:49, Rolf Wuerdemann wrote:
Hi,
Don't get me wrong, but within three days I've got 450G traffic
which can be assigned to sks by 99.9%. Estimated to 30 days this
means 4.5T (which is in good agreement of your 2+T/Key for these
two poison keys).
With this amount of traffic and the possibility to get
more of this keys (thus more traffic) every moment, I think it's
only a question of time until the network with the current
implementation will vanish. Traffic increased roughly a factor of
300 (15G->4.5T) within twelve months, nodes within the network
decreased by a factor of two at least for the same time.
So: where to go and how?
Just my 2ct,
rowue
Am 2019-01-30 22:09, schrieb Martin Dobrev:
Hi,
My observations so far show that both keys generate  2+ TB/month
traffic on average for all my clustered nodes. I'm running nginx +
Varnish in-memory cache tuned at 5 minutes TTL which gives plenty of
CPU cycles for the never-ending EventLoop alarm loops. The latter
cause load-average spikes of up to 10 with just 4 Docker containers
running on a 12 core system.
Don't get me wrong. The throttling penalty is something I'd
swallow-up
as long as we keep the network running.
Regards,
Martin
keyserver.dobrev.eu | pgp.dobrev.it
-------- Original message --------
From: Kristian Fiskerstrand
<[hidden email]>
Date: 30/01/2019 20:18 (GMT+00:00)
To: Shengjing Zhu <[hidden email]>, [hidden email]
Subject: Re: [Sks-devel] Unusual traffic for key 0x69D2EAD9 and
0xB33B4659
On 1/12/19 8:15 PM, Shengjing Zhu wrote:
I think these requests are quite unusual.
Does anyone know what happens to these two keys?
Just to add a comment on this, adding a cache on the load-balancer
is
really a nice way to slow down hits on the underlying SKS nodes, I
keep
cache for 10 minutes in nginx, which really makes life more
pleasant.
--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at <a href="hkp://pool.sks-keyservers.net" class="">hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Action is the foundational key to all success"
(Pablo Picasso)
_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

--
Security is an illusion - Datasecurity twice
Rolf Würdemann  -  [hidden email]  -  DL9ROW
GnuPG fingerprint:    EEDC BEA9 EFEA 54A9 E1A9  2D54 69CC 9F31 6C64 206A
xmpp: [hidden email] E1189573 6B4A150C A0C2BF5A 5553F865 0B9CBF7A
     [hidden email] 64CBBB68 0A3514A4 026FC1E7 5328CE87 AEE2185F

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel




_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (849 bytes) Download Attachment