Another Poison Key?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Another Poison Key?

Simon Lange
Hi,

i noticed a key, which, whenever one of my peers tries to send it to me,
is failing and causing unstable enlistment in the sks pool list.

Jan 14 06:48:05 atlas sks[17917]: 2019-01-14 06:48:05 Requesting 100
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C
Jan 14 08:51:59 atlas sks[17917]: 2019-01-14 08:51:59 Requesting 100
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C
Jan 14 11:17:11 atlas sks[17917]: 2019-01-14 11:17:11 Requesting 100
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C
Jan 14 11:38:06 atlas sks[17917]: 2019-01-14 11:38:06 Requesting 100
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C
Jan 14 12:18:52 atlas sks[17917]: 2019-01-14 12:18:52 Requesting 99
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C
Jan 14 13:01:05 atlas sks[17917]: 2019-01-14 13:01:05 Requesting 100
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C
Jan 14 13:54:08 atlas sks[17917]: 2019-01-14 13:54:08 Requesting 100
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C
Jan 14 14:12:16 atlas sks[17917]: 2019-01-14 14:12:16 Requesting 100
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C
Jan 14 14:32:09 atlas sks[10527]: 2019-01-14 14:32:09 Requesting 100
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C

As you can see its a still present problem makes it impossible to get
"these 100 missing keys" from that peer.

The error looks always like that:
Jan 14 15:00:06 atlas sks[10527]: 2019-01-14 15:00:06 111 hashes
recovered from <ADDR_INET [193.224.163.43]:11371>
Jan 14 15:00:08 atlas sks[10527]: 2019-01-14 15:00:08 Requesting 100
missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
030E4ECA584FC3298423CA5B0CB7855C
Jan 14 15:00:26 atlas sks[10527]: 2019-01-14 15:00:26 100 keys received
Jan 14 15:01:26 atlas sks[10526]: 2019-01-14 15:01:26 add_keys_merge
failed: Eventloop.SigAlarm
Jan 14 15:01:26 atlas sks[10526]: 2019-01-14 15:01:26 Key addition
failed: Eventloop.SigAlarm
Jan 14 15:01:26 atlas sks[10526]: 2019-01-14 15:01:26 100 keys found

the only issue in the archive i found so far was this one:
https://lists.nongnu.org/archive/html/sks-devel/2018-11/msg00073.html

...which describes EXACTLY the same symptoms and - maybe - the same
cause.
So is there any workaround? Does that poisoned key poison the peer
rendering it as unstable?
Is there some kind of blacklist-workaround or max-key-size setting for
the sksconf to prevent loading such?

Thanks

Simon

--
PPCIS - Project Leader

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

brent s.
On 1/14/19 9:35 AM, Simon Lange wrote:

> Hi,
>
> i noticed a key, which, whenever one of my peers tries to send it to me,
> is failing and causing unstable enlistment in the sks pool list.
>
> Jan 14 06:48:05 atlas sks[17917]: 2019-01-14 06:48:05 Requesting 100
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
> Jan 14 08:51:59 atlas sks[17917]: 2019-01-14 08:51:59 Requesting 100
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
> Jan 14 11:17:11 atlas sks[17917]: 2019-01-14 11:17:11 Requesting 100
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
> Jan 14 11:38:06 atlas sks[17917]: 2019-01-14 11:38:06 Requesting 100
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
> Jan 14 12:18:52 atlas sks[17917]: 2019-01-14 12:18:52 Requesting 99
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
> Jan 14 13:01:05 atlas sks[17917]: 2019-01-14 13:01:05 Requesting 100
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
> Jan 14 13:54:08 atlas sks[17917]: 2019-01-14 13:54:08 Requesting 100
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
> Jan 14 14:12:16 atlas sks[17917]: 2019-01-14 14:12:16 Requesting 100
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
> Jan 14 14:32:09 atlas sks[10527]: 2019-01-14 14:32:09 Requesting 100
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
>
> As you can see its a still present problem makes it impossible to get
> "these 100 missing keys" from that peer.
>
> The error looks always like that:
> Jan 14 15:00:06 atlas sks[10527]: 2019-01-14 15:00:06 111 hashes
> recovered from <ADDR_INET [193.224.163.43]:11371>
> Jan 14 15:00:08 atlas sks[10527]: 2019-01-14 15:00:08 Requesting 100
> missing keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C
> Jan 14 15:00:26 atlas sks[10527]: 2019-01-14 15:00:26 100 keys received
> Jan 14 15:01:26 atlas sks[10526]: 2019-01-14 15:01:26 add_keys_merge
> failed: Eventloop.SigAlarm
> Jan 14 15:01:26 atlas sks[10526]: 2019-01-14 15:01:26 Key addition
> failed: Eventloop.SigAlarm
> Jan 14 15:01:26 atlas sks[10526]: 2019-01-14 15:01:26 100 keys found
>
> the only issue in the archive i found so far was this one:
> https://lists.nongnu.org/archive/html/sks-devel/2018-11/msg00073.html
>
> ...which describes EXACTLY the same symptoms and - maybe - the same cause.
> So is there any workaround? Does that poisoned key poison the peer
> rendering it as unstable?
> Is there some kind of blacklist-workaround or max-key-size setting for
> the sksconf to prevent loading such?
>
> Thanks
>
> Simon

hrm.. yeah, looks malformatted.

"Error handling request: 128-bit v3 fingerprints not implemented"

https://bugs.launchpad.net/launchpad/+bug/144435
https://bugs.launchpad.net/launchpad/+bug/4746


--
brent saner
https://square-r00t.net/
GPG info: https://square-r00t.net/gpg-info


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (916 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

Yegor Timoshenko
> hrm.. yeah, looks malformatted.
>
> "Error handling request: 128-bit v3 fingerprints not
> implemented"
>
> https://bugs.launchpad.net/launchpad/+bug/144435
> https://bugs.launchpad.net/launchpad/+bug/4746

Could you upload the key somewhere and link to it?
_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

brent s.
On 1/14/19 10:06 AM, Yegor Timoshenko wrote:

>> hrm.. yeah, looks malformatted.
>>
>> "Error handling request: 128-bit v3 fingerprints not
>> implemented"
>>
>> https://bugs.launchpad.net/launchpad/+bug/144435
>> https://bugs.launchpad.net/launchpad/+bug/4746
>
> Could you upload the key somewhere and link to it?
>
well, that's the issue - hkp won't pull it, gpg won't pull it either.

anyone know of a way to dump/extract a specific key from the SKS DB?
i'd imagine there'd be a bdb way to do it but i'm not that old. 🙃

--
brent saner
https://square-r00t.net/
GPG info: https://square-r00t.net/gpg-info


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (916 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

Gabor Kiss
In reply to this post by Simon Lange
> i noticed a key, which, whenever one of my peers tries to send it to me, is
> failing and causing unstable enlistment in the sks pool list.

My typical logs in the same time:

> Jan 14 13:54:08 atlas sks[17917]: 2019-01-14 13:54:08 Requesting 100 missing
> keys from <ADDR_INET [193.224.163.43]:11371>, starting with
> 030E4ECA584FC3298423CA5B0CB7855C

2019-01-14 13:54:05 Beginning recon as server, client: <ADDR_INET [85.214.205.124]:50049>
2019-01-14 13:54:05 Joining reconciliation
2019-01-14 13:54:06 Reconciliation complete
2019-01-14 13:54:06 225 hashes recovered from <ADDR_INET [85.214.205.124]:11371>
2019-01-14 13:54:06 Disabling gossip
2019-01-14 13:54:39 Requesting 100 missing keys from <ADDR_INET [85.214.205.124]:11371>, starting with 00C3C77EA8F92FE95039D6A0C1FE7CF4
2019-01-14 13:55:37 100 keys received
2019-01-14 13:56:38 Reconciliation attempt from <ADDR_INET [85.119.82.209]:59751> while gossip disabled. Ignoring.
2019-01-14 13:56:38 Not gossiping because gossip is disabled
2019-01-14 13:56:38 Requesting 100 missing keys from <ADDR_INET [85.214.205.124]:11371>, starting with 8CBC3E4391AD9EC95BFBFA002785C339
2019-01-14 13:56:40 99 keys received
2019-01-14 13:56:41 Reconciliation attempt from <ADDR_INET [85.25.207.23]:42652> while gossip disabled. Ignoring.
2019-01-14 14:05:01 Requesting 25 missing keys from <ADDR_INET [85.214.205.124]: 11371>, starting with DE136D33BB4F5F4825A3C432A7292703
2019-01-14 14:05:03 24 keys received
2019-01-14 14:05:23 Reconciliation attempt from <ADDR_INET [91.121.92.14]:45481> while gossip disabled. Ignoring.

Is "... while gossip disabled. Ignoring." is normal?

Gabor

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

Andrew Gallagher
On 18/01/2019 06:09, Gabor Kiss wrote:
> Is "... while gossip disabled. Ignoring." is normal?

Yes. SKS is single-threaded, so while it is in the middle of a task it
politely ignores incoming network requests.

--
Andrew Gallagher


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

Simon Lange
Am 18.01.2019 11:01, schrieb Andrew Gallagher:
> On 18/01/2019 06:09, Gabor Kiss wrote:
>> Is "... while gossip disabled. Ignoring." is normal?
>
> Yes. SKS is single-threaded, so while it is in the middle of a task it
> politely ignores incoming network requests.

And how do we tell sks to use more than ONE instance for the SAME
database?!
Because ONE single thread is a pretty big bottleneck problem. And i
could not discover any sksconf or runtime option which spawns more than
one sks instance to run more stable in a productive way.

Any chance that sks will be fixed some day? allowing using a better
databasesystem for performance and stability, a better scheduler for
better processing requests and a proper logging which gives answers and
does not produce new questions. e.g. if a process stalls and sks is able
to give an error message, why doesnt it give the cause of the problem as
well? especially if debug and level is set to max? currently is pretty
hard to investigate causes for sks-problems. its like
blackbox-debugging. ;)

regards

Simon
--
PPCIS - Project Leader

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

Andrew Gallagher
On 18/01/2019 11:17, Simon Lange wrote:
> Am 18.01.2019 11:01, schrieb Andrew Gallagher:
>> On 18/01/2019 06:09, Gabor Kiss wrote:
>>> Is "... while gossip disabled. Ignoring." is normal?
>>
>> Yes. SKS is single-threaded, so while it is in the middle of a task it
>> politely ignores incoming network requests.
>
> And how do we tell sks to use more than ONE instance for the SAME
> database?!

You don't. SKS just isn't built that way. To get concurrency, you need
to run multiple separate instances of SKS and configure them to gossip
between each other. Then you can put a load balancing reverse proxy in
front to simulate a multi-threaded server. Kristian and a few others
have been operating this way for a while now.

> Any chance that sks will be fixed some day?

Short answer, no. SKS is effectively running as end-of-life software at
this point. It gets emergency bugfixes but little else. The improvements
you are talking about would require an enormous refactoring of the
codebase, likely requiring migration to a different database engine.
Given that there are fundamental design flaws (poison keys) that aren't
getting fixed, performance issues just aren't on the radar. Sorry.

--
Andrew Gallagher


_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

Kim Minh Kaplan
In reply to this post by brent s.
PM brent s. wrote:

> well, that's the issue - hkp won't pull it, gpg won't pull it either.
>
> anyone know of a way to dump/extract a specific key from the SKS DB?
> i'd imagine there'd be a bdb way to do it but i'm not that old.

I've just wrote a short snippet to pull out data directly from Berkley
DB (https://www.kim-minh.com/src/misc/bdb-get.c).


For example if I want to pull out the key by ID, use the last 8
characters as the short keyid. For example the short keyid for
748231EBCBD808A14F5E85D28C004C2F93481F6B is 93481F6B.

    $ cc bdb-get.c -ldb
    $ key_id=93481F6B
    $ key_hash=$(./a.out /var/lib/sks-tmp/DB keyid "$key_id" | hexdump
-e '/1 "%02x"')
    $ ./a.out /var/lib/sks-tmp/DB key "$key_hash" | dd bs=1 count=1 | hexdump

The first byte of the key tells how it is stored (keydb.ml, function
skey_of_string). A 0 (zero) means the payload is the key. A 1 or a 2
means that the payload is a pointer into the *.pgp files (that's when
you used fastbuild).

As I do not use fastbuild all my keys are stored with type 0. I can
then get the key itself by just skipping the first (zero) byte.

    $ ./a.out /var/lib/sks-tmp/DB key "$key_hash" | dd bs=1 skip=1
>/tmp/$key_id.pgp

The key_hash is what appears in the recon log.

Hope this can help your troubleshooting.
--
Kim Minh.

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

DevPGSV Pablo
I am receiving lots of request for key: 0x69D2EAD9

And, as it turns out, when I have an error and I clic on the browser on "View page source", I can see part of the key text is:


Do not use SKS keyserver sites (no validity checks)&lt;https:&#x2F;&#x2F;bitbucket.org&#x2F;skskeyserver&#x2F;sks-keyserver&#x2F;issues&#x2F;41&gt; ;P;w S Ӗy*7|]Ln ,k 35ZfqEi2`2b j;Ymp;uG_M&quot;$1X__}ΏbHBt; Lo)$r D::p(Br M0(S5(eXv 4 Dӽ4 8( ;ġ]]ӉT#O ~XyjI2ܗ27; d&gt; !Xk @]GoY4fk % !Oquot;Jy P P;E Ϲ 3;


Then a lot of giberish. This seems like an attack against SKS software.
I wasn't able to get the hash of that key.

I know it is not a solution, but... is there any way to blacklist keys?
If there was a way, at least I could blacklist manually these attacks, even if I have to check every day.

Some other keys I found from FreePBX:
  • 0x69D2EAD9
  • 0xB33B4659

My keyserver is completely unusable until this issue is resolved. If I can't resolve this, then there would be no point in keeping the keyserver active.

Thanks you,
Pablo



El sáb., 19 ene. 2019 a las 11:48, Kim Minh Kaplan (<[hidden email]>) escribió:
PM brent s. wrote:

> well, that's the issue - hkp won't pull it, gpg won't pull it either.
>
> anyone know of a way to dump/extract a specific key from the SKS DB?
> i'd imagine there'd be a bdb way to do it but i'm not that old.

I've just wrote a short snippet to pull out data directly from Berkley
DB (https://www.kim-minh.com/src/misc/bdb-get.c).


For example if I want to pull out the key by ID, use the last 8
characters as the short keyid. For example the short keyid for
748231EBCBD808A14F5E85D28C004C2F93481F6B is 93481F6B.

    $ cc bdb-get.c -ldb
    $ key_id=93481F6B
    $ key_hash=$(./a.out /var/lib/sks-tmp/DB keyid "$key_id" | hexdump
-e '/1 "%02x"')
    $ ./a.out /var/lib/sks-tmp/DB key "$key_hash" | dd bs=1 count=1 | hexdump

The first byte of the key tells how it is stored (keydb.ml, function
skey_of_string). A 0 (zero) means the payload is the key. A 1 or a 2
means that the payload is a pointer into the *.pgp files (that's when
you used fastbuild).

As I do not use fastbuild all my keys are stored with type 0. I can
then get the key itself by just skipping the first (zero) byte.

    $ ./a.out /var/lib/sks-tmp/DB key "$key_hash" | dd bs=1 skip=1
>/tmp/$key_id.pgp

The key_hash is what appears in the recon log.

Hope this can help your troubleshooting.
--
Kim Minh.

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

Yegor Timoshenko
Ok, so that was created with my program:
https://gitlab.com/yegortimoshenko/sks-exploits/blob/807ca89c0c2192ccb33d908ec2974779735805d8/sks-fake-uid/main.go#L15

Relevant issue:
https://bitbucket.org/skskeyserver/sks-keyserver/issues/60

> I know it is not a solution, but... is there any way to
> blacklist keys? If there was a way, at least I could blacklist
> manually these attacks, even if I have to check every day.

Sure, here is an updated patch that blacklists this key, as well
as the older poison key:
https://gist.github.com/yegortimoshenko/781c880be8f1b8a91c9c23fa83a35d58
(based off patch by Shengjing Zhu)

There are several problems with this approach:

1. Future updates for the key will be denied, including
legitimate ones by key holder (FreePBX team). 2. DoS is still
possible just by accessing/fetching the key. To fix that, you'll
have to remove the DoS packets (large user packets with random
gibberish, not valid per OpenPGP packet spec, does not validate
cryptographically) or the whole key. 3. Anyone can create another
poison key at any time and there's no way to fix that without
breaking compat, it's a fundamental flaw :-(
_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel
Reply | Threaded
Open this post in threaded view
|

Re: Another Poison Key?

DevPGSV Pablo
Thanks!

Yeah, I know that is not a solution. 
Looking at the code, it seems "reasonable" to add keys to the blacklist... so I guess I will check everyday for problematic keys, and if I find any I will be able to ban them from my SKS server (and I will notify the list with the keys, as I guess I won't be the only one with the problem).

Thanks again!



El mar., 22 ene. 2019 a las 12:23, Yegor Timoshenko (<[hidden email]>) escribió:
Ok, so that was created with my program:
https://gitlab.com/yegortimoshenko/sks-exploits/blob/807ca89c0c2192ccb33d908ec2974779735805d8/sks-fake-uid/main.go#L15

Relevant issue:
https://bitbucket.org/skskeyserver/sks-keyserver/issues/60

> I know it is not a solution, but... is there any way to
> blacklist keys? If there was a way, at least I could blacklist
> manually these attacks, even if I have to check every day.

Sure, here is an updated patch that blacklists this key, as well
as the older poison key:
https://gist.github.com/yegortimoshenko/781c880be8f1b8a91c9c23fa83a35d58
(based off patch by Shengjing Zhu)

There are several problems with this approach:

1. Future updates for the key will be denied, including
legitimate ones by key holder (FreePBX team). 2. DoS is still
possible just by accessing/fetching the key. To fix that, you'll
have to remove the DoS packets (large user packets with random
gibberish, not valid per OpenPGP packet spec, does not validate
cryptographically) or the whole key. 3. Anyone can create another
poison key at any time and there's no way to fix that without
breaking compat, it's a fundamental flaw :-(

_______________________________________________
Sks-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/sks-devel