crash inside of wrap_nettle_rnd?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

crash inside of wrap_nettle_rnd?

Jason A. Donenfeld
Hi folks,

I'm getting a crash in weechat, and after some debugging, the
maintainer of weechat told me to complain here. I've put all the info
in this gentoo bug: https://bugs.gentoo.org/show_bug.cgi?id=501078 .

Here are various backtraces:

#0  0x0000744315fa2e8e in raise () from /lib64/libc.so.6
#1  0x0000744315fa45df in abort () from /lib64/libc.so.6
#2  0x0000744316de2324 in wrap_nettle_rnd () from /usr/lib64/libgnutls.so.28
#3  0x0000744316d28218 in _gnutls_tls_create_random () from
/usr/lib64/libgnutls.so.28
#4  0x0000744316d2861a in _gnutls_set_client_random () from
/usr/lib64/libgnutls.so.28
#5  0x0000744316d2a116 in _gnutls_send_hello () from /usr/lib64/libgnutls.so.28
#6  0x0000744316d2c928 in gnutls_handshake () from /usr/lib64/libgnutls.so.28
#7  0x000000c2d34924d1 in network_connect_child_read_cb ()
#8  0x000000c2d3489a46 in hook_fd_exec ()
#9  0x000000c2d34244fe in gui_main_loop ()
#10 0x000000c2d3419820 in main ()

(gdb) bt
#0  0x00006448f718fad4 in __lll_lock_elision () from /lib64/libpthread.so.0
#1  0x00006448fa98e8be in ?? () from /usr/lib64/libgnutls.so.28
#2  0x00006448faa2c16e in ?? () from /usr/lib64/libgnutls.so.28
#3  0x00006448fa972218 in ?? () from /usr/lib64/libgnutls.so.28
#4  0x00006448fa97261a in ?? () from /usr/lib64/libgnutls.so.28
#5  0x00006448fa974116 in ?? () from /usr/lib64/libgnutls.so.28
#6  0x00006448fa976928 in gnutls_handshake () from /usr/lib64/libgnutls.so.28
#7  0x00000e43a73e14d1 in network_connect_child_read_cb ()
#8  0x00000e43a73d8a46 in hook_fd_exec ()
#9  0x00000e43a73734fe in gui_main_loop ()
#10 0x00000e43a7368820 in main ()

(gdb) bt
#0  0x000074975344fe8e in raise () from /lib64/libc.so.6
#1  0x00007497534515df in abort () from /lib64/libc.so.6
#2  0x000074975428f324 in ?? () from /usr/lib64/libgnutls.so.28
#3  0x00007497541d5218 in ?? () from /usr/lib64/libgnutls.so.28
#4  0x00007497541d561a in ?? () from /usr/lib64/libgnutls.so.28
#5  0x00007497541d7116 in ?? () from /usr/lib64/libgnutls.so.28
#6  0x00007497541d9928 in gnutls_handshake () from /usr/lib64/libgnutls.so.28
#7  0x00000fbe153084d1 in network_connect_child_read_cb ()
#8  0x00000fbe152ffa46 in hook_fd_exec ()
#9  0x00000fbe1529a4fe in gui_main_loop ()
#10 0x00000fbe1528f820 in main ()

(gdb) bt
#0  0x0000736ab354aad4 in __lll_lock_elision () from /lib64/libpthread.so.0
#1  0x0000736ab8c6d8be in ?? () from /usr/lib64/libgnutls.so.28
#2  0x0000736ab8d0b16e in ?? () from /usr/lib64/libgnutls.so.28
#3  0x0000736ab8c4c7ae in ?? () from /usr/lib64/libgnutls.so.28
#4  0x0000736ab8c477b8 in ?? () from /usr/lib64/libgnutls.so.28
#5  0x0000736ab8c49b92 in gnutls_record_send () from /usr/lib64/libgnutls.so.28
#6  0x0000736ab4c7e80c in irc_server_send () from
/usr/lib64/weechat/plugins/irc.so
#7  0x0000736ab4c7f130 in irc_server_send_one_msg () from
/usr/lib64/weechat/plugins/irc.so
#8  0x0000736ab4c7f953 in irc_server_sendf () from
/usr/lib64/weechat/plugins/irc.so
#9  0x0000736ab4c5cbfc in irc_input_send_user_message () from
/usr/lib64/weechat/plugins/irc.so
#10 0x0000736ab4c5cdfb in irc_input_data () from
/usr/lib64/weechat/plugins/irc.so
#11 0x00000909fff0eccd in input_data ()
#12 0x00000909ffec73ff in gui_input_return ()
#13 0x00000909ffee1f90 in command_input ()
#14 0x00000909fff06fc4 in hook_command_exec ()
#15 0x00000909fff0e951 in input_exec_command ()
#16 0x00000909fff0ed04 in input_data ()
#17 0x00000909ffecbb5f in gui_key_pressed ()
#18 0x00000909ffea1122 in gui_key_flush ()
#19 0x00000909ffea165e in gui_key_read_cb ()
#20 0x00000909fff07a46 in hook_fd_exec ()
#21 0x00000909ffea24fe in gui_main_loop ()
#22 0x00000909ffe97820 in main ()


Any help would be appreciated.

Thanks,
Jason

_______________________________________________
Weechat-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/weechat-dev
Reply | Threaded
Open this post in threaded view
|

Re: [gnutls-devel] crash inside of wrap_nettle_rnd?

Jason A. Donenfeld
On Wed, Feb 12, 2014 at 10:33 PM, Nikos Mavrogiannopoulos
<[hidden email]> wrote:
> Hello Jason,
>  Unfortunately these backtraces don't provide much information. You'll
> need debugging symbols (most probably included in your distribution as a
> separate package). Also the version of gnutls being used is important.
> It would be best to try with the latest released version.

This occurs with 3.2.10. I believe it also occurred with 3.2.9. I'll
recompile with more debug granularity (I'm on gentoo hardened) and
report back.

>
> However, the only reason for a mutex call to fail is if the mutex isn't
> initialized. Is gnutls_global_init() executed successful in the program
> you use?

gnutls_global_init is executed.

What's interesting is that the error only happens when weechat loads
the python plugin. I'm not too familiar how the python plugin works
internally, but I imagine it's possible it spawns a thread. Could
there be some situation in which all new threads also need to call
gnutls_global_init? I'm speculating here, but maybe it has something
to do with this...

_______________________________________________
Weechat-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/weechat-dev
Reply | Threaded
Open this post in threaded view
|

Re: [gnutls-devel] crash inside of wrap_nettle_rnd?

Jason A. Donenfeld
Here's a better backtrace. 3.2.10


#0  0x000069ec221d0e8e in raise () from /lib64/libc.so.6
#1  0x000069ec221d25df in abort () from /lib64/libc.so.6
#2  0x000069ec230586c5 in wrap_nettle_rnd (_ctx=0x0, level=0,
data=0x556db2fbeab, datasize=29) at rnd.c:441
#3  0x000069ec22f5c62d in _gnutls_rnd (level=GNUTLS_RND_NONCE,
data=0x556db2fbeab, len=29) at ./random.h:37
#4  0x000069ec22f5cbdb in _gnutls_tls_create_random (dst=0x556db2fbea8
"R\373", <incomplete sequence \365\237>)
    at gnutls_handshake.c:206
#5  0x000069ec22f5cce2 in _gnutls_set_client_random
(session=0x556db2fbe60, rnd=0x0) at gnutls_handshake.c:231
#6  0x000069ec22f62168 in _gnutls_send_client_hello
(session=0x556db2fbe60, again=0) at gnutls_handshake.c:1990
#7  0x000069ec22f62b26 in _gnutls_send_hello (session=0x556db2fbe60,
again=0) at gnutls_handshake.c:2203
#8  0x000069ec22f63892 in _gnutls_handshake_client
(session=0x556db2fbe60) at gnutls_handshake.c:2656
#9  0x000069ec22f635ea in gnutls_handshake (session=0x556db2fbe60) at
gnutls_handshake.c:2527
#10 0x00000556d7277581 in network_connect_child_read_cb
(arg_hook_connect=0x556db2f9c90, fd=14)
    at /var/tmp/portage/net-irc/weechat-0.4.3/work/weechat-0.4.3/src/core/wee-network.c:1484
#11 0x00000556d726966d in hook_fd_exec (read_fds=0x72851c083090,
write_fds=0x72851c083110, exception_fds=0x72851c083190)
    at /var/tmp/portage/net-irc/weechat-0.4.3/work/weechat-0.4.3/src/core/wee-hook.c:1329
#12 0x00000556d71eb0f2 in gui_main_loop ()
    at /var/tmp/portage/net-irc/weechat-0.4.3/work/weechat-0.4.3/src/gui/curses/gui-curses-main.c:501
#13 0x00000556d72358cf in main (argc=1, argv=0x72851c083358)
    at /var/tmp/portage/net-irc/weechat-0.4.3/work/weechat-0.4.3/src/core/weechat.c:477

_______________________________________________
Weechat-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/weechat-dev
Reply | Threaded
Open this post in threaded view
|

Re: [gnutls-devel] crash inside of wrap_nettle_rnd?

Jason A. Donenfeld
In reply to this post by Jason A. Donenfeld
From reading the man page, apparently pthread_mutex_lock can fail for
a variety of different reasons. I suppose I'll need to patch gnutls to
print me the reason before crashing. Ugh.

_______________________________________________
Weechat-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/weechat-dev
Reply | Threaded
Open this post in threaded view
|

Re: [gnutls-devel] crash inside of wrap_nettle_rnd?

Jason A. Donenfeld
Okay apparently it's failing with error 22 -- EINVAL. From the man page:

       EINVAL The  mutex was created with the protocol
              attribute     having      the      value
              PTHREAD_PRIO_PROTECT   and  the  calling
              thread's priority  is  higher  than  the
              mutex's current priority ceiling.

_______________________________________________
Weechat-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/weechat-dev
Reply | Threaded
Open this post in threaded view
|

Re: [gnutls-devel] crash inside of wrap_nettle_rnd?

Jason A. Donenfeld
Well this is interesting. Evidently sometimes it crashes in this error
path instead:

#0  0x00006cc0a1495e8e in raise () from /lib64/libc.so.6
#1  0x00006cc0a14975df in abort () from /lib64/libc.so.6
#2  0x00006cc0a148e2b2 in ?? () from /lib64/libc.so.6
#3  0x00006cc0a148e376 in __assert_fail () from /lib64/libc.so.6
#4  0x00006cc09ea2fec0 in __pthread_tpp_change_priority () from
/lib64/libpthread.so.0
#5  0x00006cc09ea23ed8 in __pthread_mutex_lock_full () from
/lib64/libpthread.so.0
#6  0x00006cc0a224ac8c in gnutls_system_mutex_lock
(priv=0x6cc0a25583b8 <rnd_mutex>) at system.c:228
#7  0x00006cc0a231d6fc in wrap_nettle_rnd (_ctx=0x0, level=0,
data=0xfc3f4a5207b, datasize=29) at rnd.c:441
#8  0x00006cc0a222162d in _gnutls_rnd (level=GNUTLS_RND_NONCE,
data=0xfc3f4a5207b, len=29) at ./random.h:37
#9  0x00006cc0a2221bdb in _gnutls_tls_create_random (dst=0xfc3f4a52078
"R\373\370y") at gnutls_handshake.c:206
#10 0x00006cc0a2221ce2 in _gnutls_set_client_random
(session=0xfc3f4a52030, rnd=0x0) at gnutls_handshake.c:231
#11 0x00006cc0a2227168 in _gnutls_send_client_hello
(session=0xfc3f4a52030, again=0) at gnutls_handshake.c:1990
#12 0x00006cc0a2227b26 in _gnutls_send_hello (session=0xfc3f4a52030,
again=0) at gnutls_handshake.c:2203
#13 0x00006cc0a2228892 in _gnutls_handshake_client
(session=0xfc3f4a52030) at gnutls_handshake.c:2656
#14 0x00006cc0a22285ea in gnutls_handshake (session=0xfc3f4a52030) at
gnutls_handshake.c:2527
#15 0x00000fc3f1c82581 in network_connect_child_read_cb
(arg_hook_connect=0xfc3f4a51470, fd=12)
    at /var/tmp/portage/net-irc/weechat-0.4.3/work/weechat-0.4.3/src/core/wee-network.c:1484
#16 0x00000fc3f1c7466d in hook_fd_exec (read_fds=0x724715d46df0,
write_fds=0x724715d46e70, exception_fds=0x724715d46ef0)
    at /var/tmp/portage/net-irc/weechat-0.4.3/work/weechat-0.4.3/src/core/wee-hook.c:1329
#17 0x00000fc3f1bf60f2 in gui_main_loop ()
    at /var/tmp/portage/net-irc/weechat-0.4.3/work/weechat-0.4.3/src/gui/curses/gui-curses-main.c:501
#18 0x00000fc3f1c408cf in main (argc=1, argv=0x724715d470b8)
    at /var/tmp/portage/net-irc/weechat-0.4.3/work/weechat-0.4.3/src/core/weechat.c:477

_______________________________________________
Weechat-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/weechat-dev
Reply | Threaded
Open this post in threaded view
|

Re: [gnutls-devel] crash inside of wrap_nettle_rnd?

Jason A. Donenfeld
In reply to this post by Jason A. Donenfeld
On Wed, Feb 12, 2014 at 11:43 PM, Jason A. Donenfeld <[hidden email]> wrote:
> Okay apparently it's failing with error 22 -- EINVAL. From the man page:
>
>        EINVAL The  mutex was created with the protocol
>               attribute     having      the      value
>               PTHREAD_PRIO_PROTECT   and  the  calling
>               thread's priority  is  higher  than  the
>               mutex's current priority ceiling.

I can't find any place PTHREAD_PRIO_PROTECT is set by gnutls nor
weechat. Could there be some uninitialized data someplace?

_______________________________________________
Weechat-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/weechat-dev
Reply | Threaded
Open this post in threaded view
|

Re: [gnutls-devel] crash inside of wrap_nettle_rnd?

Jason A. Donenfeld
Okay, this awful awful patch "fixes" the weechat crashes. I'm sure
this is crippling security, as it essentially destroys any proper
respect for the locks, but...

Anyway, I'll leave it to the gnutls and weechat devs to get to the
bottom of this; I'm way over my head right now. My best guess right
now would be that gnutls isn't properly initializing some sort of
pthread struct, and it's an old bug, but one that is only triggered by
the heap setup of recent versions of weechat. This would account for
the PTHREAD_PRIO_PROTECT related error.

stern ~ # cat /etc/portage/patches/net-libs/gnutls/rnd-mutex-crash-hack.patch
Only in gnutls-3.2.10-modified: cscope.out
diff -ru gnutls-3.2.10/lib/nettle/rnd.c gnutls-3.2.10-modified/lib/nettle/rnd.c
--- gnutls-3.2.10/lib/nettle/rnd.c      2014-01-27 19:27:10.000000000 +0100
+++ gnutls-3.2.10-modified/lib/nettle/rnd.c     2014-02-12
23:54:34.926144545 +0100
@@ -42,8 +42,8 @@

 #define SOURCES 2

-#define RND_LOCK if (gnutls_mutex_lock(&rnd_mutex)!=0) abort()
-#define RND_UNLOCK if (gnutls_mutex_unlock(&rnd_mutex)!=0) abort()
+#define RND_LOCK gnutls_mutex_lock(&rnd_mutex)
+#define RND_UNLOCK gnutls_mutex_unlock(&rnd_mutex)

 enum {
        RANDOM_SOURCE_TRIVIA = 0,

_______________________________________________
Weechat-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/weechat-dev
Reply | Threaded
Open this post in threaded view
|

Re: [gnutls-devel] crash inside of wrap_nettle_rnd?

Jason A. Donenfeld
In reply to this post by Jason A. Donenfeld
Fixed.

Turned out that weechat wasn't linking against pthreads, so gnutls was
using the stubbed out version in glibc. Then later, a plugin would
load the non-stubbed version, which would mean the mutexes initialized
prior by gnutls were uninitialized, but then put to use by libpthread.

_______________________________________________
Weechat-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/weechat-dev