gnokii 0.6.31 problem with german umlauts

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

gnokii 0.6.31 problem with german umlauts

Stefan Rupprecht
Dear all,

since upgrade from gnokii 0.6.30 to 0.6.31, I've a problem with the
encoding of german umlauts.

---------------------------------------------------------------------------------------

eistest # echo "This is a testmessage äöüÄÖÜß" | gnokii --sendsms 1234567


GNOKII Version 0.6.31

LOG: debug mask is 0x1

Config read from file /root/.gnokiirc.

Cannot open logfile /root/.cache/gnokii/gnokii-errors

WARNING: cannot open logfile, logs will be directed to stderr

phone instance config:

model = fake

port = none

connection = serial

initlength = default

serial_baudrate = 19200

serial_write_usleep = -1

handshake = software

require_dcd = 0

smsc_timeout = 10

rfcomm_channel = 0

sm_retry = 0

Initializing...

Connecting

=====> ENTER gn_sms_send()

enc_chars: 29

ext_chars: 0

total: 26

size of the input string: 29 bytes

number of characters in the input string: 29 chars

=====> ENTER sms_send_single()

General Data Coding

Default Alphabet

         size: 33

         offset: 0

         encoded size: 33

         rawsms length: 37

         rawsms user data length: 33

dcs: 0x00

Length: 0x25

user_data_length: 0x21

ValidityIndicator: 2

user_data: 54747A0E4ACF416110BD3CA7B7CBF379F85C06FD36E5DFEFF7FBFD7EBFDFEFF703

Input is 29 bytes long

SMS is 33 octets long

Sending

AT+CMGS=45

0011000781214365F70000002554747A0E4ACF416110BD3CA7B7CBF379F85C06FD36E5DFEFF7FBFD7EBFDFEFF703

Send succeeded with reference 1!

---------------------------------------------------------------------------------------


Decoding of the generated PDU shows that the german umlauts were not
correct encoded:

USER DATA PART OF SM

     USER DATA LENGTH :    37 septets

     USER DATA (TEXT) :    This is a testmessage ?€????????????




With previous version 0.6.30, the encoding was correct:
(same pc, same compiler, same locale settings)


---------------------------------------------------------------------------------------


eistest # echo "This is a testmessage äöüÄÖÜß" | gnokii --sendsms 1234567


GNOKII Version 0.6.30

Couldn't read /root/.config/gnokii/config config file.

Couldn't read /etc/xdg/gnokii/config config file.

LOG: debug mask is 0x1

Config read from file /root/.gnokiirc.

Cannot open logfile /root/.cache/gnokii/gnokii-errors

WARNING: cannot open logfile, logs will be directed to stderr

phone instance config:

model = fake

port = none

connection = serial

initlength = default

serial_baudrate = 19200

serial_write_usleep = -1

handshake = software

require_dcd = 0

smsc_timeout = 10

rfcomm_channel = 0

sm_retry = 0

Initializing...

Connecting

Cannot read the SMSC number from your phone. If the sms send will fail, please use --smsc option explicitely giving the number.

=====> ENTER gn_sms_send()

General Data Coding

Default Alphabet

         encoded size: 26

         rawsms length: 29

         rawsms user data length: 26

dcs: 0x00

Length: 0x1d

user_data_length: 0x1a

ValidityIndicator: 2

user_data: 54747A0E4ACF416110BD3CA7B7CBF379F85C06EDF9FE2DD7EB01

Input is 29 characters long

SMS is 26 octets long

Number of extended alphabet chars: 0

Sending

AT+CMGS=38

0011000781214365F70000001D54747A0E4ACF416110BD3CA7B7CBF379F85C06EDF9FE2DD7EB01

Send succeeded with reference 1!

---------------------------------------------------------------------------------------

Decoding of PDU shows that the german umlauts were correct encoded:

USER DATA PART OF SM
        USER DATA LENGTH : 29 septets
        USER DATA (TEXT) : This is a testmessage äöüÄÖÜß


Can anybody help?

Stefan


_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Pawel Kot
Hi,

On Sun, Aug 12, 2012 at 3:46 PM, Stefan Rupprecht <[hidden email]> wrote:
> since upgrade from gnokii 0.6.30 to 0.6.31, I've a problem with the encoding
> of german umlauts.

And how does your locale look like? Is it the same with the git version?

take care,
--
Pawel Kot

_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Stefan Rupprecht
Hi Pawel,

Am 13.08.2012 09:26, schrieb Pawel Kot:
> And how does your locale look like?

My locale settings are:

LANG=C
LC_CTYPE=de_DE@euro
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

 > Is it the same with the git version?

I did not try the git version so far, because up to now I didn't hear
about git ;-)
But I will check it. (Sorry, I'm not an expert in these things).

Thanks,
Stefan


_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Stefan Rupprecht

> I did not try the git version so far, because up to now I didn't hear
> about git ;-)
> But I will check it. (Sorry, I'm not an expert in these things).
Meanwhile I tried the git version (0.6.32git), but the same problem as
in 0.6.31.

I compared the sourcefile gsm-encoding.c between 0.6.30 and 0.6.31 and
found out that there were some changes.
Maybe this changes could be the reason for encoding the umlauts wrong?

Stefan


_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Pawel Kot
In reply to this post by Stefan Rupprecht
Hi,

On Mon, Aug 13, 2012 at 9:31 PM, Stefan Rupprecht <[hidden email]> wrote:
> Am 13.08.2012 09:26, schrieb Pawel Kot:
>> And how does your locale look like?
> My locale settings are:
> LANG=C
[...]
> LC_ALL=

You need to set that. In ASCII locale there are no umlauts.

--
Pawel Kot

_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Stefan Rupprecht
Hi Pawel,

Am 13.08.2012 22:16, schrieb Pawel Kot:
>> LC_ALL=
> You need to set that. In ASCII locale there are no umlauts.
>
Now I set the locale:

LANG=de_DE@euro
LC_CTYPE="de_DE@euro"
LC_NUMERIC="de_DE@euro"
LC_TIME="de_DE@euro"
LC_COLLATE="de_DE@euro"
LC_MONETARY="de_DE@euro"
LC_MESSAGES="de_DE@euro"
LC_PAPER="de_DE@euro"
LC_NAME="de_DE@euro"
LC_ADDRESS="de_DE@euro"
LC_TELEPHONE="de_DE@euro"
LC_MEASUREMENT="de_DE@euro"
LC_IDENTIFICATION="de_DE@euro"
LC_ALL=de_DE@euro

but the result is the same: The umlauts were not encoded correctly.

I also tried to use not the commandline, but the smsd with mysql
support. I inserted a text "Test ÄÖÜ" with umlauts into the outbox
table, but also here the sms encoding is wrong.

I'll give up now and further use the version 0.6.30, where the encoding
works correct (gnokii commandline and smsd).

Nevertheless, thanks for support.
Stefan


_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Axel Gruber-2
Hello

i also had a lot of Problems with german "umlauts" - i solved it this way:

declare -x LANG="de_DE.UTF-8"

And i start SMSD this way:

 /usr/local/bin/smsd -u root -p password  -d smsgw -c xxxxx.xx -m
mysql --encoding UTF8

With this configuration i can use Gnokii  Version 0.6.31 here with
mysql with german "umlauts".

But i agree with you the "Umlaute Problem" on Linux is realy bad  - i
hate it ... - i still have some problems with a PerlScript wich sent
the recieved SMS by Mail - and here are all "umlauts" away .... - but
i´m on the best way to solve it ...
----
Mit freundlichen Grüßen
Autohaus A. Gruber OHG
Axel Gruber

Tel: 0807193200
Fax: 0807193202
E-Mail: [hidden email]
Internet: www.autohaus-gruber.net

Ihr starker MAZDA und HYUNDAI Partner 4mal in der Region - einmal auch
in Ihrer Nähe.


2012/8/16 Stefan Rupprecht <[hidden email]>:

> Hi Pawel,
>
> Am 13.08.2012 22:16, schrieb Pawel Kot:
>
>>> LC_ALL=
>>
>> You need to set that. In ASCII locale there are no umlauts.
>>
> Now I set the locale:
>
> LANG=de_DE@euro
> LC_CTYPE="de_DE@euro"
> LC_NUMERIC="de_DE@euro"
> LC_TIME="de_DE@euro"
> LC_COLLATE="de_DE@euro"
> LC_MONETARY="de_DE@euro"
> LC_MESSAGES="de_DE@euro"
> LC_PAPER="de_DE@euro"
> LC_NAME="de_DE@euro"
> LC_ADDRESS="de_DE@euro"
> LC_TELEPHONE="de_DE@euro"
> LC_MEASUREMENT="de_DE@euro"
> LC_IDENTIFICATION="de_DE@euro"
> LC_ALL=de_DE@euro
>
> but the result is the same: The umlauts were not encoded correctly.
>
> I also tried to use not the commandline, but the smsd with mysql support. I
> inserted a text "Test ÄÖÜ" with umlauts into the outbox table, but also here
> the sms encoding is wrong.
>
> I'll give up now and further use the version 0.6.30, where the encoding
> works correct (gnokii commandline and smsd).
>
> Nevertheless, thanks for support.
> Stefan
>
>
>
> _______________________________________________
> gnokii-users mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/gnokii-users

_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Pawel Kot
In reply to this post by Stefan Rupprecht
Hi,

On Thu, Aug 16, 2012 at 10:47 PM, Stefan Rupprecht <[hidden email]> wrote:

> Hi Pawel,
>
> Am 13.08.2012 22:16, schrieb Pawel Kot:
>
>>> LC_ALL=
>>
>> You need to set that. In ASCII locale there are no umlauts.
>>
> Now I set the locale:
>
> LANG=de_DE@euro

That should be UTF-8 related I guess.

take care,
--
Pawel Kot

_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Daniele Forsi-2
2012/8/16 Pawel Kot:

> That should be UTF-8 related I guess.

I can reproduce the bug with a non -UTF-8 locale and I will look into
it tomorrow

I did:
dpkg-reconfigure locales
and chose it_IT.ISO-8859-15@euro
then:
echo "This is a testmessage äöüÄÖÜß" | iconv -t iso-8859-15 >/tmp/x
export LANG=it_IT.iso885915@euro
gnokii --phone fake --sendsms 1234567 </tmp/x
(to see the contents of the temporary file you also need to change the
encoding used by the terminal to display output because setting LANG
doesn't affect the running shell, only the processes started from it)

in the end I get the same PDU as the OP

this is the wrong output with ISO-8859-15 locale:
Input is 29 bytes long
SMS is 33 octets long
Sending
AT+CMGS=45
0011000781214365F70000002554747A0E4ACF416110BD3CA7B7CBF379F85C06FD36E5DFEFF7FBFD7EBFDFEFF703

this is the right output with an UTF-8 locale and it is decoded
correctly with both locales
Input is 36 bytes long
SMS is 26 octets long
Sending
AT+CMGS=38
0011000781214365F70000001D54747A0E4ACF416110BD3CA7B7CBF379F85C06EDF9FE2DD7EB01

--
Daniele Forsi

_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Lsg
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Lsg
W dniu 2012-08-17 00:49, Daniele Forsi pisze:

> 2012/8/16 Pawel Kot:
>
>> That should be UTF-8 related I guess.
>
> I can reproduce the bug with a non -UTF-8 locale and I will look into
> it tomorrow
>
> I did:
> dpkg-reconfigure locales
> and chose it_IT.ISO-8859-15@euro
> then:
> echo "This is a testmessage äöüÄÖÜß" | iconv -t iso-8859-15 >/tmp/x
> export LANG=it_IT.iso885915@euro

Maybe use something like that will help with foreign characters
/*
| -------------------------------------------------------------------
| Foreign Characters
| -------------------------------------------------------------------
| This file contains an array of foreign characters for transliteration
| conversion used by the Text helper
|
*/
$foreign_characters = array(
        '/ä|æ|ǽ/' => 'ae',
        '/ö|œ/' => 'oe',
        '/ü/' => 'ue',
        '/Ä/' => 'Ae',
        '/Ü/' => 'Ue',
        '/Ö/' => 'Oe',
        '/À|Á|Â|Ã|Ä|Å|Ǻ|Ā|Ă|Ą|Ǎ|Α|Ά/' => 'A',
        '/à|á|â|ã|å|ǻ|ā|ă|ą|ǎ|ª|α|ά/' => 'a',
        '/Ç|Ć|Ĉ|Ċ|Č/' => 'C',
        '/ç|ć|ĉ|ċ|č/' => 'c',
        '/Ð|Ď|Đ|Δ/' => 'Dj',
        '/ð|ď|đ|δ/' => 'dj',
        '/È|É|Ê|Ë|Ē|Ĕ|Ė|Ę|Ě|Ε|Έ/' => 'E',
        '/è|é|ê|ë|ē|ĕ|ė|ę|ě|έ|ε/' => 'e',
        '/Ĝ|Ğ|Ġ|Ģ|Γ/' => 'G',
        '/ĝ|ğ|ġ|ģ|γ/' => 'g',
        '/Ĥ|Ħ/' => 'H',
        '/ĥ|ħ/' => 'h',
        '/Ì|Í|Î|Ï|Ĩ|Ī|Ĭ|Ǐ|Į|İ|Η|Ή|Ί|Ι|Ϊ/' => 'I',
        '/ì|í|î|ï|ĩ|ī|ĭ|ǐ|į|ı|η|ή|ί|ι|ϊ/' => 'i',
        '/Ĵ/' => 'J',
        '/ĵ/' => 'j',
        '/Ķ|Κ/' => 'K',
        '/ķ|κ/' => 'k',
        '/Ĺ|Ļ|Ľ|Ŀ|Ł|Λ/' => 'L',
        '/ĺ|ļ|ľ|ŀ|ł|λ/' => 'l',
        '/Ñ|Ń|Ņ|Ň|Ν/' => 'N',
        '/ñ|ń|ņ|ň|ʼn|ν/' => 'n',
        '/Ò|Ó|Ô|Õ|Ō|Ŏ|Ǒ|Ő|Ơ|Ø|Ǿ|Ο|Ό|Ω|Ώ/' => 'O',
        '/ò|ó|ô|õ|ō|ŏ|ǒ|ő|ơ|ø|ǿ|º|ο|ό|ω|ώ/' => 'o',
        '/Ŕ|Ŗ|Ř|Ρ/' => 'R',
        '/ŕ|ŗ|ř|ρ/' => 'r',
        '/Ś|Ŝ|Ş|Ș|Š|Σ/' => 'S',
        '/ś|ŝ|ş|ș|š|ſ|σ|ς/' => 's',
        '/Ț|Ţ|Ť|Ŧ|τ/' => 'T',
        '/ț|ţ|ť|ŧ/' => 't',
        '/Ù|Ú|Û|Ũ|Ū|Ŭ|Ů|Ű|Ų|Ư|Ǔ|Ǖ|Ǘ|Ǚ|Ǜ/' => 'U',
        '/ù|ú|û|ũ|ū|ŭ|ů|ű|ų|ư|ǔ|ǖ|ǘ|ǚ|ǜ|υ|ύ|ϋ/' => 'u',
        '/Ý|Ÿ|Ŷ|Υ|Ύ|Ϋ/' => 'Y',
        '/ý|ÿ|ŷ/' => 'y',
        '/Ŵ/' => 'W',
        '/ŵ/' => 'w',
        '/Ź|Ż|Ž|Ζ/' => 'Z',
        '/ź|ż|ž|ζ/' => 'z',
        '/Æ|Ǽ/' => 'AE',
        '/ß/'=> 'ss',
        '/IJ/' => 'IJ',
        '/ij/' => 'ij',
        '/Œ/' => 'OE',
        '/ƒ/' => 'f',
        '/ξ/' => 'ks',
        '/π/' => 'p',
        '/β/' => 'v',
        '/μ/' => 'm',
        '/ψ/' => 'ps',
);



_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Daniele Forsi-2
2012/8/17 lsg:

> Maybe use something like that will help with foreign characters

thank you but glib is supposed to handle encodings for us

Pawel, I have a fix for the wrong encoding which is due to
gn_sms_send() converting in place the buffer to UTF-8 while later code
assumes it's still in the system locale, which for the OP is
ISO-8859-15 but my fix segfaults because str is NULL if the input
can't be converted, eg. with UTF-8 locale this segfaults after my
"fix" while it sends an empty SMS without my fix:
echo -e "\xdf" | gnokii --phone fake --sendsms 123
it doesn't matter if there are valid chars before or after the invalid one

the easier fix would be to exit with an error from gn_sms_send() if
g_locale_to_utf8() fails sending an empty SMS makes no sense to me,
but IIRC when libgnokii was using only iconv() invalid chars where
replaced with question marks which was at least an hint of where the
things went wrong

diff --git a/common/gsm-sms.c b/common/gsm-sms.c
index 77ab16d..669f3c7 100644
--- a/common/gsm-sms.c
+++ b/common/gsm-sms.c
@@ -1642,14 +1642,12 @@ GNOKII_API gn_error gn_sms_send(gn_data *data,
struct gn_statemachine *state)
                            data->sms->user_data[i].type ==
GN_SMS_DATA_NokiaText) {
                                str =
g_locale_to_utf8(data->sms->user_data[i].u.text, -1, &inlen, &outlen,
NULL);
                                data->sms->user_data[i].chars =
g_utf8_strlen(str, outlen);
-                               memset(data->sms->user_data[i].u.text,
0, sizeof(data->sms->user_data[i].u.text));
-
g_utf8_strncpy(data->sms->user_data[i].u.text, str,
data->sms->user_data[i].chars);
-                               g_free(str);
                                /* Let's make sure the encoding is correct */
-                               enc =
char_def_alphabet_string_stats(data->sms->user_data[i].u.text,
&enc_chars, &ext_chars);
+                               enc =
char_def_alphabet_string_stats(str, &enc_chars, &ext_chars);
                                if (enc == GN_SMS_DCS_UCS2)
                                        data->sms->dcs.u.general.alphabet = enc;
                                data->sms->user_data[i].chars = enc_chars;
+                               g_free(str);
                        } else {
                                /* FIXME: that's wrong for Nokia
Multipart Picture Message, does anybody still use it? */
                                data->sms->dcs.u.general.alphabet =
GN_SMS_DCS_8bit;


this is the segfault:
#0  0xb7d47ef0 in g_utf8_validate () from /lib/i386-linux-gnu/libglib-2.0.so.0
#1  0x08079877 in char_def_alphabet_string_stats (str=str@entry=0x0,
enc_chars=enc_chars@entry=0xbffeebdc,
ext_chars=ext_chars@entry=0xbffeebe0)
    at ../../gnokii/common/gsm-encoding.c:516
#2  0x08076071 in gn_sms_send (data=data@entry=0xb78815c4,
state=state@entry=0xb7881008) at ../../gnokii/common/gsm-sms.c:1646
#3  0x0805a4dd in sendsms (argc=argc@entry=5, argv=0xbffff4c4,
data=0xb78815c4, state=0xb7881008) at
../../gnokii/gnokii/gnokii-sms.c:333
#4  0x0804da6f in parse_options (argc=argc@entry=5, argv=<optimized
out>) at ../../gnokii/gnokii/gnokii.c:890
#5  0x0804d1e9 in parse_options (argc=argc@entry=5,
argv=argv@entry=0xbffff4c4) at ../../gnokii/gnokii/gnokii.c:857
#6  0x0804c697 in main (argc=5, argv=0xbffff4c4) at
../../gnokii/gnokii/gnokii.c:1234

--
Daniele Forsi

_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Pawel Kot
Hi,

On Sat, Aug 18, 2012 at 1:07 PM, Daniele Forsi <[hidden email]> wrote:

> 2012/8/17 lsg:
>
>> Maybe use something like that will help with foreign characters
>
> thank you but glib is supposed to handle encodings for us
>
> Pawel, I have a fix for the wrong encoding which is due to
> gn_sms_send() converting in place the buffer to UTF-8 while later code
> assumes it's still in the system locale, which for the OP is
> ISO-8859-15

Well, yes... It may be tha case. I might wanted to simplify the code
with some assumptions, like "who the hell does not use utf-8 these
days". My bad it is not documented not correctly handled.

Stefan - is there any particular reason you don't use UTF-8?

> but my fix segfaults because str is NULL if the input
> can't be converted, eg. with UTF-8 locale this segfaults after my
> "fix" while it sends an empty SMS without my fix:
> echo -e "\xdf" | gnokii --phone fake --sendsms 123
> it doesn't matter if there are valid chars before or after the invalid one
>
> the easier fix would be to exit with an error from gn_sms_send() if
> g_locale_to_utf8() fails sending an empty SMS makes no sense to me,
> but IIRC when libgnokii was using only iconv() invalid chars where
> replaced with question marks which was at least an hint of where the
> things went wrong

Well, I need some time to look at it and think again trying to remind
myself rationale for all this stuff.

thanks,
--
Pawel Kot

_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users
Reply | Threaded
Open this post in threaded view
|

Re: gnokii 0.6.31 problem with german umlauts

Stefan Rupprecht
Am 20.08.2012 10:06, schrieb Pawel Kot:
> Stefan - is there any particular reason you don't use UTF-8?

Not a particular reason. But "de_DE@euro" is preconfigured on the
system, and I'm not really a linux expert to know how I can change it,
especially if a change would have any side effects for the other
applications and scripts running on these machine. So I don't like to
change it if it is not really necessary.

Kind regards,
Stefan


_______________________________________________
gnokii-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/gnokii-users