[nmh-workers] nmh 1.7.1: both bcc and dcc broken for mts sendmail/pipe

[Alexander Zangerl-4]

i've just raised bug https://savannah.nongnu.org/bugs/?55700
and i'm working on a fix for this for debian.

however, the best way forward isn't totally clear to me, hence this email.

the issues that i'm trying to find good solutions for:

1. with sendmail/pipe the headers of what we pass to the mta must make
sense for message routing, but the warning-encrusted modified draft
baked for bccfil doesn't work because it has no to: and no bcc: headers,
so the mta rejects that as unroutable. the original message that post
also submits to the mta is left with bcc intact (and
thus the mta does deliver it to the blind recipients), which duplicates
the (currently nonfunctional) warning-encrusted message.

so, in order to make bcc: be both blind and warning-encrusted as per
the documentation we'd have to modify the original draft and nuke
its bcc: header, and add a bcc: header to the bccfil draft.

the patch that i've already attached to the bug report doesn't go that far,
it makes bcc with sendmail/pipe work like dcc elsewhere. (it also doesn't
contain any documentation updates.)

my question: is that good enough? or should we aim for bcc working exactly
the same regardless of mts?

2. the docs say dcc isn't supported for sendmail/pipe, which is ok.
however, that fact is not overly visibly documented, which is slightly bad.

what's quite bad in my opinion is that post with sendmail/pipe leaves
the dcc: header in place and so that leaks to the actual recipients (and doesn't
cause any message to be sent to the dcc: recipients).

my question: wouldn't it be best if dcc in the sendmail/pipe case was
handled by simply replacing the header with bcc: and letting the mta do
its job? or should post with sendmail/pipe reject messages with dcc?

regards
az


--
Alexander Zangerl + GPG Key 2FCCF66BB963BD5F + http://snafu.priv.at/
bash: syntax error near unexpected token `=:)'

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Ken Hornstein-2]

>i've just raised bug https://savannah.nongnu.org/bugs/?55700
>and i'm working on a fix for this for debian.

Can I just say, up front, that this just another example of why sendmail/pipe
is a TERRIBLE idea?  There was a reason it was shoved into the undocumented
"spost" program in MH.  But, fine .... I just had to get that off my chest.
Moving on.

>1. with sendmail/pipe the headers of what we pass to the mta must make
>sense for message routing, but the warning-encrusted modified draft
>baked for bccfil doesn't work because it has no to: and no bcc: headers,
>so the mta rejects that as unroutable. the original message that post
>also submits to the mta is left with bcc intact (and
>thus the mta does deliver it to the blind recipients), which duplicates
>the (currently nonfunctional) warning-encrusted message.

Alright, I guess that is a bug that has been around for ... 7 years?  That's
when spost was merged with post.

>so, in order to make bcc: be both blind and warning-encrusted as per
>the documentation we'd have to modify the original draft and nuke
>its bcc: header, and add a bcc: header to the bccfil draft.
>
>the patch that i've already attached to the bug report doesn't go that far,
>it makes bcc with sendmail/pipe work like dcc elsewhere. (it also doesn't
>contain any documentation updates.)
>
>my question: is that good enough? or should we aim for bcc working exactly
>the same regardless of mts?

Many electrons have been spilled about Bcc, Dcc, and nmh's use of
them.  I kind of feel that nmh's Bcc is kind of dumb, but that behavior
has been around for approximately forever so I think changing that is
not a good idea.  And it strikes me as a bad idea to have Bcc behave
differently depending on the MTS you are using.  So I think when
stripping Bcc out of the original draft and putting it IN the Bcc draft
makes the most sense.  I don't even think this is hard, because we already
do that for other MTSes (well, other than post is kind of a mess, but I
think the pieces are there).

>2. the docs say dcc isn't supported for sendmail/pipe, which is ok.
>however, that fact is not overly visibly documented, which is slightly bad.

Well ... where should this be documented better?  I am all for improving
our documentation, it's just there is a lot of it and it could use some
organizational improvements.  Really, if you have some suggestions I would
be glad to make them; I don't want to shove this on you, especially if
you are contributing a bug fix.

>my question: wouldn't it be best if dcc in the sendmail/pipe case was
>handled by simply replacing the header with bcc: and letting the mta do
>its job? or should post with sendmail/pipe reject messages with dcc?

I firmly believe that post should reject the message immediately and nothing
should be sent.

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Alexander Zangerl-4]

On Tue, 12 Feb 2019 09:57:49 -0500, Ken Hornstein writes:
>And it strikes me as a bad idea to have Bcc behave
>differently depending on the MTS you are using.  So I think when
>stripping Bcc out of the original draft and putting it IN the Bcc draft
>makes the most sense.

very good, we're thinking along the same lines here.

>Well ... where should this be documented better?

what i meant was that the send manpage for example just says 'if you have dcc
and anything but sendmail/pipe then...' but doesn't say what happens
when you're on pipe.
it's similar regarding that bcc is meant to produce those warning
encrustations; not having used non-silent bcc in decades
i couldn't figure that out fully from the manpages, but had to look
at the source.

>Really, if you have some suggestions I would
>be glad to make them; I don't want to shove this on you, especially if
>you are contributing a bug fix.

no problem, i'll try to update the send and post manpages where applicable.

>I firmly believe that post should reject the message immediately and nothing
>should be sent.

thanks for your advice; i'll work towards these in the next few days
and will let you know when i have something workable.

regards
az


--
Alexander Zangerl + GPG Key 2FCCF66BB963BD5F + http://snafu.priv.at/
Windows95 is a 32-bit extension to a 16-bit shell of an 8-bit OS originally
written for a 4-bit microprocessor by a 2-bit company that can't stand
1-bit of competition.

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Ken Hornstein-2]

>what i meant was that the send manpage for example just says 'if you have dcc
>and anything but sendmail/pipe then...' but doesn't say what happens
>when you're on pipe.

That's a fair point.  I guess we the obvious answer is that when you're
using sendmail/pipe, it does nothing, which is accurate (but I think
we both agree that is probably incorrect).

>it's similar regarding that bcc is meant to produce those warning
>encrustations; not having used non-silent bcc in decades
>i couldn't figure that out fully from the manpages, but had to look
>at the source.

Hm.  Well, here's what the manpage says:

       If a "Bcc:" field is encountered, its addresses will be used for deliv-
       ery,  and  the  "Bcc:"  field  will be removed from the message sent to
       sighted recipients.  The blind recipients will receive an entirely  new
       message  with  a  minimal  set of headers.  Included in the body of the
       message will be a copy of the message sent to the sighted recipients.

Which I think is an accurate description of what exactly happens (I
mean, with sendmail/pipe it doesn't currently WORK, but that's a bug).  I am
always open to a better explanation.

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Alexander Zangerl-4]

On Tue, 12 Feb 2019 09:57:49 -0500, Ken Hornstein writes:
>So I think when
>stripping Bcc out of the original draft and putting it IN the Bcc draft
>makes the most sense.

done. patch is attached.

>>or should post with sendmail/pipe reject messages with dcc?
>I firmly believe that post should reject the message immediately and nothing
>should be sent.

also done.

i've updated the post and send manpages a little; they now both
spell out the bcc/dcc situation more or less the same way and do mention
the dcc rejection aspect.


regards
az


--
Alexander Zangerl + GPG Key 2FCCF66BB963BD5F + http://snafu.priv.at/
When you understand UNIX, you will understand the world.
When you understand NT....you will understand NT -- R. Thieme

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Ralph Corderoy]

Hi,

Two of az's recent emails to the list have been bounced by Gmail as
spam, with no further detail why given.  Both bounces were for the same
gmail.com subscriber.  I'm assuming the other 17 gmail.com subscribers
received the email OK otherwise Gmail would have sent separate bounces?

Perhaps this is because that particular recipient, matching /^j.*ra/,
has been tagging nmh emails as spam, training his `personal' Gmail to
bounce them.  Perhaps not.  But I thought the list, and archive, should
know.

--
Cheers, Ralph.

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Ralph Corderoy]

Hi az,

Thanks for the patch, Ken knows the topic better than me, but the man
page caught my eye.

>  If a \*(lqBcc:\*(rq field is encountered, its addresses will be used for
>  delivery, and the \*(lqBcc:\*(rq field will be removed from the message
> -sent to sighted recipients.  The blind recipients will receive an entirely
> -new message with a minimal set of headers.  Included in the body of the
> -message will be a copy of the message sent to the sighted recipients.
> +sent to sighted recipients. The blind recipients will receive an entirely
> +new message with a minimal set of headers. The body of this new message
> +will contain a copy of the message sent to the sighted recipients, either
> +marked up with the indicator text "Blind-Carbon-Copy" or encapsulated
> +as a MIME digest.

A man page is troff source and so the `text' parts aren't free-format
but instructions to the typesetter.  In particular, the end of a
sentence must be followed by two spaces to indicate to the formatter
that it is indeed the end of a sentence and not a word following an
abbreviation or something else that happens to end in a full stop.
This is independent of any preference for whether the post-sentence gap
should be wider than an inter-word one.

Because of this, it's often easier to end the line at the end of a
sentence.  This also makes diffs simpler because they tend to stay
within a sentence and not ripple through a re-formatted paragraph.
For a similar reason, long lines are sometimes broken after a comma.

The «"» around `Blind-Carbon-Copy' should be \(lq and \(rq, or the
equivalent strings for consistency with the style used at start of the
paragraph.

> +recipients. *WARNING* Recipients listed in the \*(lqDcc:\*(rq field receive no

I'd suggest «.B Warning:» rather than ASCII mark-up and shouting.  :-)
Though, reading on, I realise that you copied how it was already done
elsewhere.

Given I'm writing about the patch anyway,

> -    fprintf (out, "BCC:\n");
> +
> +    /* for sendmail/pipe, insert all bcc recipients here so that the email can be routed based on the bcc: header */
> +    if (sm_mts == MTS_SENDMAIL_PIPE)
> +    {
> +       char *allbcc = NULL;
> +       struct mailname *lp;
> +
> +       for (lp = localaddrs.m_next; lp; lp = lp->m_next)
> +  if (lp->m_bcc)
> +     allbcc = allbcc? add(concat(", ", lp->m_mbox, NULL), allbcc)
> + : mh_xstrdup(lp->m_mbox);
> +       for (lp = netaddrs.m_next; lp; lp = lp->m_next)
> +  if (lp->m_bcc)
> +     allbcc = allbcc? add(
> + concat(", ", lp->m_mbox, "@", lp->m_host, NULL),
> + allbcc)
> + : concat(lp->m_mbox, "@", lp->m_host, NULL);
> +       if (allbcc)
> +       {
> +  fprintf (out, "BCC: %s\n",allbcc);
> +  free(allbcc);
> +       }
> +    }
> +    else
> +    {
> +       fprintf (out, "BCC:\n");
> +    }

I'd be tempted to make it an if-then with no else clause by hoisting the
"BCC:" prefix and "\n" suffix outside of the if-then.

And it tends to simplify the code when having to join items with a
separator like ", " if it always starts every item, giving ", foo, bar",
and then if allbcc isn't NULL then `allbcc + 2' can be printed.  (It can
be "foo, bar, " instead but then the end of the string needs finding to
shorten it by two.)  I think this would remove the ternary operators and
duplicated terms like `lp->m_mbox, "@", lp->m_host, NULL' so the reader
has less to parse and check they're the same.

--
Cheers, Ralph.

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Valdis Klētnieks]

On Thu, 14 Feb 2019 17:30:28 +0000, Ralph Corderoy said:

> The «"» around `Blind-Carbon-Copy' should be \(lq and \(rq, or the
> equivalent strings for consistency with the style used at start of the
> paragraph.

And in today's "get off my lawn" rant.. :)

Am I the only guy who's been bitten by documentation that has single
and double quotes that look cut-n-paste-able but actually aren't?

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Paul Fox-3]

[hidden email] wrote:
 > On Thu, 14 Feb 2019 17:30:28 +0000, Ralph Corderoy said:
 >
 > > The  �" � around `Blind-Carbon-Copy' should be \(lq and \(rq, or the
 > > equivalent strings for consistency with the style used at start of the
 > > paragraph.
 >
 > And in today's "get off my lawn" rant.. :)
 >
 > Am I the only guy who's been bitten by documentation that has single
 > and double quotes that look cut-n-paste-able but actually aren't?

No, definitely not.  

=----------------------
paul fox, [hidden email] (arlington, ma, where it's 29.7 degrees)



--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Paul Vixie-2]

[hidden email] wrote on 2019-02-14 14:16:
> ...
>
> Am I the only guy who's been bitten by documentation that has single
> and double quotes that look cut-n-paste-able but actually aren't?

nope.

--
P Vixie


--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[bergman-5]

In the message dated: Thu, 14 Feb 2019 17:18:28 -0500,
The pithy ruminations from Paul Fox on
[Re: [nmh-workers] nmh 1.7.1: both bcc and dcc broken for mts sendmail/pipe] were:
=> [hidden email] wrote:
=>  > On Thu, 14 Feb 2019 17:30:28 +0000, Ralph Corderoy said:
=>  >
=>  > > The  �" � around `Blind-Carbon-Copy' should be \(lq and \(rq, or the
=>  > > equivalent strings for consistency with the style used at start of the
=>  > > paragraph.
=>  >
=>  > And in today's "get off my lawn" rant.. :)
=>  >
=>  > Am I the only guy who's been bitten by documentation that has single
=>  > and double quotes that look cut-n-paste-able but actually aren't?
=>
=> No, definitely not.  

Same.

I've got some experience in typesetting, and appreciate a good-looking
printed page. However, in the electronic copy I'll always take function
over form. Plain double- or single-quotes instead of left-quote and
right-quote, please.

Mark "don't get me started on things that look like space separators but paste as printable Unicode characters" Bergman


=>
=> =----------------------
=> paul fox, [hidden email] (arlington, ma, where it's 29.7 degrees)
=>
=>
=>
=> --
=> nmh-workers
=> https://lists.nongnu.org/mailman/listinfo/nmh-workers
--
Mark Bergman    Biker, Rock Climber, SCUBA Diver, Unix mechanic, IATSE #1 Stagehand
'94 Yamaha GTS1000A^1      2015 Aprilia Caponord
[hidden email]       https://www.flickr.com/photos/rmsppu

I want a newsgroup with a infinite S/N ratio! Now taking CFV on:
rec.motorcycles.stagehands.pet-bird-owners.pinballers.unix-supporters
15+ So Far--Want to join? Check out: http://www.panix.com/~bergman 

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Ralph Corderoy]

Hi

> > > > The «"» around `Blind-Carbon-Copy' should be \(lq and \(rq, or
> > > > the equivalent strings for consistency with the style used at
> > > > start of the paragraph.
> ...
> > > Am I the only guy who's been bitten by documentation that has
> > > single and double quotes that look cut-n-paste-able but actually
> > > aren't?
>
> Same.
>
> I've got some experience in typesetting, and appreciate a good-looking
> printed page. However, in the electronic copy I'll always take
> function over form. Plain double- or single-quotes instead of
> left-quote and right-quote, please.

But in this case the text is prose, not pipelines to paste, and it
should have the best quality quotes available on the output device.

Having ASCII appear as pasteable ASCII is a different kettle of worms.

--
Cheers, Ralph.

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Alexander Zangerl-4]

On Thu, 14 Feb 2019 17:30:28 +0000, Ralph Corderoy writes:
>In particular, the end of a
>sentence must be followed by two spaces to indicate to the formatter
>that it is indeed the end of a sentence and not a word following an
>abbreviation or something else that happens to end in a full stop.

thanks, i wasn't aware of that particular *roff quirk. sorry about missing
the doublequotes.

>Though, reading on, I realise that you copied how it was already done
>elsewhere.

yes, i aimed for a minimum delta.

>I'd be tempted to make it an if-then with no else clause by hoisting the
>"BCC:" prefix and "\n" suffix outside of the if-then.

hmm. i see your point, but don't entirely agree. my aim here was to
contain all the related logic within the smallest possible/sensible
horizon.

apart from the small ugliness of having the string "bcc:"
hardcoded twice i prefer that 'if' block doing its thing (and all of
its thing!) over strings conditionally accumulating across a
pageful of code or more.

>(It canbe "foo, bar, " instead but then the end of the string needs finding to
>shorten it by two.)
...
>I think this would remove the ternary operators and
>duplicated terms like `lp->m_mbox, "@", lp->m_host, NULL' so the reader
>has less to parse and check they're the same.

true, but again i beg to differ; i find parsing string retrofiddlery like that
shortening after lazily accumulating them earlier quite a bit less
clean than one ternary.

anyway, these are my personal preferences; i hope we can agree to disagree
and that somebody with commit rights will apply the patch or equivalent
to the code.

regards
az


--
Alexander Zangerl + GPG Key 2FCCF66BB963BD5F + http://snafu.priv.at/
>We're talking about a bleeding mainframe, mate.
It's got hemoglobin in its fluorinert?
 -- Peter da Silva and AdB

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Ken Hornstein-2]

>The «"» around `Blind-Carbon-Copy' should be \(lq and \(rq, or the
>equivalent strings for consistency with the style used at start of the
>paragraph.

So, in a mostly unrelated note ... I couldn't help noticing that Ralph
used guillemets («») in one of his messages on this thread (way to push
non-US-ASCII characters, Ralph!), and after a series of replies to his note
things devolved into classic mojibake.  And since hopefully most everyone
on this thread is an nmh user, I wanted to understand why, because really
that shouldn't have happened.

I went back to the raw archives (ftp://lists.gnu.org/nmh-workers/2019-02)
because the mailing list software will sometimes translate stuff into
base64 encoding when it sees non-ASCII characters.  And, well, I hate to
assign blame, but I think it's a bit unavoidable ... please, don't anyone
take this as a personal attack, I am just trying to understand how we
could do better.

Ralph's original note containing the guillemets (Message-Id
<[hidden email]>) was text/plain, a
character set of utf-8, and encoded using quoted-printable.  The
characters were encoded properly using quoted-printable, specifically
they were listed as =C2=AB and =C2=BB.

Valdis was the first reply to that (Message-ID
<[hidden email]>), and HIS email was text/plain,
character set iso-8859-1, and encoded using quoted-printable.  He quoted
Ralph's message, and the guillemets were encoded as =AB and =BB.  Which seems
correct to me.

Paul Fox replied to Valdis's note (Message-Id
<[hidden email]>), and THAT note
was text/plain, character set UTF-8, encoded using quoted-printable ...
but it seems like this was the start of where things went off the rails.
The original line in Valdis's email was (in raw form):

   > The =AB=22=BB around ...

But in Paul's note it ended up as (extra > added in the reply)

   > > The  =AB" =BB around

This is NOT correct.  First, there is an extra space in front of
the encoded bytes.  Secondly, they're not valid UTF-8; they're the
ISO-8859-1 bytes.  So I am guessing whatever Paul used to quote the reply
didn't translate the ISO-8859-1 characters properly into UTF-8.

However, whatever Mark Bergman uses for email actually made an intelligent
decision.  When he replied to Paul's note, those invalid UTF-8 characters
got converted to the Unicode Replacement Character (U+FFFD), which was
sent out as =EF=BF=BD (utf-8, quoted-printable).

Further muddying the waters ... when Ralph replied to Mark's email,
those Unicode Replacement Characters somehow got converted back to
the correct guillemets (=C2=AB and =C2=BB).  Which means Ralph has
perhaps the most intelligent reply quoting program ever and he should
immediately share it as it would revolutionize AI, or he went back and
manually corrected it when he replied to Mark's note.  I'm 50/50 on
which one of those scenarios is more likely.

If anyone involved with this email thread wants to pipe up with some
more explanation on what exactly they used to compose their email
replies, I would love to hear it.  No judgements; I just want to know
how nmh could help everyone do better.  Like, do we need to include
better tools for composing reply messages?  Well, duh, the answer to
that is "yes", and I think replyfilter does ok here but obviously we
need to do better.  But if we're SENDING something that is not valid
UTF-8, should we be smarter and flag it?  People were upset when we
refused to send out 8-bit characters when your locale was US-ASCII (I
mean, REALLY?  I couldn't believe it), so I don't know what makes sense.
Sending out invalid UTF-8 just seems wrong to me.

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[David Levine-3]

az writes:

> apart from the small ugliness of having the string "bcc:"
> hardcoded twice i prefer that 'if' block doing its thing (and all of
> its thing!) over strings conditionally accumulating across a
> pageful of code or more.

I agree with your coding decisions, but don't feel strongly if Ralph does.

One warning with the flags I use:

    uip/post.c:874:14: warning: increment of a boolean expression [-Wbool-operation]
        badmsg++;
              ^~

And of course we need tests :-)  Attached is a first pass.  With Bcc, the test passed before your fix because fakesendmail, used by the tests, was too lenient.

David


--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Valdis Klētnieks]

On Thu, 14 Feb 2019 21:08:41 -0500, Ken Hornstein said:

> Valdis was the first reply to that (Message-ID
> <[hidden email]>), and HIS email was text/plain,
> character set iso-8859-1, and encoded using quoted-printable.  He quoted
> Ralph's message, and the guillemets were encoded as =AB and =BB.  Which seems
> correct to me.

> But in Paul's note it ended up as (extra > added in the reply)
>
>    > > The  =AB" =BB around

There's something else wonky about Paul's note. My preliminary guess is that
something got mangled when MailMan tacked the  us-ascii footer onto Paul's
utf-8.  The end result is that exmh gets confoozled, even though it usually
gets this sort of thing right.

Not sure yet why, and I don't see anything in the raw message that would cause
this. Damn. Now it's going to bug me till I figure it out. :)

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Ken Hornstein-2]

>There's something else wonky about Paul's note. My preliminary guess is
>that something got mangled when MailMan tacked the us-ascii footer onto
>Paul's utf-8.  The end result is that exmh gets confoozled, even though
>it usually gets this sort of thing right.

FWIW, I saw the same exact thing.  I thought maybe it was because of the
invalid encoding, but maybe that isn't right (it doesn't make sense when
I type it out loud; "normal" nmh can view it fine).  AFAIK, on THIS list
Mailmail always is able to append the footer to the message properly and
make sure the message is valid MIME (I am on another where it always
appends it in us-ascii, even when the message is encoded in base64 which
causes exmh to fail to read it).

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Valdis Klētnieks]

On Thu, 14 Feb 2019 22:51:43 -0500, Ken Hornstein said:

> make sure the message is valid MIME (I am on another where it always
> appends it in us-ascii, even when the message is encoded in base64 which
> causes exmh to fail to read it).

Thanks for reminding me, that's now on my to-fix list.

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Valdis Klētnieks]

On Thu, 14 Feb 2019 22:51:43 -0500, Ken Hornstein said:
> >There's something else wonky about Paul's note. My preliminary guess is
> >that something got mangled when MailMan tacked the us-ascii footer onto
> >Paul's utf-8.  The end result is that exmh gets confoozled, even though
> >it usually gets this sort of thing right.
>
> FWIW, I saw the same exact thing.  I thought maybe it was because of the
> invalid encoding, but maybe that isn't right

Found it.

Content-ID: <6521.1550182708.1@grass>

Removing it causes exmh to behave. Added to my bug list.

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

[Robert Elz]

    Date:        Thu, 14 Feb 2019 21:08:41 -0500
    From:        Ken Hornstein <[hidden email]>
    Message-ID:  <[hidden email]>

  | >The «"» around `Blind-Carbon-Copy'

I am leaving that there just so you can see what happens...   What I
see when composing this is (ignoring my "|" quoting marker, ">The "
(which I assume is fine for everyone), capital A with a caret (I hope
that is the right name, like the ^ char, but smaller), the opening
guillemets (never heard that name before...) a normal double quote
(ascii), another capital A-caret, and the closing guillemets (and then
a space, and the rest of the text).

  | If anyone involved with this email thread wants to pipe up with some
  | more explanation on what exactly they used to compose their email
  | replies, I would love to hear it.

I wasn't, but am now ... I am using exmh (sedit) and nmh (and then
sendmail/smtp but I doubt that is in any way related).

I don't use replyfilter, I use mhl

The exmh-defaults line for the button I pressed to reply contains...

        *Mops.irepl.command: Msg_Reply -cc to -cc cc -filt mhl_irepl

(irepl is in the *Mops.ubuttonlist list).

mhl_irepl (aside from the generation of the headers), is just

        body:component="  | ",width=1000

The original message file (the one containing your message) in my
Mail tree looks correct, and exmh displays it correctly.

What I think is happening, is that everything I do is "un-localed",
that is, I have no LC_* or LANG settings at all, which means that
everything runs in the C (aka POSIX) locale (more or less US-ASCII).

If I use nmh (ie: show) to look at your message, I see:

        >The ?"? around `Blind-Carbon-Copy'

which is correct as I understand things.

Then, what I expect happens, is that when the reply is composed, and
the 2 byte UTF-8 character is read, it is instead interpreted as 2
characters, one of which is the A-Caret, and the other is, probably
not entirely by fluke, the opening « (which I just pasted from your
message, no idea in what form it will be sent out).

It turns out that for various reasons, I deal with e-mail that is not
ASCII a lot, and it would be good if I could work out how to set things
up so that it all worked better (reading the messages works fine,
replying is a mess ... and I suspect that this message will say it
is 8859-1 and would do, even if I included some totally non European
text in it, here is some, more cut&paste, not from an MH related
message, obviously, just to illustrate...

        ????? ????????

vi displays that line (normal C locale vi, in a non-font-understanding
xterm) as ...

Content-Type: text/plain; charset=utf-8; format=flowed

        \xe0\xb9\x80\xe0\xb8\xa3\xe0\xb8\xb5\xe0\xb8\xa2\xe0\xb8\x99
        \xe0\xb8\x84\xe0\xb8\x93\xe0\xb8\xb2\xe0\xb8\x88\xe0\xb8\xb2
        \xe0\xb8\xa3\xe0\xb8\xa2\xe0\xb9\x8c

(really all as one long line, I added the leading tabs, broke
the line at the only space in the original (first line break above)
(and deleted the space ... so if you reconstruct things, put a space
back in place of the first \n\t) and then simply at an arbitrary point
between 2 chars (second line break) (the second \n\t should simply be
deleted).

If I used exmh's "cite selection" I'd get the same chars as those I
pasted above (plus the attribution, and '>').

If I use repl (on the Thai message) I get the text included, and it
all looks correct UTF-8 encoded data, but if sedit attempts to interpret
that, it looks like it has lots of a-grave, Euro symbol, superscript 1 and 2
"cents" and English-pound ... probably from interpreting the bytes as
8859-1.

kre


--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers