[nmh-workers] fetchmail and SNI (and pop.gmail.com)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[nmh-workers] fetchmail and SNI (and pop.gmail.com)

Michael Richardson-5

I have used:

   fetchmail --verbose --sslcertpath="/etc/ssl/certs" --sslcertck --proto POP3 --mda "rcvstore -sequence gmail +inbox" --logfile /var/tmp/gmail.log pop.gmail.com

to get my gmail downloaded for some time now.
It seems that fetchmail doesn't enable SNI for it's TLS connection, and I
don't see any new versions of fetchmail in years.  It looks like
pop.gmail.com wants SNI:

fetchmail: Trying to connect to 2607:f8b0:4001:c16::6c/995...connected.
fetchmail: Server certificate:
fetchmail: Unknown Organization
fetchmail: Issuer CommonName: invalid2.invalid
fetchmail: Subject CommonName: invalid2.invalid
fetchmail: Server CommonName mismatch: invalid2.invalid != pop.gmail.com
fetchmail: pop.gmail.com key fingerprint: 90:4A:C8:D5:44:5A:D0:6A:8A:10:FF:CD:8B:11:BE:16
fetchmail: Server certificate verification error: self signed certificate
fetchmail: Missing trust anchor certificate: /OU=No SNI provided; please fix your client./CN=invalid2.invalid

[nice hack to send a message back to the user Google...]

I don't think that inc has any TLS support.
(kerberos support, yes)

Maybe there are other ways to skin this cat?

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [hidden email]  http://www.sandelman.ca/        |   ruby on rails    [


--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: fetchmail and SNI (and pop.gmail.com)

Ken Hornstein-2
>I don't think that inc has any TLS support.

You are incorrect!  Supported as of 1.7 when the unified security framework
was implemented.  From the NEWS file:

- Complete unification of network security support.  All network protocols
  (currently, POP and SMTP) have been refactored to use a common set of
  security routines.  This means all protocols support all SASL mechanisms
  (via the Cyrus-SASL library) and TLS.  TLS support has been strengthened
  to perform certificate name validation and to require TLS 1.1 as a
  minimum protocol.  Also, all protocols can make use of the OAuth2/XOAUTH
  SASL mechanism, which is supported by Gmail.

The last may be interesting to you.  I had not heard of SNI before, but
a quick test suggests to me that we work fine with pop.gmail.com (we don't
error out, at least).  The Interwebs suggest I should use a special
API call to make that work and I definitely didn't do that, but it seems
to be ok?

And geez Mike, we talked about this a lot!  Wasn't a secret!

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: fetchmail and SNI (and pop.gmail.com)

Ralph Corderoy
In reply to this post by Michael Richardson-5
Hi Michael,

> I have used:
>
>    fetchmail --verbose --sslcertpath="/etc/ssl/certs" --sslcertck
>        --proto POP3 --mda "rcvstore -sequence gmail +inbox"
>        --logfile /var/tmp/gmail.log pop.gmail.com
>
> to get my gmail downloaded for some time now.

Has your OpenSSL been upgraded recently?

> It seems that fetchmail doesn't enable SNI for it's TLS connection

Try adding `--sslproto TLS1' to fetchmail's arguments.

--
Cheers, Ralph.

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: fetchmail and SNI (and pop.gmail.com)

Ken Hornstein-2
>> It seems that fetchmail doesn't enable SNI for it's TLS connection
>
>Try adding `--sslproto TLS1' to fetchmail's arguments.

I guess the core issue is that for Google servers when using TLS 1.2 SNI
isn't required, but for TLS 1.3 it is; well, let me rephrase that.  If
you negotiate TLS 1.3 you get the bogus certificate if you don't send a
SNI.  But it seems like the 'right' solution is we should be sending a
SNI to avoid this problem?

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: fetchmail and SNI (and pop.gmail.com)

Michael Richardson-5
In reply to this post by Ken Hornstein-2

Ken Hornstein <[hidden email]> wrote:
    > And geez Mike, we talked about this a lot!  Wasn't a secret!

I read the man page. I wonder if my man pages are coming from debian, while
my binaries are manually installed.

SNI === Server Name Indicator, which lets a server know which name
a client meant to connect to, and therefore, which certificate to respond to.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [hidden email]  http://www.sandelman.ca/        |   ruby on rails    [



--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: fetchmail and SNI (and pop.gmail.com)

Michael Richardson-5
In reply to this post by Ralph Corderoy
Ralph Corderoy <[hidden email]> wrote:
    >> I have used:
    >>
    >> fetchmail --verbose --sslcertpath="/etc/ssl/certs" --sslcertck
    >> --proto POP3 --mda "rcvstore -sequence gmail +inbox"
    >> --logfile /var/tmp/gmail.log pop.gmail.com
    >>
    >> to get my gmail downloaded for some time now.

    > Has your OpenSSL been upgraded recently?

Yes-ish, I'm usually running something from git.

    >> It seems that fetchmail doesn't enable SNI for it's TLS connection

    > Try adding `--sslproto TLS1' to fetchmail's arguments.

That worked perfectly.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [hidden email]  http://www.sandelman.ca/        |   ruby on rails    [


--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: fetchmail and SNI (and pop.gmail.com)

Ken Hornstein-2
In reply to this post by Michael Richardson-5
>    > And geez Mike, we talked about this a lot!  Wasn't a secret!
>
>I read the man page. I wonder if my man pages are coming from debian, while
>my binaries are manually installed.

It looks like Debian buster is the earliest version of Debian which has
nmh 1.7.1.  And it looks like that will be officially released next week.
If you upgraded, would that be enough for you to switch away from
fetchmail? :-)  We support XOAUTH2!

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

[nmh-workers] success using the OAUTH2 with gmail.

Michael Richardson-5
In reply to this post by Michael Richardson-5

I've made sure I am running 1.7 (from git), and I went through mhlogin
correctly.  I pasted the result in, and then I did this with a second
account.

The token seems to be stored into Mail/oauth-gmail.

%inc -initialtls -host pop.gmail.com -user [hidden email] -port pop3s
   -snoop -sasl -saslmech xoauth2 -authservice gmail

-authservice is missing from the inc.1 man page.

I think that maybe we should have --gmail which sets all the right
parameters.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [hidden email]  http://www.sandelman.ca/        |   ruby on rails    [


--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: fetchmail and SNI (and pop.gmail.com)

Michael Richardson-5
In reply to this post by Ken Hornstein-2

Ken Hornstein <[hidden email]> wrote:
    > It looks like Debian buster is the earliest version of Debian which has
    > nmh 1.7.1.  And it looks like that will be officially released next week.
    > If you upgraded, would that be enough for you to switch away from
    > fetchmail? :-)  We support XOAUTH2!

I won't upgrade, I just installed from source.
I needed libcurl-dev.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [hidden email]  http://www.sandelman.ca/        |   ruby on rails    [


--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: success using the OAUTH2 with gmail.

Ken Hornstein-2
In reply to this post by Michael Richardson-5
>-authservice is missing from the inc.1 man page.

Um, is it?  I just looked at my copy, and it's there!  And according to
the revision history that was added in 2016, and nmh 1.7 was released
in 2017 (and I didn't update the NEWS file for 1.7.1; dammit).

>I think that maybe we should have --gmail which sets all the right
>parameters.

I believe that -authservice gmail does that already (it should use our
compiled-in default values for gmail, including our registered client
identifier).

If you mean --gmail does everything like -initialtls, -host, -port, -sasl
-saslmech xoauth2 and -authservice gmail, weeeellll .... actually, I have
some thoughts on that.

Let's say in a hypothetical future we support IMAP.  That means that nearly
every command would take a whole pile of arguments like -initialtls,
-host, -port, -sasl, and more.  Obviously changing your profile for every
nmh command would be awful.  So there should be some way of handling that.
What I had thought maybe was tying profile entries to mailboxes, so if
you did "scan my-imap-server:foo" it could possibly look in your profile
and find:

my-imap-server: -host my.server.com -port imap -tls -sasl -saslmech GSSAPI -user me

You get the idea.  But thinking about this more makes me think that we
should extend this a bit so it's not tied to folders, but a generic
connection profile defaults and we could provide ones that work with
Gmail.  I don't have it all jelled in my head how this would look and
you'd need to do something to ADD to an existing connection profile so
you could supply your own username, for example.  But it seems like it
should be doable.  But I guess my idea is that you should be able to
do something like

inc -conn gmail -user [hidden email]

and the right stuff should happen.  Make sense?

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: success using the OAUTH2 with gmail.

Michael Richardson-5

Ken Hornstein <[hidden email]> wrote:
    >> -authservice is missing from the inc.1 man page.

    > Um, is it?  I just looked at my copy, and it's there!  And according to
    > the revision history that was added in 2016, and nmh 1.7 was released
    > in 2017 (and I didn't update the NEWS file for 1.7.1; dammit).

Sorry, I should be more precise.
The values which are valid for "authservice" are not, as far as I can, mentioned.

    > I believe that -authservice gmail does that already (it should use our
    > compiled-in default values for gmail, including our registered client
    > identifier).

okay, I suggest an example in inc.

    > If you mean --gmail does everything like -initialtls, -host, -port, -sasl
    > -saslmech xoauth2 and -authservice gmail, weeeellll .... actually, I have
    > some thoughts on that.

    > Let's say in a hypothetical future we support IMAP.  That means that nearly
    > every command would take a whole pile of arguments like -initialtls,
    > -host, -port, -sasl, and more.  Obviously changing your profile for every
    > nmh command would be awful.  So there should be some way of handling that.
    > What I had thought maybe was tying profile entries to mailboxes, so if
    > you did "scan my-imap-server:foo" it could possibly look in your profile
    > and find:

Interesting idea.

    > You get the idea.  But thinking about this more makes me think that we
    > should extend this a bit so it's not tied to folders, but a generic
    > connection profile defaults and we could provide ones that work with
    > Gmail.  I don't have it all jelled in my head how this would look and
    > you'd need to do something to ADD to an existing connection profile so
    > you could supply your own username, for example.  But it seems like it
    > should be doable.  But I guess my idea is that you should be able to
    > do something like

    > inc -conn gmail -user [hidden email]

    > and the right stuff should happen.  Make sense?

Yeah, that's what I'd want to happen.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [hidden email]  http://www.sandelman.ca/        |   ruby on rails    [


--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: success using the OAUTH2 with gmail.

Ken Hornstein-2
>    > Um, is it?  I just looked at my copy, and it's there!  And according to
>    > the revision history that was added in 2016, and nmh 1.7 was released
>    > in 2017 (and I didn't update the NEWS file for 1.7.1; dammit).
>
>Sorry, I should be more precise.
>The values which are valid for "authservice" are not, as far as I can,
>mentioned.

Reading that with fresh eyes .... yeah, I see what you mean.  We kind
of skate on the details with regards to that, don't we?  It's meant to
be generic, in theory, but we don't really say anywhere "Hey, we have
included the right magic for gmail so you just need to say -authservice
gmail".  We do point to the mhlogin(1) man page, but we don't go into
great detail there either.

Thanks for this feedback!  It's always good to get someone new looking
at this, because since I was involved with the integration of this code
(not as much as David and Eric, though) I was familiar with the basic
ideas so I didn't see what was missing.

Back in:

  https://lists.gnu.org/archive/html/nmh-workers/2019-05/msg00000.html

One of my bullet points for a possible 1.8 release was:

- Write a man page explaining about the network security for inc/post and
  friends and how it works in nmh, since it's kind of confusing.

And this sounds like a perfect candidate for this.

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: fetchmail and SNI (and pop.gmail.com)

Ralph Corderoy
In reply to this post by Ken Hornstein-2
Hi Ken,

> I guess the core issue is that for Google servers when using TLS 1.2
> SNI isn't required, but for TLS 1.3 it is; well, let me rephrase that.
> If you negotiate TLS 1.3 you get the bogus certificate if you don't
> send a SNI.  But it seems like the 'right' solution is we should be
> sending a SNI to avoid this problem?

I agree nmh should employ SNI; I was just getting Michael up and running
the simplest way possible.

--
Cheers, Ralph.

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: success using the OAUTH2 with gmail.

David Levine-3
In reply to this post by Ken Hornstein-2
Ken writes:

> It's meant to
> be generic, in theory, but we don't really say anywhere "Hey, we have
> included the right magic for gmail so you just need to say -authservice
> gmail".  We do point to the mhlogin(1) man page, but we don't go into
> great detail there either.

inc(1) man page:

    which must be specified with the -authservice service switch.
    Before using this, the user must authorize nmh by running
    mhlogin and granting authorization to that account.  See
    mhlogin(1) for more details.

mhlogin(1) man page:

    mhlogin currently only supports OAuth for Gmail.  Run mhlogin
    -user username -saslmech xoauth2 -authservice gmail

    ...

    even though only Gmail is supported out of the box

Updates welcome.

David

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: success using the OAUTH2 with gmail.

Ralph Corderoy
In reply to this post by Ken Hornstein-2
Hi Ken,

> Let's say in a hypothetical future we support IMAP.  That means that
> nearly every command would take a whole pile of arguments like
> -initialtls, -host, -port, -sasl, and more.  Obviously changing your
> profile for every nmh command would be awful.  So there should be some
> way of handling that.  What I had thought maybe was tying profile
> entries to mailboxes, so if you did "scan my-imap-server:foo" it could
> possibly look in your profile and find:
>
>     my-imap-server: -host my.server.com -port imap -tls -sasl
>         -saslmech GSSAPI -user me

That seems very specific and introduces a new colon operator that
restricts what's available for other features later.

How about allowing an mh-profile(5) in a folder's directory with its
content having higher priority than the ancestor folders' .mh_profile
and the general ~/.mh_profile.  This could be used for more general
things, e.g. the template used for replies to emails in that folder, or
the preferred format for scanning it.

In the IMAP case, the folder exists locally, and its .mh_profile, but
the emails are remote, as are sub-folders.

> You get the idea.  But thinking about this more makes me think that we
> should extend this a bit so it's not tied to folders, but a generic
> connection profile defaults and we could provide ones that work with
> Gmail.  I don't have it all jelled in my head how this would look and
> you'd need to do something to ADD to an existing connection profile so
> you could supply your own username, for example.  But it seems like it
> should be doable.  But I guess my idea is that you should be able to
> do something like
>
>     inc -conn gmail -user [hidden email]
>
> and the right stuff should happen.  Make sense?

If `foo: -bar xyzzy...' is in an .mh_profile then often, depending on
the value's complexity, it can be interpolated with «`mhparam foo`» in
the shell.  What if a similar capability existed on the value side of an
.mh_profile's `key: value'.  Except it could cope with the interpolated
value having quotes.

This would allow collections of options to be defined and then
referenced with a shorthand.  Either back quotes copied from sh(1), or
`-use foo' so it can work easily at the shell too.

--
Cheers, Ralph.

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: success using the OAUTH2 with gmail.

Ken Hornstein-2
>> Let's say in a hypothetical future we support IMAP.  That means that
>> nearly every command would take a whole pile of arguments like
>> -initialtls, -host, -port, -sasl, and more.  Obviously changing your
>> profile for every nmh command would be awful.  So there should be some
>> way of handling that.  What I had thought maybe was tying profile
>> entries to mailboxes, so if you did "scan my-imap-server:foo" it could
>> possibly look in your profile and find:
>>
>>     my-imap-server: -host my.server.com -port imap -tls -sasl
>>         -saslmech GSSAPI -user me
>
>That seems very specific and introduces a new colon operator that
>restricts what's available for other features later.

Well, it was just a strawman .... like I said, I'm not really sure about
it yet.  It seems a bit clumsy that you have to add that folder-specific
config file to bind an IMAP folder, but maybe extending the folder(1)
command would work?  More thought is needed.

>How about allowing an mh-profile(5) in a folder's directory with its
>content having higher priority than the ancestor folders' .mh_profile
>and the general ~/.mh_profile.  This could be used for more general
>things, e.g. the template used for replies to emails in that folder, or
>the preferred format for scanning it.

In that vein, when playing around with DavMail I found that it cached
a lot of message headers with IMAP but not the message bodies, so a
theoretical scan template which fetched the first part of the body
of the message has lousy performance, so that's a case where you would
want a folder-specific scan template.  Something else to think about.

>> You get the idea.  But thinking about this more makes me think that we
>> should extend this a bit so it's not tied to folders, but a generic
>> connection profile defaults and we could provide ones that work with
>> Gmail.  I don't have it all jelled in my head how this would look and
>> you'd need to do something to ADD to an existing connection profile so
>> you could supply your own username, for example.  But it seems like it
>> should be doable.  But I guess my idea is that you should be able to
>> do something like
>>
>>     inc -conn gmail -user [hidden email]
>>
>> and the right stuff should happen.  Make sense?
>
>If `foo: -bar xyzzy...' is in an .mh_profile then often, depending on
>the value's complexity, it can be interpolated with «`mhparam foo`» in
>the shell.  What if a similar capability existed on the value side of an
>.mh_profile's `key: value'.  Except it could cope with the interpolated
>value having quotes.
>
>This would allow collections of options to be defined and then
>referenced with a shorthand.  Either back quotes copied from sh(1), or
>`-use foo' so it can work easily at the shell too.

Well ... I see where you are going.  But ...

It occurs to me that one problem with our configuration, especially for
newer users, is that it's all spread out between different tools.  For
example, your SMTP servers settings are in send(1) (and if you want
to be complete, probably whom(1) as well), POP server settings belong
in inc(1) and msgchk(1), and with a theoretical IMAP server it would
be a lot of tools.  This is confusing.

When users are given information about the settings for an email provider,
they are usually given them all at once.  "Here's the SMTP server, here's
the POP server, here's the IMAP server", etc etc.  So if we had some
facility where we could say "Here's where you put the block of network
settings for GMail", for example, and then we could have people just
refer to the "gmail" settings, that would be a huge win.  Making those
things easily be extractable via mhparam would also be a huge win.
Again, I'm not sure what all of this would look like right now.
This is just idle musings.

--Ken

--
nmh-workers
https://lists.nongnu.org/mailman/listinfo/nmh-workers