Future directions for nmh

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
55 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Ken Hornstein-2
>But this makes it *your* turn to come up with an insane idea.
>
>Waiting ...

Well ... okay!  So, honest critique time: you know more about IMAP than
I do.  If you see problems with this, please let me know what they are.

I'm speaking in terms of existing nmh interfaces; presumably they would
be all different if the internal API is redone.

For each IMAP folder, we have a cache file.  It might look like this:

#
# This is an nmh folder cache file for folder IMAP:gmail/poop
#
EXISTS 35
UIDVALIDITY 7721
NEXTUID 998765
MSG 1 40861
MSG 2 66934
MSG 3 22851
MSG 4 77993

... and so on.

When an nmh program calls folder_read() (the function to scan a folder and
build the internal folder structure of each message), an IMAP SELECT on the
specified folder is done.  So you'd issue a SELECT gmail/poop (I don't
know if that's a valid folder name).

If your EXISTS, UIDVALIDITY, and NEXTUID values all matched what was stored
in the cache file, you're good!  If you want to look at message 3, you'd
issue a UID FETCH 22851 with what you wanted.

If your UIDVALIDITY did NOT match, then you'd just regenerate the whole
cache file with new message numbers, starting from 1 (since you'd have no
idea which message corresponded to which UID).  Maybe emit a warning in that
case.

If your EXISTS or NEXTUID did not match was you stored, you'd do something
like (I think I have this right, assume there are now 37 messages):

        FETCH 1:37 (UID)

You'd get that list of UIDs and compare them to your cache.  If a UID was
missing from your cache, you'd have a hole in your nmh message numbers;
UIDs that were new would be added right after the highest-numbered nmh
message number.  So let's say you had 3 messages added in the above example
and the message with UID 66934 deleted; nmh message '2' would now be missing,
and there would be new messages 36, 37, and 38.  So, persistent message
numbers on the nmh side which AFAICT is 90% of the problem.  You'd perform
all IMAP operations based on UIDs rather than IMAP message sequence numbers.
Commands like 'folder -pack' and 'sortm' just change the local cache and
would not change the IMAP store.

So ... thoughts?

--Ken

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Conrad Hughes
In reply to this post by Ken Hornstein-2
Ken> it makes it WORSE; each nmh command starts with a brand-new scan of a
Ken> folder, so messages added or removed between commands work out fine.
Ken> But a FUSE interface would have no idea when an nmh command is starting
Ken> or stopping, so you'd have to do a lot of caching or a new IMAP session
Ken> for each file access.

Arguably this is a problem already: all the time I have to keep in my
head that I must be careful about changing nmh state among my (usual)
four terminal windows: I'm reading one email in one folder, see
something arrive in another folder, look at that in another window, go
back to the first window, decide the original message is irrelevant and
rmm it — except now I'm rmm'ing something else in another folder because
I switched context in the other window.

If we solve that, do we solve your IMAP/FUSE issue too?

Conrad

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Ken Hornstein-2
>Arguably this is a problem already: all the time I have to keep in my
>head that I must be careful about changing nmh state among my (usual)
>four terminal windows: I'm reading one email in one folder, see
>something arrive in another folder, look at that in another window, go
>back to the first window, decide the original message is irrelevant and
>rmm it — except now I'm rmm'ing something else in another folder because
>I switched context in the other window.

You are allowed to have multiple contexts (see mh-profile(5), the MHCONTEXT
environment variable).  You could in your shell startup script do something
like:

export MHCONTEXT context-$$

Might want to do some cleanup of those files, but you get the idea.
That would give each shell it's own context, so a folder change in one
window wouldn't affect other windows.  You could do private sequences if
you want to have per-window sequences for things like "cur", but you'd
need to think carefully about the implications there.

That might solve the specific problem of being in the wrong folder when you
issue an rmm.  But remember ... with individual commands, I am unclear
how you can fundamentally solve this problem without requiring the user
to know what they are doing.  My question to you would be: in the above
scenario, what do you EXPECT nmh to do?

--Ken

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Paul Vixie-2
In reply to this post by Lyndon Nerenberg (VE6BBM/VE7TFX)


Lyndon Nerenberg wrote:

>> On Mar 16, 2016, at 4:56 PM, Ken Hornstein<[hidden email]>  wrote:
>>
>> And I believe
>> it makes it WORSE; each nmh command starts with a brand-new scan of a
>> folder, so messages added or removed between commands work out fine.
>> But a FUSE interface would have no idea when an nmh command is starting
>> or stopping, so you'd have to do a lot of caching or a new IMAP session
>> for each file access.
>
> That pretty much nuts down the issue. If you want to play IMAP in
> MH,you must become a full-on IMAP client, playing all those locking games.

probably the right thing here is to do what the prayer webmail system does.

http://www-uxsup.csx.cam.ac.uk/~dpc22/prayer/

Persistent Login Sessions:

> Uses persistent connections to IMAP server and support servers.
> Target folders remain SELECTed: not a simple-minded proxy.
> Full caching (including sort/thread cache) for each open folder.
> Up to five persistent IMAP connections (typically one or two in use):
> INBOX and one other folder
> Postponed message folder stream
> Preferences stream
> Folder transfer stream
> Various optimisations/sharing to minimise actual IMAP connections
> Directory cache: single round trip to IMAP server for directory listing.
> Works well with UW IMAP server (even using Unix format mail folders).
 > ...
 > Written entirely in C as HTTP to IMAP Gateway. No scripting languages.

prayer is the simplest and faster webmail system i ever found for my
family's use while traveling.

i suggest that some of the prayer webmail source code could be bundled
into an MH-IMAP superproxy, or that prayer could be refactored to
support a restful API that MH could access on localhost.

--
P Vixie

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Lyndon Nerenberg (VE6BBM/VE7TFX)

> On Mar 16, 2016, at 6:04 PM, Paul Vixie <[hidden email]> wrote:
>
> prayer is the simplest and faster webmail system i ever found for my family's use while traveling.

But. How. Does webmail have anything to do with this?  
_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Paul Vixie-2
In reply to this post by Conrad Hughes


Conrad Hughes wrote:

> Ken>  it makes it WORSE; each nmh command starts with a brand-new scan of a
> Ken>  folder, so messages added or removed between commands work out fine.
> Ken>  But a FUSE interface would have no idea when an nmh command is starting
> Ken>  or stopping, so you'd have to do a lot of caching or a new IMAP session
> Ken>  for each file access.
>
> Arguably this is a problem already: all the time I have to keep in my
> head that I must be careful about changing nmh state among my (usual)
> four terminal windows: I'm reading one email in one folder, see
> something arrive in another folder, look at that in another window, go
> back to the first window, decide the original message is irrelevant and
> rmm it — except now I'm rmm'ing something else in another folder because
> I switched context in the other window.

i make liberal use of -nochangecur in all my non-interactive use of MH,
so that my interactive use won't produce the result you're describing.

--
P Vixie

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Paul Vixie-2
In reply to this post by Lyndon Nerenberg (VE6BBM/VE7TFX)


Lyndon Nerenberg wrote:
>> On Mar 16, 2016, at 6:04 PM, Paul Vixie<[hidden email]>  wrote:
>>
>> prayer is the simplest and faster webmail system i ever found for my family's use while traveling.
>
> But. How. Does webmail have anything to do with this?

webmail and mh both have a fork+exec before every operation, thus the
need to keep an IMAP session open persistenly across many fork+exec's.

if MH could proxy its IMAP operations through a daemon running on a unix
domain socket or the loopback interface, then we could idealize the RPC
API between MH and that daemon. one thought is restful json.

i mentioned prayer webmail because it's an extremely high quality C
implementation of this idea, and has an MH-compatible license.

i bring up this proxying idea because i'd be the first to write a proxy
that spoke Maildir instead of IMAP, and ken would be the first to write
a proxy that spoke MH-Dir, at which point the whole existing MH code
base would be simpler and more modular than it is now.

--
P Vixie

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Ken Hornstein-2
>if MH could proxy its IMAP operations through a daemon running on a unix
>domain socket or the loopback interface, then we could idealize the RPC
>API between MH and that daemon. one thought is restful json.

I'm fine with that idea as well (although I see some issues in terms of
authentication, but nothing unsolveable).  I'd be happy if that was
a version "1.1" or "2.0" version of some hypothetical IMAP nmh code.

I had not heard of prayer webmail, but I will check it out!  But it
looks like it forks off all of the hard work on the c-client library?
And it runs a daemon out of xinetd.

--Ken

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Conrad Hughes
In reply to this post by Ken Hornstein-2
Ken> export MHCONTEXT context-$$

.. I live and learn, thanks.

Ken> in the above scenario, what do you EXPECT nmh to do?

Well, from experience, I expect it to do what I tell it, even if that's
not what I intend :-/

My preference would be for actions (rmm, refile, repl) to note there's
been a context change and ask for confirmation, I think.  The machine is
better than I am at tracking consistency.  If context-in-this-window and
most-recent-context are different (or more particularly, the action
target (cur, most likely) differs between the two contexts), then there
is a significant chance I'm trying to do something other than what I
appear to be doing.

[Paul's use of -nochangecur noted with thanks; also noticed its use in a
 script recently posted to the list]

Another scenario (which I've gotten under control by iron discipline) is
that I sync my home directory among a variety of machines, several of
which can receive email.  This more-or-less corresponds to the IMAP
situation I think: a folder can change under your feet because
synchronisation updates the folder with state change from another
machine.

Taken to its ultimate conclusion, that means that context actually
should record both the full sequence of message-IDs (say) in all folders
*and* all mh_sequences.  As has been mentioned, that could impact
performance.

Wouldn't a similar consistency check catch IMAP changes too (contingent
on the suggested UUID <-> local number mapping)?

Conrad

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Ken Hornstein-2
>My preference would be for actions (rmm, refile, repl) to note there's
>been a context change and ask for confirmation, I think.  The machine is
>better than I am at tracking consistency.  If context-in-this-window and
>most-recent-context are different (or more particularly, the action
>target (cur, most likely) differs between the two contexts), then there
>is a significant chance I'm trying to do something other than what I
>appear to be doing.

Okay, dumb question time.  You run command (a) which changes the context.
How is command (b) supposed to know that the context has been changed?
Think carefully about your answer.

>Taken to its ultimate conclusion, that means that context actually
>should record both the full sequence of message-IDs (say) in all folders
>*and* all mh_sequences.  As has been mentioned, that could impact
>performance.

The problem is that information is, in the MH model, redundant; the "unique
identifier" is the message number, full stop.  If you're changing the
message store behind nmh, you're really supposed to Know What You're Doing.
But the unique identifer can easily be changed with things like
folder -pack and sortm.  My point being is that if there are changes to
the store but they all conform to unique message numbers, all of the nmh
utilities should work fine.

>Wouldn't a similar consistency check catch IMAP changes too (contingent
>on the suggested UUID <-> local number mapping)?

I don't see the same issues applying.

--Ken

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Paul Fox-3
In reply to this post by Conrad Hughes
conrad wrote:
 > Ken> export MHCONTEXT context-$$
 >
 > .. I live and learn, thanks.
 >
 > Ken> in the above scenario, what do you EXPECT nmh to do?
 >
 > Well, from experience, I expect it to do what I tell it, even if that's
 > not what I intend :-/
 >
 > My preference would be for actions (rmm, refile, repl) to note there's
 > been a context change and ask for confirmation, I think.  The machine is

i'd never considered changing MHCONTEXT either.  but i think i'd
prefer the context be shared more often than i'd want it separate.

as usual, i have a half-assed solution to this problem which works
works surprisingly well in practice:  all of my mh usage goes through
a multi-invocation wrapper script, mostly because i'd rather type one
letter commands.  most invocations of my wrapper commands record their
parent's pid in a well-known file.  the wrapper for rmm, when not
given a specific argument (i.e., "d", rather than "d 32") checks to
make sure its parent's pid is the same as what's recorded in the file --
if not, it warns me.  this catches changes to cur and to the current
folder, and also simply reminds me that i've been running mh commands
elsewhere and that i should think a bit.

paul

excerpt:

    me=${0##*/}
    PPIDFILE=${MAILDIR:-$Mail}/.lastmhppid
    ...
    case $me in
        d)
            if [ ! "$1" -a $PPID != "$(cat $PPIDFILE)" ]
            then
                (
                echo Warning: last mh command started from different shell.
                echo Will delete:
                scan cur | map_binary
                echo -n Current message may be WRONG!  Hit y if okay...
                ) >&2
                read ans
                case $ans in
                    [yY]*)
                        ;;
                    *)
                        exit
                        ;;
                esac
            fi
            ;;

        *)
            echo $PPID >$PPIDFILE
            ;;
    esac


=----------------------
 paul fox, [hidden email] (arlington, ma, where it's 39.9 degrees)

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Conrad Hughes
In reply to this post by Ken Hornstein-2
Ken> You run command (a) which changes the context.  How is command (b)
Ken> supposed to know that the context has been changed?

Given your mentioning of MHCONTEXT, I could envisage a wrapper for MH
commands much like Paul Fox's example — perhaps a context directory
containing per-shell-PID context files, and the crucial test would be
whether another context file was more recent than and different to this
shell's one when you take an action.

There's a problem there though, in that context only records your
current folder, with sequences being folder-local — so a sortm in one
shell could render a previous scan in another shell obsolete.  Trying to
solve that by merging all sequences up into context would introduce
other difficulties: for example sortm would have to rewrite *all*
context+sequence records containing sequences affected by the sort.  The
difficulty of tracking that probably renders Paul's approach ("Something
happened somewhere else — you're about to affect message(s) X — are you
sure?") safer.

I don't use other message stores, but it seems to me that maintaining a
map of MH unique IDs to another store could be quite closely related,
and perhaps Paul's approach would still apply: it's simple and safe,
although it could be inconvenient if "third party" changes couldn't be
tracked at a sufficiently fine granularity.

Conrad

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Ralph Corderoy
In reply to this post by Paul Fox-3
Hi Paul,

> most invocations of my wrapper commands record their parent's pid in a
> well-known file.  the wrapper for rmm, when not given a specific
> argument (i.e., "d", rather than "d 32") checks to make sure its
> parent's pid is the same as what's recorded in the file

Nice idea.

Cheers, Ralph.

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Ken Hornstein-2
In reply to this post by Conrad Hughes
>Given your mentioning of MHCONTEXT, I could envisage a wrapper for MH
>commands much like Paul Fox's example — perhaps a context directory
>containing per-shell-PID context files, and the crucial test would be
>whether another context file was more recent than and different to this
>shell's one when you take an action.

... that seems like a complicated mess prone to race conditions if you
don't do some locking.

>There's a problem there though, in that context only records your
>current folder, with sequences being folder-local

That is not true; you can have 'private' sequences which are stored in
the context, so in theory you could have different sequences for each
shell.  That might not be so great for things like the unseen sequence,
though (but you can have some sequences public and others private).

>— so a sortm in one
>shell could render a previous scan in another shell obsolete.  Trying to
>solve that by merging all sequences up into context would introduce
>other difficulties: for example sortm would have to rewrite *all*
>context+sequence records containing sequences affected by the sort.  The
>difficulty of tracking that probably renders Paul's approach ("Something
>happened somewhere else — you're about to affect message(s) X — are you
>sure?") safer.

I'm fine with Paul's approach, but I think it should remain as an add-on;
I don't think it makes sense to integrate it into nmh.

>I don't use other message stores, but it seems to me that maintaining a
>map of MH unique IDs to another store could be quite closely related,
>and perhaps Paul's approach would still apply: it's simple and safe,
>although it could be inconvenient if "third party" changes couldn't be
>tracked at a sufficiently fine granularity.

I am not sure it's appropriate.  If you look at the email I posted about
how I envision nmh would work with IMAP, it should be obvious from it
that the scheme I proposed would (hopefully) be relatively resistant to
problems if the backend store was changed.  Replace "UID" with "filename"
in that message, and you could probably use the same scheme with Maildir.

--Ken

_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
Reply | Threaded
Open this post in threaded view
|

Re: Future directions for nmh

Chad Brown-6
In reply to this post by Paul Vixie-2
The context issue (and a related client issue) is actually why I stopped using mh a while back. I switched professions and needed the email client on my phone to interact reasonably with the client on my laptop (or server typically accessed via laptop). For a while, I ssh’d into a server and ran nmh commands from my phone, but that proved to be reasonably painful and didn’t look like it was going to improve anytime soon (frequent if simple MIME content didn’t help).

I looked at a couple webmail systems (Squirrel is the name I recall, but there were a couple, including at least one home-brew), but the overhead of syncing status between the systems proved to be irritating enough that I just gave in to imap everywhere.

This makes me happy to hear about a potential future mh (style?) interface to imap servers. I had briefly considered something like what Paul Vixie is (I believe) suggesting could be adapted from prayer, but at the time I thought the overhead of a personal mh daemon seemed like a dealbreaker. I had that initial reaction reading Paul’s mail, but since it’s relatively common for my laptop to run a few hundred processes while doing “nothing”, maybe that’s outdated thinking.

Sorry for the ramble; mostly what I wanted to say was “interesting idea” and “I’m glad I’m still following nmh, even if I haven’t really used it in a while”.

Thanks,
~Chad


_______________________________________________
Nmh-workers mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/nmh-workers
123