So, since my simple question about a new release spawned a whole thread
about the future direction of nmh I wanted to create a distinct thread to discuess those ideas. If you're interested in commenting on future ideas for nmh, this is the place to do it. I am first going on the assumption that completely redoing the email parser and making all of the nmh tools MIME-aware is a noncontroversial subject. If you disagree, please speak up. Also, if you are unclear what exactly I mean by that, also speak up. To address other things brought up recently: 1) I do not think converting and storing incoming messages as UTF-8 is wise. In terms of just simplicity alone I think messages should be stored (somewhere) in their on-the-wire format; this is more robust and would prevent loss of mail if there is an issue with character conversion at mail incorporation time. I'm talking about the default here; if you want to convert them to some other form using mhfixmsg or some yet-written tool, that's your business. 2) Assuming you agree with 1) (I know Lyndon does not :-) ), then I think converting stuff internally to UTF-8 before output would not be a net gain; it might make some things easier, but I think in general it would make things more complex. The system we have where character conversion happens somewhere after parsing and before display, while a bit haphazard, seems to be fine in practice, and it's close to what other open source MUAs do when I looked at them. If we were on Plan 9 then this would be a different decision, but we've heard from enough users that do not use UTF-8 locales that makes me think this is a serious concern. 3) Like I said, I am officially neutral on creating a grep(1)able mail store; I think it's an exercise in futility, but enough people want it that it's clearly not something that is worth dismissing. As Jerrad pointed out, you can probably accomplish most of this today with his MIME-Hooks. Do we include that? If we do not, we should put it in contrib if there isn't a good home for it. Would that be sufficient for people that wanted it? I just don't feel great about having the "exploded" message being the canonical format. Maybe Paul Vixie is right about me holding onto this idea that nmh should keep the original mail store; maybe that's driven by me still using exmh. I'd make the point that at some point we have to process a RFC5322-format message for every piece of email, so that code still needs to be written regardless of doing anything else. 4) In terms of alternate mail stores, be they Maildir or IMAP (I think those are the two major candidates now, right?), I think those ideas are interesting. The #1 problem with those ideas is how to map MH message numbers (which can range 1-MAXINT, with holes) to the internal key used by those mail stores; everything else is relatively easy to deal with. For IMAP ... well, it occurs to me that locally you have a database that would map message numbers to IMAP UIDs. This local database could also store annotations, sequences, and maybe something else I'm forgetting. If you found a message that wasn't in your database, you'd have a new message and add it to your local database; if a message was missing, you'd have a hole in the message number sequence. sort(1)/pack(1) would really be about shuffling around this list of message numbers. This would have the disadvantage that if you accessed your IMAP store from another system you'd have new message numbers, sequences, and annotations, but it would be tons better than what we have now (which is that it doesn't work at all). I have some thoughts on how to create a shareable database for that stuff in IMAP, but I'd need to include a trigger warning for Lyndon so he could start drinking heavily before he read it :-) For Maildir, a similar idea, except that you could do annontations directly in the message. Really, none of that seems hard to me. Maybe some details to work out, but I don't see any major challenges. I'd be interested to hear people tell me if I'm wrong, or if they have suggestions or better ideas. --Ken _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
Ken Hornstein <[hidden email]> writes:
> >4) In terms of alternate mail stores, be they Maildir or IMAP (I think >those are the two major candidates now, right?), I think those ideas >are interesting. The #1 problem with those ideas is how to map MH >message numbers (which can range 1-MAXINT, with holes) to the internal >key used by those mail stores; everything else is relatively easy >to deal with. I can't quite tell if by "alternate", you meant optional. I certainly hope so. _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
>>4) In terms of alternate mail stores, be they Maildir or IMAP (I think
>>those are the two major candidates now, right?), I think those ideas >>are interesting. The #1 problem with those ideas is how to map MH >>message numbers (which can range 1-MAXINT, with holes) to the internal >>key used by those mail stores; everything else is relatively easy >>to deal with. > >I can't quite tell if by "alternate", you meant optional. I certainly hope >so. I am not proposing that we replace the current MH mailstore with Maildir, if that's what you mean. You should still be able to use the traditional MH mailstore (and we'll keep that the default) for the forseeable future. I am saying that we have people who want to use the nmh tools with both IMAP and Maildir mailstores. So making the nmh tools work with those mailstores would be useful. --Ken _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
Ken> I am saying that we have people who want to use the nmh tools with both
Ken> IMAP and Maildir mailstores. So making the nmh tools work with those Ken> mailstores would be useful. .. with migration via refile between different store types .. that sounds cool.. C. _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Ken Hornstein-2
Thus said Ken Hornstein on Wed, 09 Mar 2016 10:18:04 -0500:
> 1) I do not think converting and storing incoming messages as UTF-8 is > wise. In terms of just simplicity alone I think messages should be > stored (somewhere) in their on-the-wire format; I too agree with this position. The email that nmh writes into my Mail folder should not differ from the message that my MTA queued on disk. > 4) In terms of alternate mail stores, be they Maildir or IMAP (I think > those are the two major candidates now, right?), I think those > ideas are interesting. The #1 problem with those ideas is how to > map MH message numbers (which can range 1-MAXINT, with holes) to > the internal key used by those mail stores; everything else is > relatively easy to deal with. What's wrong with the way nmh does it today? Maildir is certainly superior for some of the reasons detailed below, but nmh's Mail store isn't very far off: http://cr.yp.to/proto/maildir.html Also, I'm confused why Maildir and IMAP are both alternate mail stores. Email is stored on server in a Maildir and accessed via IMAP. I don't see them as mutually exclusive. Thanks, Andy -- TAI64 timestamp: 4000000056e19411 _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Ken Hornstein-2
Thus said Ken Hornstein on Wed, 09 Mar 2016 10:44:55 -0500:
> I am saying that we have people who want to use the nmh tools with > both IMAP and Maildir mailstores. So making the nmh tools work with > those mailstores would be useful. Oh, I misunderstood. So part of my last email is no longer relevant. Andy -- TAI64 timestamp: 4000000056e19446 _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Andy Bradford-2
Andy Bradford wrote: > What's wrong with the way nmh does it today? Maildir is certainly > superior for some of the reasons detailed below, but nmh's Mail store > isn't very far off: > > http://cr.yp.to/proto/maildir.html some of us have converted to Maildir, but we miss MH's CLI capability. > Also, I'm confused why Maildir and IMAP are both alternate mail stores. > Email is stored on server in a Maildir and accessed via IMAP. I don't > see them as mutually exclusive. in IMAP, Maildir is an implementation detail. some IMAP servers use SQL, BDB, Berkely Mail, or even MMDF format for mail stores. i happen to be using a Maildir store on Dovecot2, but there are other options. -- P Vixie _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Ken Hornstein-2
Howdy. I want to start by thanking those of you who actually have
time to work on this. Wish that I had some time to spare. Here's my grumpy-don't-break-things opinion. I support redoing the email parser and making everything MIME-aware. As you might guess if you've been on this mailing list forever, I'd really like to see a better user interface for reading attachments. In short, I'd like to keep track of the "current part", and have "next part" and "previous part" options to show. There should be a flag that allows switching to the next/prev message if next/prev part runs out of parts. To me, this would be so much better than having to do a mhlist and then type part numbers. I looked at doing this years ago and got stopped by dealing with the parsing stuff. There should also be an option to scan that lists the parts in messages. So if the parser is redone maybe it will be done in such a way to support this. I think that we should keep the mail store as is. I agree that the MIME and character set stuff has made it less grep-able than it used to be, but it's still useful. I am emboldened by David's posting regarding add-hook, etc. Didn't know that anybody actually used that stuff other than me. The hook stuff was an expedient hack on my part, and it's not really as robust as it could be. I would like to see it made more of an integral part of nmh as opposed to the add-on hack that I did. The reason that having a solid hooks implementation is important is that it allows users to have whatever alternate mail stores suit them without having to make them part of nmh. The way that I manage my mail here is to have hooks that mirror my mail folders in ElasticSearch. I have some additional commands (gpick/gscan) that work just like pick/scan but are very fast by comparison. I wrote a mail parser years ago that makes this work. It's pretty out of date now. It has a feature that would be nice to support, which is that it allows (in a manner similar to stuff in mh_profile) external helper programs to be specified so that text can be extracted from non-ASCII attachments. For example, it can pipe a PDF attachment through pdftotext. It also has the ability to control what header fields get indexed. So I guess my opinion is that improving the parser is a great thing, and improving the way that external stuff can work with nmh is preferable to changing the way that nmh works as far as mail directories, etc. And, of course, I'd like a better ui for part handling. Jon _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
Jon Steinhart wrote: > ... > I think that we should keep the mail store as is. I agree that the MIME > and character set stuff has made it less grep-able than it used to be, > but it's still useful. useful for what? i have no remaining use cases of my own. > I am emboldened by David's posting regarding add-hook, etc. Didn't know > that anybody actually used that stuff other than me. The hook stuff was > an expedient hack on my part, and it's not really as robust as it could > be. I would like to see it made more of an integral part of nmh as > opposed to the add-on hack that I did. i love your hooks. used them in mhdb: http://family.redbarn.org/~vixie/mhdb.shar > The reason that having a solid hooks implementation is important is that > it allows users to have whatever alternate mail stores suit them without > having to make them part of nmh. my situation is the opposite. i'd like multiple mail store types inside NMH, or a pluggable hooks-style interface allowing the creation of same, so that i can use NMH command line tools to attack my Maildir store as created for me by Dovecot. a mail store that i can only reach by the command line isn't maximally useful. a mail store that i can't reach by the command line isn't maximally useful, either. -- P Vixie _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
Paul Vixie writes:
> Jon Steinhart wrote: > > ... > > I think that we should keep the mail store as is. I agree that the MIME > > and character set stuff has made it less grep-able than it used to be, > > but it's still useful. > > useful for what? i have no remaining use cases of my own. All of the email about which I care has ASCII subject lines. Also, I often grep for a particular attachment name. > > I am emboldened by David's posting regarding add-hook, etc. Didn't know > > that anybody actually used that stuff other than me. The hook stuff was > > an expedient hack on my part, and it's not really as robust as it could > > be. I would like to see it made more of an integral part of nmh as > > opposed to the add-on hack that I did. > > i love your hooks. used them in mhdb: > > http://family.redbarn.org/~vixie/mhdb.shar Glad to hear it. Never got any feedback on 'em so it's cool to hear that people are using them. > > The reason that having a solid hooks implementation is important is that > > it allows users to have whatever alternate mail stores suit them without > > having to make them part of nmh. > > my situation is the opposite. i'd like multiple mail store types inside > NMH, or a pluggable hooks-style interface allowing the creation of same, > so that i can use NMH command line tools to attack my Maildir store as > created for me by Dovecot. > > a mail store that i can only reach by the command line isn't maximally > useful. > > a mail store that i can't reach by the command line isn't maximally > useful, either. I'm a bit confused here. I use nmh because I can reach the command store via the command line. What other way is their to use it? And other programs can operate on the mail store, which is part of what makes it great. Are you saying that you'd like to see the nmh cli abstracted to the point where it could work on different flavors of mail store? If so, that seems like a big change. Sort of like the hooks, but with a DLL interface instead of the shell? Jon _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
Jon Steinhart wrote: > ... > Are you saying that you'd like to see the nmh cli abstracted to the point > where it could work on different flavors of mail store? If so, that seems > like a big change. Sort of like the hooks, but with a DLL interface > instead of the shell? yes. -- P Vixie _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Jon Steinhart
>All of the email about which I care has ASCII subject lines.
>Also, I often grep for a particular attachment name. Right or wrong, I get subject lines that are encoded using RFC 2047 rules. I know you think of those as 'foreign languages', but that is wrong; they are just 'characters'. Also, if you talk to anyone younger than 25 you're going to be getting a fair number of emoji. It's fair to say that you don't care about that stuff, but I think as younger people get older they are going to consider things like emoji as 'normal', and it's something that we have to deal with. Again, it's fine if you want to just send and receive email between old farts, but I would kindly suggest that's not appropriate for everyone. >Glad to hear it. Never got any feedback on 'em so it's cool to hear that >people are using them. We had a big discussion about them based on Jarrad's work with his MIME hooks (we had never really laid down how locking was supposed to work with them). >I'm a bit confused here. I use nmh because I can reach the command store >via the command line. What other way is their to use it? And other >programs can operate on the mail store, which is part of what makes it >great. I believe Paul is saying that he wants to have the command-line power of nmh work on mail stores he shares with other MUAs. To me, that is an admirable goal. It is KIND of true that a subset of MUAs can sort-of operate on a MH mail store, but that's only true if basically you disregard any sort of support for sequences and locking. And I don't see that support becoming more common, as MH/nmh isn't getting many new adherents because of (among other things) lousy MIME support. From where I'm sitting, the two major mailstores in use today are Maildir and IMAP. Paul would be happy with either, since he has access to his IMAP mailserver which uses Maildir as it's backend storage, but I think it would be great if we could do both. I understand this may have no particular interest to you; that is fine, but in my mind there is a clear need. --Ken _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Paul Vixie-2
Paul Vixie writes:
> > Jon Steinhart wrote: > > ... > > Are you saying that you'd like to see the nmh cli abstracted to the point > > where it could work on different flavors of mail store? If so, that seems > > like a big change. Sort of like the hooks, but with a DLL interface > > instead of the shell? > > yes. OK. That's a really cool idea. One could even contemplate using gmail as a mail store to make it easier for the NSA to read your mail. Maybe the hooks should be replaced by one of the DLLs. This sounds like a huge change in the codebase, so hopefully someone is up for it. Being geezer-ish, I would like to avoid as much as possible the modern plague of having a program spend more time loading libraries than doing anything else. I forsee some heated API design discussions :-) Jon _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
>OK. That's a really cool idea. One could even contemplate using gmail as a
>mail store to make it easier for the NSA to read your mail. Maybe the hooks >should be replaced by one of the DLLs. This sounds like a huge change in >the codebase, so hopefully someone is up for it. Being geezer-ish, I would >like to avoid as much as possible the modern plague of having a program >spend more time loading libraries than doing anything else. I forsee some >heated API design discussions :-) I wasn't actually planning on doing DLLs for that; I mean, I don't see the point. I don't forsee people writing third-party plugins for nmh mailstores. And the API? Well, I think a significant amount of the details of that behavior has been hashed out already on the list; not specific function names and arguments, but a broad brush outline of how to deal with messages. --Ken _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Ken Hornstein-2
Ken Hornstein writes:
> >All of the email about which I care has ASCII subject lines. > >Also, I often grep for a particular attachment name. > > Right or wrong, I get subject lines that are encoded using RFC 2047 > rules. I know you think of those as 'foreign languages', but that is > wrong; they are just 'characters'. Also, if you talk to anyone younger > than 25 you're going to be getting a fair number of emoji. It's fair to > say that you don't care about that stuff, but I think as younger people > get older they are going to consider things like emoji as 'normal', and > it's something that we have to deal with. Again, it's fine if you want > to just send and receive email between old farts, but I would kindly > suggest that's not appropriate for everyone. I do have teenagers. It's not an issue, because email is a last resort for them. Maybe we need support for instagram as a mail store :-) > >Glad to hear it. Never got any feedback on 'em so it's cool to hear that > >people are using them. > > We had a big discussion about them based on Jarrad's work with his > MIME hooks (we had never really laid down how locking was supposed to > work with them). Must have missed that. But in any case, it sounds like it supports my request that the hooks be made more robust. > >I'm a bit confused here. I use nmh because I can reach the command store > >via the command line. What other way is their to use it? And other > >programs can operate on the mail store, which is part of what makes it > >great. > > I believe Paul is saying that he wants to have the command-line power of > nmh work on mail stores he shares with other MUAs. To me, that is > an admirable goal. Yes, I get it now that Paul has clarified it. > It is KIND of true that a subset of MUAs can sort-of operate on a MH > mail store, but that's only true if basically you disregard any sort > of support for sequences and locking. And I don't see that support > becoming more common, as MH/nmh isn't getting many new adherents because > of (among other things) lousy MIME support. From where I'm sitting, the > two major mailstores in use today are Maildir and IMAP. Paul would be > happy with either, since he has access to his IMAP mailserver which > uses Maildir as it's backend storage, but I think it would be great > if we could do both. > > I understand this may have no particular interest to you; that is fine, > but in my mind there is a clear need. No problem. I was just stating what I'd like. The only time that I'm likely to object to something that someone else wants is when it breaks things. Jon _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Jon Steinhart
Jon Steinhart wrote: > OK. That's a really cool idea. One could even contemplate using gmail as a > mail store to make it easier for the NSA to read your mail. Maybe the hooks > should be replaced by one of the DLLs. This sounds like a huge change in > the codebase, so hopefully someone is up for it. Being geezer-ish, I would > like to avoid as much as possible the modern plague of having a program > spend more time loading libraries than doing anything else. I forsee some > heated API design discussions :-) i believe that the gnu team has an MH-like tool set that's designed this way. if so then we might just tell people like me to go use that. -- P Vixie _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
>i believe that the gnu team has an MH-like tool set that's designed this
>way. if so then we might just tell people like me to go use that. If you're talking about GNU mailutils, then yes. I mean, it does include a MH-compatibility layer and works with IMAP; I do not know how full-featured it is. I do know that the major target of that is MH-E. I'd still rather bring us up to snuff. --Ken _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
Ken Hornstein wrote: >> i believe that the gnu team has an MH-like tool set that's designed this >> way. if so then we might just tell people like me to go use that. > > If you're talking about GNU mailutils, then yes. I mean, it does > include a MH-compatibility layer and works with IMAP; I do not know how > full-featured it is. I do know that the major target of that is MH-E. > I'd still rather bring us up to snuff. well, since it's from the era i grew up in, i'd likewise rather bring nmh up to snuff. sign me up for difficult api discussions, implementation, and testing. -- P Vixie _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Jon Steinhart
I'd just like to mention that I am still getting lots of use out of
the mh mail storage. A lot of the mail that I receive is code checkins. Perhaps when everybody on the planet has moved all of their code to github (as seems to be happening whenever I turn around) I won't need my tools to find out exactly who made the checkin that makes the tests fail, but I would be seriously inconvenienced if mh directory mail store went away. Laura _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
In reply to this post by Jon Steinhart
>As you might guess if you've been on this mailing list forever, I'd
>really like to see a better user interface for reading attachments. >In short, I'd like to keep track of the "current part", and have >"next part" and "previous part" options to show. There should be >a flag that allows switching to the next/prev message if next/prev part >runs out of parts. To me, this would be so much better than having to >do a mhlist and then type part numbers. I looked at doing this years >ago and got stopped by dealing with the parsing stuff. There should >also be an option to scan that lists the parts in messages. So if the >parser is redone maybe it will be done in such a way to support this. So I think those things would be possible. The current thinking is that the message parser would be 'lazy', in that any time a message is opened you're going to get the complete headers, and then parse the body if you ask for it via the API. I'm not convinced 'scan' is the right tool to list message parts; we'd have to think about how that would work. I would also caution you that to completely parse a message's MIME structure you need to read the whole thing; maybe this wouldn't be so bad on IMAP since you could simply ask the IMAP server for the message's MIME structure (hopefully once it built the structure it would be cached on the server). My point is that you had a scan format that showed MIME structure, the performance would probably be significantly impacted. But I have to ask about the idea of going from one MIME part to another; do you really want to do that? I can only go on how I interact with messages; almost always, I want to read the text part, and then interact in some other way with non-text parts (generally save them or open them with another tool). 'Stepping' through MIME parts is just not something that makes sense to me. Sure, I want to make it as flexible as possible and just because something doesn't make sense to me doesn't mean it shouldn't be possible! But I do want to understand the usage case completely. It occurs to me that if we do it right we could make it easy to accomplish this via scripts, which strikes me as more of the 'MH way' (basically, provide the tools to interrogate the MIME structure of a message and then go to the "next" MIME part). >I am emboldened by David's posting regarding add-hook, etc. Didn't know >that anybody actually used that stuff other than me. The hook stuff was >an expedient hack on my part, and it's not really as robust as it could >be. I would like to see it made more of an integral part of nmh as >opposed to the add-on hack that I did. Well ... do you have any thoughts there? I mean, seems like it's integrated pretty well now. Probably the two things that are lacking is a man page and tests for the test suite (more people might know about it if the various commands had pointers saying which hooks were invoked). The bottom line is that all of this is a large amount to bite off at once. My first thought is to do the MIME parser, but keep the rest of everything as close to the same as possible. Then we can talk about different mailstores --Ken _______________________________________________ Nmh-workers mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/nmh-workers |
Free forum by Nabble | Edit this page |