mutt corrupting mboxes?

[Jim Diamond]

I'm using mutt 1.5.23 on a number of Slackware 64 14.1 systems, all
with ext4 file systems.

Every now and then, one of my mailboxes gets corrupted. Looking at
the mailbox with a text editor, it is like parts of one message got
randomly thrown down on top of one or more other messages. Further,
the parts of the (apparent) "thrown-down" message are not always in
order. For example, I may get part of the header, part of the body,
more header, and more body.

I've never seen this happen in my inbox, only in mailboxes into which
I've moved some message. (I use fetchmail + procmail to d/l my mail
to my inbox.)


I don't know quite what to make of this. I would think that if mutt
was clobbering mailboxes on a regular basis, there would be a lot of
noise in this newsgroup (assuming there are more than three of us
reading here).

On the other hand, if there were problems with ext4 file systems, I
would think that
(a) there would be lots of screaming world-wide, and
(b) I'd see other corrupted files on my system.


Q1: Has anyone reading this message ever seen something like that?

Q2: If there are any mutt devs reading this, any suggestions on
how I might track this down, even if it involves a bit of hacking
of mutt's source code?

Thanks.

Jim

[Jorgen Grahn]

Post by Jim Diamond
I'm using mutt 1.5.23 on a number of Slackware 64 14.1 systems, all
with ext4 file systems.
Every now and then, one of my mailboxes gets corrupted.

And I suppose Mutt tells you so, while trying to access it?

Post by Jim Diamond
Looking at
the mailbox with a text editor, it is like parts of one message got
randomly thrown down on top of one or more other messages. Further,
the parts of the (apparent) "thrown-down" message are not always in
order. For example, I may get part of the header, part of the body,
more header, and more body.
I've never seen this happen in my inbox, only in mailboxes into which
I've moved some message. (I use fetchmail + procmail to d/l my mail
to my inbox.)

Ok, so new mail ends up in your inbox, and the problem mboxes (it's
mbox format you use, right?) are the ones where you manually make mutt
place messages, using (s)ave and similar commands, right?

Post by Jim Diamond
I don't know quite what to make of this. I would think that if mutt
was clobbering mailboxes on a regular basis, there would be a lot of
noise in this newsgroup (assuming there are more than three of us
reading here).

Mutt is used quite a lot by various people -- even I know other users
personally -- but mutt 1.5.23, with whatever patches Slackware applied
to it ... perhaps that's a more narrow target.

Post by Jim Diamond
On the other hand, if there were problems with ext4 file systems, I
would think that
(a) there would be lots of screaming world-wide, and
(b) I'd see other corrupted files on my system.

Yes. And bugs which cause file system corruption is something the
Linux kernel developers avoid.

Post by Jim Diamond
Q1: Has anyone reading this message ever seen something like that?

No, never.

Post by Jim Diamond
Q2: If there are any mutt devs reading this, any suggestions on
how I might track this down, even if it involves a bit of hacking
of mutt's source code?

I'm not a Mutt developer, but if you could find a reliable way of
reproducing it, that would help a lot. If you could say to them "take
<these> two mboxes, and perform <these> actions, and one of them will
become corrupted into <this>" ... you have the perfect bug report.

Perhaps the bug report should go to Slackware first.

/Jorgen

[Jim Diamond]

Post by Jorgen Grahn

Post by Jim Diamond
I'm using mutt 1.5.23 on a number of Slackware 64 14.1 systems, all
with ext4 file systems.
Every now and then, one of my mailboxes gets corrupted.

And I suppose Mutt tells you so, while trying to access it?

Two ways:
(1) mairix crashes and burns when it runs across a "From " line which
is neither at the beginning of the file nor following a blank line.
(2) If I go to read a corrupted message, it becomes obvious it has
been clobbered.

Post by Jorgen Grahn

Post by Jim Diamond
I've never seen this happen in my inbox, only in mailboxes into which
I've moved some message. (I use fetchmail + procmail to d/l my mail
to my inbox.)

Ok, so new mail ends up in your inbox, and the problem mboxes (it's
mbox format you use, right?)

Yes.

Post by Jorgen Grahn
are the ones where you manually make mutt place messages, using
(s)ave and similar commands, right?

Exactly.

Post by Jorgen Grahn

Post by Jim Diamond
I don't know quite what to make of this. I would think that if mutt
was clobbering mailboxes on a regular basis, there would be a lot of
noise in this newsgroup (assuming there are more than three of us
reading here).

Mutt is used quite a lot by various people -- even I know other users
personally -- but mutt 1.5.23, with whatever patches Slackware applied
to it ... perhaps that's a more narrow target.

The Slackware BDFL tends not to patch software. However, you are
possibly on to something...
I am using a version I compiled myself to get the compressed mailbox
capability. So it is possible that is (part of?) the problem. I will
mention that the problems I'm seeing are in uncompressed mailboxes.

Post by Jorgen Grahn

Post by Jim Diamond
Q1: Has anyone reading this message ever seen something like that?

No, never.

You or everyone? :-)

Post by Jorgen Grahn

Post by Jim Diamond
Q2: If there are any mutt devs reading this, any suggestions on
how I might track this down, even if it involves a bit of hacking
of mutt's source code?

I'm not a Mutt developer, but if you could find a reliable way of
reproducing it, that would help a lot.

Don't I know it.

Post by Jorgen Grahn
If you could say to them "take
<these> two mboxes, and perform <these> actions, and one of them will
become corrupted into <this>" ... you have the perfect bug report.

I would. Part of the problem is that by the time I discover the
problem, I no longer have the original uncorrupted mail message to try
saving into another mbox again.

Post by Jorgen Grahn
Perhaps the bug report should go to Slackware first.

For the reasons above, that won't work.

Thanks for your thoughts.

Jim

[Jorgen Grahn]

...

Post by Jim Diamond
I am using a version I compiled myself to get the compressed mailbox
capability. So it is possible that is (part of?) the problem. I will
mention that the problems I'm seeing are in uncompressed mailboxes.

That patch (or one such patch at least) is widely used; Debian's
version of Mutt has it, for example. So at least it's not a famously
broken patch, but I guess you knew that already ...

/Jorgen

[Jim Diamond]

Post by Jorgen Grahn
...

Post by Jim Diamond
I am using a version I compiled myself to get the compressed mailbox
capability. So it is possible that is (part of?) the problem. I will
mention that the problems I'm seeing are in uncompressed mailboxes.

That patch (or one such patch at least) is widely used; Debian's
version of Mutt has it, for example. So at least it's not a famously
broken patch, but I guess you knew that already ...

I had that impression. But thanks for the info, I didn't know it was
"mainstream" in Debian.

Cheers.
Jim

[Ian Zimmerman]

Post by Jim Diamond
Every now and then, one of my mailboxes gets corrupted. Looking at
the mailbox with a text editor, it is like parts of one message got
randomly thrown down on top of one or more other messages. Further,
the parts of the (apparent) "thrown-down" message are not always in
order. For example, I may get part of the header, part of the body,
more header, and more body.
I've never seen this happen in my inbox, only in mailboxes into which
I've moved some message. (I use fetchmail + procmail to d/l my mail
to my inbox.)

Does any part of your pipeline do "From escaping"? You can test it
right here: _in mutt_, does the following line start with a > ?

Post by Jim Diamond
From Ian, who has given up on mbox recently due to related issues.

[Jim Diamond]

Post by Ian Zimmerman

Post by Jim Diamond
Every now and then, one of my mailboxes gets corrupted. Looking at
the mailbox with a text editor, it is like parts of one message got
randomly thrown down on top of one or more other messages. Further,
the parts of the (apparent) "thrown-down" message are not always in
order. For example, I may get part of the header, part of the body,
more header, and more body.
I've never seen this happen in my inbox, only in mailboxes into which
I've moved some message. (I use fetchmail + procmail to d/l my mail
to my inbox.)

Does any part of your pipeline do "From escaping"? You can test it
right here: _in mutt_, does the following line start with a > ?

Post by Jim Diamond
From Ian, who has given up on mbox recently due to related issues.

Hmmm... in general, I do end up with escaped "From " lines, although
I'm not sure "who" does it. (Sendmail?) (I just sent myself a
message with a line starting with "From " and in the (local) copy mutt
makes for me, there is no '>', and in the copy I received via SMTP, my
mail server, fetchmail and procmail there is a '>'.)

Specifically, I'm not sure exactly what you were asking me to do. I
read news with slrn and I saved your message to a file, and when
looking at the text file with both less and mutt, I see the '>'.

Cheers.
Jim

[Ian Zimmerman]

Post by Jim Diamond
Hmmm... in general, I do end up with escaped "From " lines, although
I'm not sure "who" does it. (Sendmail?) (I just sent myself a
message with a line starting with "From " and in the (local) copy mutt
makes for me, there is no '>', and in the copy I received via SMTP, my
mail server, fetchmail and procmail there is a '>'.)

If you just save an unescaped message to its own file there's no
problem. But if you append one to a mbox (and that's what mutt S
command does, if the file already exists), it's a big problem.

In my case, I had long assumed procmail did it if the earlier stages
didn't. I was wrong and I ended up with corrupted inbox. Had to muck
with my MTA (exim) settings for it to happen.

Then I switched to maildirs (only recently) and I'm happy. IMO this is
a much better selling point for maildirs than lock freedom.

Post by Jim Diamond
Specifically, I'm not sure exactly what you were asking me to do. I
read news with slrn and I saved your message to a file, and when
looking at the text file with both less and mutt, I see the '>'.

I had sent you a copy via email, and that copy was the one which
mattered.

[Eike Rathke]

Post by Ian Zimmerman
I had sent you a copy via email, and that copy was the one which
mattered.

He didn't want your precious help, he mangled his From address.

Eike

[Jim Diamond]

Post by Eike Rathke

Post by Ian Zimmerman
I had sent you a copy via email, and that copy was the one which
mattered.

He didn't want your precious help, he mangled his From address.

That's a bit theatrical, don't you think?

Admittedly, someone wanting to e-mail has to notice "deletethis.",
figure out what it means, and take a bit of action. Further, the
beauty of newsgroups is that in most cases someone can help you
without sending you mail.

Jim

[Jim Diamond]

Post by Ian Zimmerman

Post by Jim Diamond
Hmmm... in general, I do end up with escaped "From " lines, although
I'm not sure "who" does it. (Sendmail?) (I just sent myself a
message with a line starting with "From " and in the (local) copy mutt
makes for me, there is no '>', and in the copy I received via SMTP, my
mail server, fetchmail and procmail there is a '>'.)

If you just save an unescaped message to its own file there's no
problem. But if you append one to a mbox (and that's what mutt S
command does, if the file already exists), it's a big problem.

I just tried this out by inserting (with vim) the line
From aslkas laks lakskl alkalskalks laks
(but with no leading whitespace) into the middle of a message (the
third message in an mbox with three messages). My version of mutt
doesn't have a problem reading that nmox. (Without looking at the
code, I might guess that mutt is smart enough to realize the above
line isn't a message-starting line.)

Post by Ian Zimmerman
In my case, I had long assumed procmail did it if the earlier stages
didn't. I was wrong and I ended up with corrupted inbox. Had to muck
with my MTA (exim) settings for it to happen.
Then I switched to maildirs (only recently) and I'm happy. IMO this is
a much better selling point for maildirs than lock freedom.

I'm beginning to see that. Having said that, I have over 100,000
messages scattered through about 350 mailboxes and at first thought
I'm not sure I like the idea of creating another 100K files in this
file system. (But, having said that, maybe it would not be an issue.)

Post by Ian Zimmerman

Post by Jim Diamond
Specifically, I'm not sure exactly what you were asking me to do. I
read news with slrn and I saved your message to a file, and when
looking at the text file with both less and mutt, I see the '>'.

I had sent you a copy via email, and that copy was the one which
mattered.

As you may know by now, my addr above is very-lightly munged. Sorry
that your effort to help went astray because of this.

Having said all this, I don't believe an unescaped From is causing the
problem I'm seeing, because it isn't just that mutt is getting
confused reading an mbox, it is that the mbox itself is corrupted with
pieces of a message mixed up and scattered.

Thanks for your thoughts.

Jim

[Jorgen Grahn]

Post by Ian Zimmerman

Post by Jim Diamond
Hmmm... in general, I do end up with escaped "From " lines, although
I'm not sure "who" does it. (Sendmail?) (I just sent myself a
message with a line starting with "From " and in the (local) copy mutt
makes for me, there is no '>', and in the copy I received via SMTP, my
mail server, fetchmail and procmail there is a '>'.)

If you just save an unescaped message to its own file there's no
problem. But if you append one to a mbox (and that's what mutt S
command does, if the file already exists), it's a big problem.
In my case, I had long assumed procmail did it if the earlier stages
didn't. I was wrong and I ended up with corrupted inbox. Had to muck
with my MTA (exim) settings for it to happen.

On my system, the inbox lets From_ mark message borders (so it escapes
From_ in the messages), but when Mutt (s)aves to a folder or saves an
Fcc copy, it uses the variation[0] where

Content-Length: 164
Lines: 7

marks the messages. Here no >From escaping is added by Mutt.

I can imagine copying from one of these folders to the inbox would
cause problems.

I guess other folders which procmail writes to would work like the
inbox.

Post by Ian Zimmerman
Then I switched to maildirs (only recently) and I'm happy. IMO this is
a much better selling point for maildirs than lock freedom.

Yeah; if it ever happened to me, I'd consider switching. But I've
used mbox heavily for 25 years without problems. I admit that perhaps
I've just been lucky.

/Jorgen

[0] Can't be bothered to look it up, but Wikipedia's entry for 'mbox'
leads to names and tentative definitions for these file formats.

[Jim Diamond]

Post by Jorgen Grahn

Post by Ian Zimmerman

Post by Jim Diamond
Hmmm... in general, I do end up with escaped "From " lines, although
I'm not sure "who" does it. (Sendmail?) (I just sent myself a
message with a line starting with "From " and in the (local) copy mutt
makes for me, there is no '>', and in the copy I received via SMTP, my
mail server, fetchmail and procmail there is a '>'.)

If you just save an unescaped message to its own file there's no
problem. But if you append one to a mbox (and that's what mutt S
command does, if the file already exists), it's a big problem.
In my case, I had long assumed procmail did it if the earlier stages
didn't. I was wrong and I ended up with corrupted inbox. Had to muck
with my MTA (exim) settings for it to happen.

On my system, the inbox lets From_ mark message borders (so it escapes
From_ in the messages), but when Mutt (s)aves to a folder or saves an
Fcc copy, it uses the variation[0] where
Content-Length: 164
Lines: 7
marks the messages. Here no >From escaping is added by Mutt.

I tried deleting (with vim) those lines from the header of a message
with an unescaped "From ...gibberish..." line, and mutt still did not
interpret it as the beginning of a new message.

Post by Jorgen Grahn
I can imagine copying from one of these folders to the inbox would
cause problems.
I guess other folders which procmail writes to would work like the
inbox.

In my case, I only have procmail write to my main inbox and a spam mbox.

Jim

[RS Wood]

Interesting problem. I haven't experienced it myself, but use mostly
IMAP myself. But I use offlineimap to replicate mailboxes locally -
I'm not sure what format that's considered, but it consists of a folder
tree with /cur, /new/ and /tmp.

Just out of curiosity, is it not potentially filesystem trouble or disk
trouble? Are you seeing flakiness anywhere else in your system? Other
culprits - backup software, recent hard shutdowns, etc. Does any other
software access that mail? I recall having trouble in 2003 when I was
using Thunderbird, Kmail, Mutt, and Alpine simultaneously from the same
mailbox (maybe not Thunderbird, can't recall). They really didn't play
well together; the move to IMAP solved that problem.

Can't offer much more help than that. Very strange problem.

[Jim Diamond]

Post by RS Wood
Interesting problem.

Indeed. I wish it was someone else's interesting problem ;-)

Post by RS Wood
I haven't experienced it myself, but use mostly IMAP myself. But I
use offlineimap to replicate mailboxes locally - I'm not sure what
format that's considered, but it consists of a folder tree with
/cur, /new/ and /tmp.
Just out of curiosity, is it not potentially filesystem trouble or disk
trouble?

It is possible, but...

Post by RS Wood
Are you seeing flakiness anywhere else in your system?

No. And while I don't do a checksum of every file every day, I would
think that sooner or later I'd see a problem with some file other than
a mailbox.

Post by RS Wood
Other culprits - backup software, recent hard shutdowns, etc. Does
any other software access that mail?

procmail dumps mail only into my inbox + a SPAM mailbox. The problems
I've seen are only with mailboxes into which I have moved a mail
message (using mutt to do that).

Post by RS Wood
I recall having trouble in 2003 when I was using Thunderbird, Kmail,
Mutt, and Alpine simultaneously from the same mailbox (maybe not
Thunderbird, can't recall). They really didn't play well together;
the move to IMAP solved that problem.

For better or worse, that isn't the issue for me.

Post by RS Wood
Can't offer much more help than that. Very strange problem.

It is. Thanks for the thoughts.

Jim

[gregor herrmann]

Post by RS Wood
Interesting problem. I haven't experienced it myself, but use mostly
IMAP myself. But I use offlineimap to replicate mailboxes locally -
I'm not sure what format that's considered, but it consists of a folder
tree with /cur, /new/ and /tmp.

That's maildir.


gregor