RE: [Exim] Maildirsize files: opinions sought

Philip Hazel wrote:
> 1. It specifies that, when adding up the sizes of files to compute the
> size of a mailbox, only the contents of the "new" and "cur"
> directories should be counted, and *not* the "tmp" directory.

As others have stated, I don't think counting files in the tmp directory
should be done either. It is after all a temporary folder for the storage
of messages to be delivered or something isn't it? If so, then a message in
the tmp folder at the time of counting quota should not penalize the user
until it's actually put in its proper place. Also as someone mentioned, if
a delivery thread crashes and leaves a tmp file there, the user can't access
it anyways, so it shouldn't count.

> 2. It also specifies that the contents of a maildirfolder called
> "Trash" should not be counted.

This I really didn't like about couriers implementation of this extension to
maildirs. I personally have been constantly playing a game of tug-rope with
myself about choosing courier-imap or cyrus-imap mainly for the quota
features. I really do prefer how cyrus does quotas, and if it ever came
down to thinkng that having couriers maildir++ quota size file implemented
as the final stage to how advanced quota support will get in Exim, I'd turn
green and break stuff.

Now, if we're really going for courier-imap compatibility, then some simple
colon delimited list of mailboxes (which would allow regexes, and even
general string expansion) of what won't be counted (default count all
folders) should be added. Not counting a Trash folder is a very
courier-imap-ism that really doesn't make sense. They even went in and
added some feature to auto purge the Trash folder after x days or whatever.
I wouldn't want to imagine the number of calls/emails my clients would send
when they find out a message they wanted trashed but saved got nuked. They
already scream enough when their webmail session times out because they take
too long to compose a message!

If you're more or less leaning towards an idea for making quota calculation
more efficient, then I'd actually like to see something that allowed more of
a cyrus quota system, where individual folders could have quotas (child
overrides parent). It would really be nice to give users say a 5mb quota,
but only let them dedicate 1mb of that to their trash folder (for all those
clients who think marking a message for deletion means they deleted it...
grrrr). As for how accurate quota calculation could be done without mua
support - well, what if you let us do db queries (and just ignore the mua
aspect). At delivery time we know the size of the message, so we could
easily store this into a db, and then do a quick total to find out if
they're over quota or not. I'm sure this would be far faster than even
parsing file names with a regular expression. If muas want to support yet
another quota method (it had better be a good one!), it would be nice...

I really don't like how the maildirsize has + and - beside filenames. If a
file is deleted (physically deleted, not just marked for), its recorded in
the maildirsize file by putting a - beside the size (and filename? I forget
what the exact contents look like). Why not just nuke the entry? They say
as soon as a - is encountered, the maildirsize is to be recalculated... Well
that kinda defeats the purpose of the file half the time if the mailbox is
seeing constant user interaction. I really don't see any benefit to
recording a deleted message. In my personal view, the maildirsize
implementation - although the idea of being able to speed up quota
calculations is nice - the resulting way courier-imap did it is somewhat
lacking. If you record only te files that exist - both the mta and mua will
be modifying the file (as they are anyways), so just record the message you
insert to the folder and nuke any that was removed. Each file should have a
unique file name, so record that with the size. When an insertion is made,
just append (who cares about the order in the file). When a deletion is
made, just do a quick search on the file names and delete. Heck, if you
want to get faster, we should be using at the very least a db file, or some
sort of binary file or whatever no? Text searches are so slow. And my
ramblings go on and on... I'm sure you get the idea and can see what
direction I'm leaning in.

> But maybe some of you disagree? As we don't use maildir round here, I
> don't have any actual experience to go on.

I just wanted to use maildirsize files because of the possible i/o
improvement from not constantly stat()ing files to get sizes and such. I've
since switched to using regexes - although neither have been in production
and/or have seen any actual speed tests, I just think the regex stuff is
nicer at the moment.

If we implemented a whole new db driven scheme for quota calculation, it
would make exim just that much more cooler, and it would be one hell of a
big improvement - especially if it could do per folder quotas.

Eli.