ph10 2004/10/08 11:38:48 BST
Added files:
exim-doc/doc-misc ABOUT Ext-maildir Ext-maildir++
Ext-mbx-locking LongTermIssues
RFC.conform TexiNotes WishList
exim-doc/doc-scripts ABOUT
exim-doc/doc-src ABOUT
exim-doc/doc-txt ABOUT
exim-src ABOUT
Log:
Start
Revision Changes Path
1.1 +11 -0 exim/exim-doc/doc-misc/ABOUT (new)
1.1 +109 -0 exim/exim-doc/doc-misc/Ext-maildir (new)
1.1 +394 -0 exim/exim-doc/doc-misc/Ext-maildir++ (new)
1.1 +400 -0 exim/exim-doc/doc-misc/Ext-mbx-locking (new)
1.1 +200 -0 exim/exim-doc/doc-misc/LongTermIssues (new)
1.1 +401 -0 exim/exim-doc/doc-misc/RFC.conform (new)
1.1 +193 -0 exim/exim-doc/doc-misc/TexiNotes (new)
1.1 +1727 -0 exim/exim-doc/doc-misc/WishList (new)
1.1 +9 -0 exim/exim-doc/doc-scripts/ABOUT (new)
1.1 +11 -0 exim/exim-doc/doc-src/ABOUT (new)
1.1 +9 -0 exim/exim-doc/doc-txt/ABOUT (new)
1.1 +11 -0 exim/exim-src/ABOUT (new)
Index: ABOUT
====================================================================
$Cambridge: exim/exim-doc/doc-misc/ABOUT,v 1.1 2004/10/08 10:38:47 ph10 Exp $
CVS directory exim/exim-doc/doc-misc
------------------------------------
This directory contains some miscellaneous documentation files that do not form
part of Exim distributions, but are related to its maintenance and development.
Those whose names start with "Ext-" are external documents that won't be
modified (and hence have no local CVS Ids).
End
Index: Ext-maildir
====================================================================
The following information is from the maildir man page of qmail.
INTRODUCTION
maildir is a structure for directories of incoming mail
messages. It solves the reliability problems that plague
mbox files and mh folders.
RELIABILITY ISSUES
A machine may crash while it is delivering a message. For
both mbox files and mh folders this means that the message
will be silently truncated. Even worse: for mbox format,
if the message is truncated in the middle of a line, it
will be silently joined to the next message. The mail
transport agent will try again later to deliver the mes-
sage, but it is unacceptable that a corrupted message
should show up at all. In maildir, every message is guar-
anteed complete upon delivery.
A machine may have two programs simultaneously delivering
mail to the same user. The mbox and mh formats require
the programs to update a single central file. If the pro-
grams do not use some locking mechanism, the central file
will be corrupted. There are several mbox and mh locking
mechanisms, none of which work portably and reliably. In
contrast, in maildir, no locks are ever necessary. Dif-
ferent delivery processes never touch the same file.
A user may try to delete messages from his mailbox at the
same moment that the machine delivers a new message. For
mbox and mh formats, the user's mail-reading program must
know what locking mechanism the mail-delivery programs
use. In contrast, in maildir, any delivered message can
be safely updated or deleted by a mail-reading program.
Many sites use Sun's Network Failure System (NFS), presum-
ably because the operating system vendor does not offer
anything else. NFS exacerbates all of the above problems.
Some NFS implementations don't provide any reliable lock-
ing mechanism. With mbox and mh formats, if two machines
deliver mail to the same user, or if a user reads mail
anywhere except the delivery machine, the user's mail is
at risk. maildir works without trouble over NFS.
THE MAILDIR STRUCTURE
A directory in maildir format has three subdirectories,
all on the same filesystem: tmp, new, and cur.
Each file in new is a newly delivered mail message. The
modification time of the file is the delivery date of the
message. The message is delivered without an extra UUCP-
style From_ line, without any >From quoting, and without
an extra blank line at the end. The message is normally
in RFC 822 format, starting with a Return-Path line and a
Delivered-To line, but it could contain arbitrary binary
data. It might not even end with a newline.
Files in cur are just like files in new. The big differ-
ence is that files in cur are no longer new mail: they
have been seen by the user's mail-reading program.
HOW A MESSAGE IS DELIVERED
The tmp directory is used to ensure reliable delivery, as
discussed here.
A program delivers a mail message in six steps. First, it
chdir()s to the maildir directory. Second, it stat()s the
name tmp/time.pid.host, where time is the number of sec-
onds since the beginning of 1970 GMT, pid is the program's
process ID, and host is the host name. Third, if stat()
returned anything other than ENOENT, the program sleeps
for two seconds, updates time, and tries the stat() again,
a limited number of times. Fourth, the program creates
tmp/time.pid.host. Fifth, the program NFS-writes the mes-
sage to the file. Sixth, the program link()s the file to
new/time.pid.host. At that instant the message has been
successfully delivered.
The delivery program is required to start a 24-hour timer
before creating tmp/time.pid.host, and to abort the deliv-
ery if the timer expires. Upon error, timeout, or normal
completion, the delivery program may attempt to unlink()
tmp/time.pid.host.
NFS-writing means (1) as usual, checking the number of
bytes returned from each write() call; (2) calling fsync()
and checking its return value; (3) calling close() and
checking its return value. (Standard NFS implementations
handle fsync() incorrectly but make up for it by abusing
close().)
HOW A MESSAGE IS READ
A mail reader operates as follows.
It looks through the new directory for new messages. Say
there is a new message, new/unique. The reader may freely
display the contents of new/unique, delete new/unique, or
rename new/unique as cur/unique:info. See
http://pobox.com/~djb/maildir.html for the meaning of
info.
The reader is also expected to look through the tmp direc-
tory and to clean up any old files found there. A file in
tmp may be safely removed if it has not been accessed in
36 hours.
It is a good idea for readers to skip all filenames in new
and cur starting with a dot. Other than this, readers
should not attempt to parse filenames.
###
Index: Ext-maildir++
====================================================================
Maildir++
In this document:
* HOWTO.maildirquota
* Mission statement
* Definitions and goals
* Contents of a maildirsize
* Calculating maildirsize
* Calculating the quota for a Maildir++
* Delivering to a Maildir++
* Reading from a Maildir++
* Bugs
HOWTO.maildirquota
The remaining portion of this document is a technical description of
the maildir quota extension. This section is a brief overview of this
extension.
What is a maildirquota?
If you would like to have a quota on your maildir mailboxes, the best
solution is to always use filesystem-based quotas: per-user usage
quotas that is enforced by the operating system.
This is the best solution when the default Maildir is located in each
account's home directory. This solution will NOT work if Maildirs are
stored elsewhere, or if you have a large virtual domain setup where a
single userid is used to hold many individual Maildirs, one for each
virtual user.
This extension to the maildir format allows a "voluntary" maildir
quota implementation that does not rely on filesystem-based quotas.
When maildirquota will not work.
For this quota mechanism to work, all software that accesses a maildir
must observe this quota protocol. It follows that this quota mechanism
can be easily circumvented if users have direct (shell) access to the
filesystem containing the users' maildirs.
Furthermore, this quota mechanism is not 100% effective. It is
possible to have a situation where someone may go over quota. This
quota implementation uses a deliverate trade-off. It is necessary to
use some form of locking in order to have a complete bulletproof quota
enforcement, but maildirs mail stores were explicitly designed to
avoid any kind of locking. This quota approach does not use locking,
and the tradeoff is that sometimes it is possible for a few extra
messages to be delivered to the maildir, before the door is
permanently shot.
For best performance, all maildir clients should support this quota
extension, however there's a wide degree of tolerance here. As long as
the mail delivery agent that puts new messages into a Maildir uses
this extension, the quota will be enforced without excessive
degradation.
In the worst case scenario, quotas are automatically recalculated
every fifteen minutes. If a maildir goes over quota, and a mail client
that does not support this quota extension removes enough mail from
the maildir, the mail delivery agent will not be immediately informed
that the maildir is now under quota. However, eventually the correct
quota will be recalculated and mail delivery will resume.
Mail user agents sometimes put messages into the maildir themselves.
Messages added to a maildir by a mail user agent that does not
understand the quota extension will not be immediately counted towards
the overall quota, and may not be counted for an extensive period of
time. Additionally, if there are a lot of messages that have been
added to a maildir from these mail user agents, quota recalculation
may impose non-trivial load on the system, as the quota recalculator
will have to issue the stat system call for each message.
How to implement the quota
The best way to do that is to modify your mail server to implement the
protocol defined by this document. Not everyone, of course, has this
ability. Therefore, an alternate approach is available.
This package creates a very short utility called "deliverquota". It
will NOT be installed anywhere by default, unless this maildir quota
implementation is a part of a larger package, in which case the parent
package may install this utility somewhere. If you obtained the
maildir package separately, you will need to compile it by running the
configure script, then by running make.
deliverquota takes two arguments. deliverquota reads the message from
standard input, then delivers it to the maildir specified by the first
argument to deliverquota. The second argument specifies the actual
quota for this maildir, as defined elsewhere in this document.
deliverquota will deliver the message to the maildir, making a best
effort not to exceed the stated quota. If the maildir is over quota,
deliverquota terminates with exit code 77. Otherwise, it delivers the
message, updates the quota, and terminates with exit code 0.
Therefore, proceed as follows:
* Copy deliverquota to some convenient location, say /usr/local/bin.
* Configure your mail server to use deliverquota. For example, if
you use Qmail and your maildirs are all located in $HOME/Maildir,
replace the './Maildir/' argument to qmail-start with the
following:
'| /usr/local/bin/deliverquota ./Maildir 1000000S'
This sets a one million byte limit on all Maildirs. As I
mentioned, this is meaningless if login access is available,
because the individual account owner can create his own
$HOME/.qmail file, and ignore deliverquota. Note that in this
case, you MUST use apostrophes on the qmail-start command line, in
order to quote this as one argument.
If you would like to use different quotas for different users, you
will have to put together a separate process or a script that looks up
the appropriate quota for the recipient, and runs deliverquota
specifying the quota. If no login access to the mail server is
available, you can simply create a separate $HOME/.qmail for every
recipient.
That's pretty much it. If you handle a moderate amount of mail, I have
one more suggestion. For the first couple of weeks, run deliverquota
setting the second argument to an empty string. This disables quota
enforcement, however it still activates certain optimizations that
permit very fast quota recalculation. Messages delivered by
deliverquota have their message size encoded in their filename; this
makes it possible to avoid stat-ing the message in the Maildir, when
recalculating the quota. Then, after most messages in your maildirs
have been delivered by deliverquota, activate the quotas!!!
maildirquota-enhanced applications
This is a list of applications that have been enhanced to support the
maildirquota extension:
* maildrop - mail delivery agent/mail filter.
* SqWebmail - webmail CGI binary.
These applications fall into two classes:
* Mail delivery agents. These applications read some externally
defined table of mail recipients and their maildir quota.
* Mail clients. These applications read maildir quota information
that has been defined by the mail delivery agent.
Mail clients generally do not need any additional setup in order to
use the maildirquota extension. They will automatically read and
implement any quota specification set by the mail delivery agent.
On the other hand, mail delivery agents will require some kind of
configuration in order to activate the maildirquota extension for some
or all recipients. The instructions for doing that depends upon the
mail delivery agent. The documentation for the mail delivery agent
should be consulted for additional information.
_________________________________________________________________
Mission statement
Maildir++ is a mail storage structure that's based on the Maildir
structure, first used in the Qmail mail server. Actually, Maildir++ is
just a minor extension to the standard Maildir structure.
For more information, see http://www.qmail.org/man/man5/maildir.html.
I am not going to include the definition of a Maildir in this
document. Consider it included right here. This document only
describes the differences.
Maildir++ adds a couple of things to a standard Maildir: folders and
quotas.
Quotas enforce a maximum allowable size of a Maildir. In many
situations, using the quota mechanism of the underlying filesystem
won't work very well. If a filesystem quota mechanism is used, then
when a Maildir goes over quota, Qmail does not bounce additional mail,
but keeps it queued, changing one bad situation into another bad
situation. Not only know you have an account that's backed up, but now
your queue starts to back up too.
Definitions, and goals
Maildir++ and Maildir shall be completely interchangeable. A Maildir++
client will be able to use a standard Maildir, automatically
"upgrading" it in the process. A Maildir client will be able to use a
Maildir++ just like a regular Maildir. Of course, a plain Maildir
client won't be able to enforce a quota, and won't be able to access
messages stored in folders.
Folders are created as subdirectories under the main Maildir. The name
of the subdirectory always starts with a period. For example, a folder
named "Important" will be a subdirectory called ".Important". You
can't have subdirectories that start with two periods.
A Maildir++ client ignores anything in the main Maildir that starts
with a period, but is not a subdirectory.
Each subdirectory is a fully-fledged Maildir of its own, that is you
have .Important/tmp, .Important/new, and .Important/cur. Everything
that applies to the main Maildir applies equally well to the
subdirectory, including automatically cleaning up old files in tmp. A
Maildir++ enhancement is that a message can be moved between folders
and/or the main Maildir simply by moving/renaming the file (into the
cur subdirectory of the destination folder). Therefore, the entire
Maildir++ must reside on the same filesystem.
Within each subdirectory there's an empty file, maildirfolder. Its
existence tells the mail delivery agent that this Maildir is a really
a folder underneath a parent Maildir++.
Only one special folder is reserved: Trash (subdirectory .Trash).
Instead of marking deleted messages with the D flag, Maildir++ clients
move the message into the Trash folder. Maildir++ readers are
responsible for expunging messages from Trash after a system-defined
retention interval.
When a Maildir++ reader sees a message marked with a D flag it may at
its option: remove the message immediately, move it into Trash, or
ignore it.
Can folders have subfolders, defined in a recursive fashion? The
answer is no. If you want to have a client with a hierarchy of
folders, emulate it. Pick a hierarchy separator character, say ":".
Then, folder foo/bar is subdirectory .foo:bar.
This is all that there's to say about folders. The rest of this
document deals with quotas.
The purpose of quotas is to temporarily disable a Maildir, if it goes
over the quota. There is one and only major goal that this quota
implementation tries to achieve:
* Place as little overhead as possible on the mail system that's
delivering to the Maildir++
That's it. To achieve that goal, certain compromises are made:
* Mail delivery will stop as soon as possible after Maildir++'s size
goes over quota. Certain race conditions may happen with Maildir++
going a lot over quota, in rare circumstances. That is taken into
account, and the situation will eventually resolve itself, but you
should not simply take your systemwide quota, multiply it by the
number of mail accounts, and allocate that much disk space. Always
leave room to spare.
* How well the quota mechanism will work will depend on whether or
not everything that accesses the Maildir++ is a Maildir++ client.
You can have a transition period where some of your mail clients
are just Maildir clients, and things should run more or less well.
There will be some additional load because the size of the Maildir
will be recalculated more often, but the additional load shouldn't
be noticeable.
This won't be a perfect solution, but it will hopefully be good
enough. Maildirs are simply designed to rely on the filesystem to
enforce individual quotas. If a filesystem-based quota works for you,
use it.
A Maildir++ may contain the following additional file: maildirsize.
Contents of maildirsize
maildirsize contains two or more lines terminated by newline
characters.
The first line contains a copy of the quota definition as used by the
system's mail server. Each application that uses the maildir must know
what it's quota is. Instead of configuring each application with the
quota logic, and making sure that every application's quota definition
for the same maildir is exactly the same, the quota specification used
by the system mail server is saved as the first line of the
maildirsize file. All other application that enforce the maildir quota
simply read the first line of maildirsize.
The quota definition is a list, separate by commas. Each member of the
list consists of an integer followed by a letter, specifying the
nature of the quota. Currently defined quota types are 'S' - total
size of all messages, and 'C' - the maximum count of messages in the
maildir. For example, 10000000S,1000C specifies a quota of 10,000,000
bytes or 1,000 messages, whichever comes first.
All remaining lines all contain two integers separated by a single
space. The first integer is interpreted as a byte count. The second
integer is interpreted as a file count. A Maildir++ writer can add up
all byte counts and file counts from maildirsize and enforce a quota
based either on number of messages or the total size of all the
messages.
Calculating maildirsize
In most cases, changes to maildirsize are recorded by appending an
additional line. Under some conditions maildirsize has to be
recalculated from scratch. These conditions are defined later. This is
the procedure that's used to recalculate maildirsize:
1. If we find a maildirfolder within the directory, we're delivering
to a folder, so back up to the parent directory, and start again.
2. Read the contents of the new and cur subdirectories. Also, read
the contents of the new and cur subdirectories in each Maildir++
folder, except Trash. Before reading each subdirectory, stat() the
subdirectory itself, and keep track of the latest timestamp you
get.
3. If the filename of each message is of the form xxxxx,S=nnnnn or
xxxxx,S=nnnnn:xxxxx where "xxxxx" represents arbitrary text, then
use nnnnn as the size of the file (which will be conveniently
recorded in the filename by a Maildir++ writer, within the
conventions of filename naming in a Maildir). If the message was
not written by a Maildir++ writer, stat() it to obtain the message
size. If stat() fails, a race condition removed the file, so just
ignore it and move on to the next one.
4. When done, you have the grand total of the number of messages and
their total size. Create a new maildirsize by: creating the file
in the tmp subdirectory, observing the conventions for writing to
a Maildir. Then rename the file as maildirsize.Afterwards, stat
all new and cur subdirectories again. If you find a timestamp
later than the saved timestamp, REMOVE maildirsize.
5. Before running this calculation procedure, the Maildir++ user
wanted to know the size of the Maildir++, so return the calculated
values. This is done even if maildirsize was removed.
Calculating the quota for a Maildir++
This is the procedure for reading the contents of maildirsize for the
purpose of determine if the Maildir++ is over quota.
1. If maildirsize does not exist, or if its size is at least 5120
bytes, recalculate it using the procedure defined above, and use
the recalculated numbers. Otherwise, read the contents of
maildirsize, and add up the totals.
2. The most efficient way of doing this is to: open maildirsize, then
start reading it into a 5120 byte buffer (some broken NFS
implementations may return less than 5120 bytes read even before
reaching the end of the file). If we fill it, which, in most
cases, will happen with one read, close it, and run the
recalculation procedure.
3. In many cases the quota calculation is for the purpose of adding
or removing messages from a Maildir++, so keep the file descriptor
to maildirsize open. A file descriptor will not be available if
quota recalculation ended up removing maildirsize due to a race
condition, so the caller may or may not get a file descriptor
together with the Maildir++ size.
4. If the numbers we got indicated that the Maidlir++ is over quota,
some additional logic is in order: if we did not recalculate
maildirsize, if the numbers in maildirsize indicated that we are
over quota, then if maildirsize was more than one line long, or if
the timestamp on maildirsize indicated that it's at least 15
minutes old, throw out the totals, and recalculate maildirsize
from scratch.
Eventually the 5120 byte limitation will always cause maildirsize to
be recalculated, which will compensate for any race conditions which
previously threw off the totals. Each time a message is delivered or
removed from a Maildir++, one line is added to maildirsize (this is
described below in greater detail). Most messages are less than 10K
long, so each line appended to maildirsize will be either between
seven and nine bytes long (four bytes for message count, space, digit
1, newline, optional minus sign in front of both counts if the message
was removed). This results in about 640 Maildir++ operations before a
recalculation is forced. Since most messages are added once and
removed once from a Maildir, expect recalculation to happen
approximately every 320 messages, keeping the overhead of a
recalculation to a minimum. Even if most messages include large
attachments, most attachments are less than 100K long, which brings
down the average recalculation frequency to about 150 messages.
Also, the effect of having non-Maildir++ clients accessing the
Maildir++ is reduced by forcing a recalculation when we're potentially
over quota. Even if non-Maildir++ clients are used to remove messages
from the Maildir, the fact that the Maildir++ is still over quota will
be verified every 15 minutes.
Delivering to a Maildir++
Delivering to a Maildir++ is like delivering to a Maildir, with the
following exceptions:
1. Follow the usual Maildir conventions for naming the filename used
to store the message, except that append ,S=nnnnn to the name of
the file, where nnnnn is the size of the file. This eliminates the
need to stat() most messages when calculating the quota. If the
size of the message is not known at the beginning, append ,S=nnnnn
when renaming the message from tmp to new.
2. As soon as the size of the message is known (hopefully before it
is written into tmp), calculate Maildir++'s quota, using the
procedure defined previously. If the message is over quota, back
out, cleaning up anything that was created in tmp.
3. If a file descriptor to maildirsize was opened for us, after
moving the file from tmp to new append a line to the file
containing the message size, and "1".
Reading from a Maildir++
Maildir++ readers should mind the following additional tasks:
1. Make sure to create the maildirfolder file in any new folders
created within the Maildir++.
2. When moving a message to the Trash folder, append a line to
maildirsize, containing a negative message size and a '-1'.
3. When moving a message from the Trash folder, follow the steps
described in "Delivering to Maildir++", as far as quota logic
goes. That is, refuse to move messages out of Trash if the
Maildir++ is over quota.
4. Moving a message between other folders carries no additional
requirements.
Index: Ext-mbx-locking
====================================================================
UNIX Advisory File Locking Implications on c-client
Mark Crispin, 28 November 1995
THIS DOCUMENT HAS BEEN UPDATED TO REFLECT THE CODE IN THE
IMAP-4 TOOLKIT AS OF NOVEMBER 28, 1995. SOME STATEMENTS
IN THIS DOCUMENT DO NOT APPLY TO EARLIER VERSIONS OF THE
IMAP TOOLKIT.
INTRODUCTION
Advisory locking is a mechanism by which cooperating processes
can signal to each other their usage of a resource and whether or not
that usage is critical. It is not a mechanism to protect against
processes which do not cooperate in the locking.
The most basic form of locking involves a counter. This counter
is -1 when the resource is available. If a process wants the lock, it
executes an atomic increment-and-test-if-zero. If the value is zero,
the process has the lock and can execute the critical code that needs
exclusive usage of a resource. When it is finished, it sets the lock
back to -1. In C terms:
while (++lock) /* try to get lock */
invoke_other_threads (); /* failed, try again */
.
. /* critical code here */
.
lock = -1; /* release lock */
This particular form of locking appears most commonly in
multi-threaded applications such as operating system kernels. It
makes several presumptions:
(1) it is alright to keep testing the lock (no overflow)
(2) the critical resource is single-access only
(3) there is shared writeable memory between the two threads
(4) the threads can be trusted to release the lock when finished
In applications programming on multi-user systems, most commonly
the other threads are in an entirely different process, which may even
be logged in as a different user. Few operating systems offer shared
writeable memory between such processes.
A means of communicating this is by use of a file with a mutually
agreed upon name. A binary semaphore can be passed by means of the
existance or non-existance of that file, provided that there is an
atomic means to create a file if and only if that file does not exist.
In C terms:
/* try to get lock */
while ((fd = open ("lockfile",O_WRONLY|O_CREAT|O_EXCL,0666)) < 0)
sleep (1); /* failed, try again */
close (fd); /* got the lock */
.
. /* critical code here */
.
unlink ("lockfile"); /* release lock */
This form of locking makes fewer presumptions, but it still is
guilty of presumptions (2) and (4) above. Presumption (2) limits the
ability to have processes sharing a resource in a non-conflicting
fashion (e.g. reading from a file). Presumption (4) leads to
deadlocks should the process crash while it has a resource locked.
Most modern operating systems provide a resource locking system
call that has none of these presumptions. In particular, a mechanism
is provided for identifying shared locks as opposed to exclusive
locks. A shared lock permits other processes to obtain a shared lock,
but denies exclusive locks. In other words:
current state want shared want exclusive
------------- ----------- --------------
unlocked YES YES
locked shared YES NO
locked exclusive NO NO
Furthermore, the operating system automatically relinquishes all
locks held by that process when it terminates.
A useful operation is the ability to upgrade a shared lock to
exclusive (provided there are no other shared users of the lock) and
to downgrade an exclusive lock to shared. It is important that at no
time is the lock ever removed; a process upgrading to exclusive must
not relenquish its shared lock.
Most commonly, the resources being locked are files. Shared
locks are particularly important with files; multiple simultaneous
processes can read from a file, but only one can safely write at a
time. Some writes may be safer than others; an append to the end of
the file is safer than changing existing file data. In turn, changing
a file record in place is safer than rewriting the file with an
entirely different structure.
FILE LOCKING ON UNIX
In the oldest versions of UNIX, the use of a semaphore lockfile
was the only available form of locking. Advisory locking system calls
were not added to UNIX until after the BSD vs. System V split. Both
of these system calls deal with file resources only.
Most systems only have one or the other form of locking. AIX
emulates the BSD form of locking as a jacket into the System V form.
Ultrix and OSF/1 implement both forms.
?
BSD
BSD added the flock() system call. It offers capabilities to
acquire shared lock, acquire exclusive lock, and unlock. Optionally,
the process can request an immediate error return instead of blocking
when the lock is unavailable.
FLOCK() BUGS
flock() advertises that it permits upgrading of shared locks to
exclusive and downgrading of exclusive locks to shared, but it does so
by releasing the former lock and then trying to acquire the new lock.
This creates a window of vulnerability in which another process can
grab the exclusive lock. Therefore, this capability is not useful,
although many programmers have been deluded by incautious reading of
the flock() man page to believe otherwise. This problem can be
programmed around, once the programmer is aware of it.
flock() always returns as if it succeeded on NFS files, when in
fact it is a no-op. There is no way around this.
Leaving aside these two problems, flock() works remarkably well,
and has shown itself to be robust and trustworthy.
?
SYSTEM V/POSIX
System V added new functions to the fnctl() system call, and a
simple interface through the lockf() subroutine. This was
subsequently included in POSIX. Both offer the facility to apply the
lock to a particular region of the file instead of to the entire file.
lockf() only supports exclusive locks, and calls fcntl() internally;
hence it won't be discussed further.
Functionally, fcntl() locking is a superset of flock(); it is
possible to implement a flock() emulator using fcntl(), with one minor
exception: it is not possible to acquire an exclusive lock if the file
is not open for write.
The fcntl() locking functions are: query lock station of a file
region, lock/unlock a region, and lock/unlock a region and block until
have the lock. The locks may be shared or exclusive. By means of the
statd and lockd daemons, fcntl() locking is available on NFS files.
When statd is started at system boot, it reads its /etc/state
file (which contains the number of times it has been invoked) and
/etc/sm directory (which contains a list of all remote sites which are
client or server locking with this site), and notifies the statd on
each of these systems that it has been restarted. Each statd then
notifies the local lockd of the restart of that system.
lockd receives fcntl() requests for NFS files. It communicates
with the lockd at the server and requests it to apply the lock, and
with the statd to request it for notification when the server goes
down. It blocks until all these requests are completed.
There is quite a mythos about fcntl() locking.
One religion holds that fcntl() locking is the best thing since
sliced bread, and that programs which use flock() should be converted
to fcntl() so that NFS locking will work. However, as noted above,
very few systems support both calls, so such an exercise is pointless
except on Ultrix and OSF/1.
Another religion, which I adhere to, has the opposite viewpoint.
FCNTL() BUGS
For all of the hairy code to do individual section locking of a
file, it's clear that the designers of fcntl() locking never
considered some very basic locking operations. It's as if all they
knew about locking they got out of some CS textbook with not
investigation of real-world needs.
It is not possible to acquire an exclusive lock unless the file
is open for write. You could have append with shared read, and thus
you could have a case in which a read-only access may need to go
exclusive. This problem can be programmed around once the programmer
is aware of it.
If the file is opened on another file designator in the same
process, the file is unlocked even if no attempt is made to do any
form of locking on the second designator. This is a very bad bug. It
means that an application must keep track of all the files that it has
opened and locked.
If there is no statd/lockd on the NFS server, fcntl() will hang
forever waiting for them to appear. This is a bad bug. It means that
any attempt to lock on a server that doesn't run these daemons will
hang. There is no way for an application to request flock() style
``try to lock, but no-op if the mechanism ain't there''.
There is a rumor to the effect that fcntl() will hang forever on
local files too if there is no local statd/lockd. These daemons are
running on mailer.u, although they appear not to have much CPU time.
A useful experiment would be to kill them and see if imapd is affected
in any way, but I decline to do so without an OK from UCS! ;-) If
killing statd/lockd can be done without breaking fcntl() on local
files, this would become one of the primary means of dealing with this
problem.
The statd and lockd daemons have quite a reputation for extreme
fragility. There have been numerous reports about the locking
mechanism being wedged on a systemwide or even clusterwide basis,
requiring a reboot to clear. It is rumored that this wedge, once it
happens, also blocks local locking. Presumably killing and restarting
statd would suffice to clear the wedge, but I haven't verified this.
There appears to be a limit to how many locks may be in use at a
time on the system, although the documentation only mentions it in
passing. On some of their systems, UCS has increased lockd's ``size
of the socket buffer'', whatever that means.
?
C-CLIENT USAGE
c-client uses flock(). On System V systems, flock() is simulated
by an emulator that calls fcntl(). This emulator is provided by some
systems (e.g. AIX), or uses c-client's flock.c module.
BEZERK AND MMDF
Locking in the traditional UNIX formats was largely dictated by
the status quo in other applications; however, additional protection
is added against inadvertantly running multiple instances of a
c-client application on the same mail file.
(1) c-client attempts to create a .lock file (mail file name with
``.lock'' appended) whenever it reads from, or writes to, the mail
file. This is an exclusive lock, and is held only for short periods
of time while c-client is actually doing the I/O. There is a 5-minute
timeout for this lock, after which it is broken on the presumption
that it is a stale lock. If it can not create the .lock file due to
an EACCES (protection failure) error, it once silently proceeded
without this lock; this was for systems which protect /usr/spool/mail
from unprivileged processes creating files. Today, c-client reports
an error unless it is built otherwise. The purpose of this lock is to
prevent against unfavorable interactions with mail delivery.
(2) c-client applies a shared flock() to the mail file whenever
it reads from the mail file, and an exclusive flock() whenever it
writes to the mail file. This lock is freed as soon as it finishes
reading. The purpose of this lock is to prevent against unfavorable
interactions with mail delivery.
(3) c-client applies an exclusive flock() to a file on /tmp
(whose name represents the device and inode number of the file) when
it opens the mail file. This lock is maintained throughout the
session, although c-client has a feature (called ``kiss of death'')
which permits c-client to forcibly and irreversibly seize the lock
from a cooperating c-client application that surrenders the lock on
demand. The purpose of this lock is to prevent against unfavorable
interactions with other instances of c-client (rewriting the mail
file).
Mail delivery daemons use lock (1), (2), or both. Lock (1) works
over NFS; lock (2) is the only one that works on sites that protect
/usr/spool/mail against unprivileged file creation. Prudent mail
delivery daemons use both forms of locking, and of course so does
c-client.
If only lock (2) is used, then multiple processes can read from
the mail file simultaneously, although in real life this doesn't
really change things. The normal state of locks (1) and (2) is
unlocked except for very brief periods.
TENEX AND MTX
The design of the locking mechanism of these formats was
motivated by a design to enable multiple simultaneous read/write
access. It is almost the reverse of how locking works with
bezerk/mmdf.
(1) c-client applies a shared flock() to the mail file when it
opens the mail file. It upgrades this lock to exclusive whenever it
tries to expunge the mail file. Because of the flock() bug that
upgrading a lock actually releases it, it will not do so until it has
acquired an exclusive lock (2) first. The purpose of this lock is to
prevent against expunge taking place while some other c-client has the
mail file open (and thus knows where all the messages are).
(2) c-client applies a shared flock() to a file on /tmp (whose
name represents the device and inode number of the file) when it
parses the mail file. It applies an exclusive flock() to this file
when it appends new mail to the mail file, as well as before it
attempts to upgrade lock (1) to exclusive. The purpose of this lock
is to prevent against data being appended while some other c-client is
parsing mail in the file (to prevent reading of incomplete messages).
It also protects against the lock-releasing timing race on lock (1).
?
OBSERVATIONS
In a perfect world, locking works. You are protected against
unfavorable interactions with the mailer and against your own mistake
by running more than one instance of your mail reader. In tenex/mtx
formats, you have the additional benefit that multiple simultaneous
read/write access works, with the sole restriction being that you
can't expunge if there are any sharers of the mail file.
If the mail file is NFS-mounted, then flock() locking is a silent
no-op. This is the way BSD implements flock(), and c-client's
emulation of flock() through fcntl() tests for NFS files and
duplicates this functionality. There is no locking protection for
tenex/mtx mail files at all, and only protection against the mailer
for bezerk/mmdf mail files. This has been the accepted state of
affairs on UNIX for many sad years.
If you can not create .lock files, it should not affect locking,
since the flock() locks suffice for all protection. This is, however,
not true if the mailer does not check for flock() locking, or if the
the mail file is NFS-mounted.
What this means is that there is *no* locking protection at all
in the case of a client using an NFS-mounted /usr/spool/mail that does
not permit file creation by unprivileged programs. It is impossible,
under these circumstances, for an unprivileged program to do anything
about it. Worse, if EACCES errors on .lock file creation are no-op'ed
, the user won't even know about it. This is arguably a site
configuration error.
The problem with not being able to create .lock files exists on
System V as well, but the failure modes for flock() -- which is
implemented via fcntl() -- are different.
On System V, if the mail file is NFS-mounted and either the
client or the server lacks a functioning statd/lockd pair, then the
lock attempt would have hung forever if it weren't for the fact that
c-client tests for NFS and no-ops the flock() emulator in this case.
Systemwide or clusterwide failures of statd/lockd have been known to
occur which cause all locks in all processes to hang (including
local?). Without the special NFS test made by c-client, there would
be no way to request BSD-style no-op behavior, nor is there any way to
determine that this is happening other than the system being hung.
The additional locking introduced by c-client was shown to cause
much more stress on the System V locking mechanism than has
traditionally been placed upon it. If it was stressed too far, all
hell broke loose. Fortunately, this is now past history.
?
TRADEOFFS
c-client based applications have a reasonable chance of winning
as long as you don't use NFS for remote access to mail files. That's
what IMAP is for, after all. It is, however, very important to
realize that you can *not* use the lock-upgrade feature by itself
because it releases the lock as an interim step -- you need to have
lock-upgrading guarded by another lock.
If you have the misfortune of using System V, you are likely to
run into problems sooner or later having to do with statd/lockd. You
basically end up with one of three unsatisfactory choices:
1) Grit your teeth and live with it.
2) Try to make it work:
a) avoid NFS access so as not to stress statd/lockd.
b) try to understand the code in statd/lockd and hack it
to be more robust.
c) hunt out the system limit of locks, if there is one,
and increase it. Figure on at least two locks per
simultaneous imapd process and four locks per Pine
process. Better yet, make the limit be 10 times the
maximum number of processes.
d) increase the socket buffer (-S switch to lockd) if
it is offered. I don't know what this actually does,
but giving lockd more resources to do its work can't
hurt. Maybe.
3) Decide that it can't possibly work, and turn off the
fcntl() calls in your program.
4) If nuking statd/lockd can be done without breaking local
locking, then do so. This would make SVR4 have the same
limitations as BSD locking, with a couple of additional
bugs.
5) Check for NFS, and don't do the fcntl() in the NFS case.
This is what c-client does.
Note that if you are going to use NFS to access files on a server
which does not have statd/lockd running, your only choice is (3), (4),
or (5). Here again, IMAP can bail you out.
These problems aren't unique to c-client applications; they have
also been reported with Elm, Mediamail, and other email tools.
Of the other two SVR4 locking bugs:
Programmer awareness is necessary to deal with the bug that you
can not get an exclusive lock unless the file is open for write. I
believe that c-client has fixed all of these cases.
The problem about opening a second designator smashing any
current locks on the file has not been addressed satisfactorily yet.
This is not an easy problem to deal with, especially in c-client which
really doesn't know what other files/streams may be open by Pine.
Aren't you so happy that you bought an System V system?
Index: LongTermIssues
====================================================================
$Cambridge: exim/exim-doc/doc-misc/LongTermIssues,v 1.1 2004/10/08 10:38:47 ph10 Exp $
Exim Long Term Issues
---------------------
I restarted this list from scratch for Exim 4. I amalgamated it with another
list when creating the CVS repository (October 2004). But it still probably
needs a substantial spring clean. Some of it is very old now.
AUTOCONF
--------
Somebody once tried to \(autoconf)\ Exim, but found it too big a job. I now
have some experience with using \(autoconf)\ for PCRE, and I think maybe some
use could be made of it. I don't, however, believe that \(all)\ Exim build-time
configuration should be done that way. The reason is that, unlike something
like PCRE, there is quite a lot of information that is "user choice". Giving it
all as options to a \(configure)\ command does not seem the best way of doing
things.
Whenever I build something that needs more than a couple of obvious options to
\(configure)\, I always save them in a file anyway, so I know what I did for
next time. Therefore, I think it is sensible to retain the current Local file
structure for all the user choice configuration.
However, it might be helpful to use \(autoconf)\ to dig out various bits of
information about the operating system. At present, the \(OS/Makefile-*)\ files
have hard-wired settings, and maybe this information could be figured out by
running \(autoconf)\, which would save having to keep maintaining these files.
I would arrange things so that \(configure)\ is run automatically the first
time that \(make)\ is run, but it would be possible to run it manually first,
to override defaults. (For example, if you have both \(cc)\ and \(gcc)\
installed on your system, as I do, you need to be able to specify which to
use.) I will need to do some experiments to see exactly how this would work.
EXIMON and other utilities
--------------------------
. Consider optionally making it possible to link with something other than
Athena widgets - for example, gtk. Or indeed re-write the whole thing!
GENERAL
-------
. Convert os.c into a directory of separate functions, with the macro
switches defined elsewhere. Then make it into a library.
. Use a pointer to an address structure for expanding $domain etc, to make it
easier to save/restore this collection of variables. But note that $domain
and $local_part aren't always in an address. Check out when these are set.
Note also the new $address_data possibility.
. Spool_in and spool_out - speed up by using a table?
. Find a more compact way of encoding the options interpretation, and also of
checking for incompatible options.
. Find a more compact way of passing an open SMTP channel without having
to use options. What about the TLS state information? Could use a pipe to
pass more data.
. Some people have suggested separately loadable modules. But do all systems
have them? Is this going too far for just a few specialist users? In
particular, people want to be able to replace the logging with his own code.
Can we arrange this without going for the separately loaded modules? (cf the
incoming checking code.)
. SIGHUP the daemon - don't close the sockets; instead pass a list of them
somewhere for the new daemon to pick up. Iff started by exim or root, of
course. There might be quite a long list of them - argv might not be the best
idea. If this were done, then a non-setuid exim daemon could be SIGHUPped.
. Parallel deliveries. Currently dead host information doesn't get propagated
between them very well. Is there anyway this could be improved?
. In some environments the use of gethostbyname() seems to cause problems.
Check out its use, and see if having a "force DNS" option could be helpful.
But people would have to know what they were doing.
. accept_max_per_host is a slow, linear search. If smtp_accept_max is large,
this can be very slow. Is there some way we can speed this up? Some kind of
index based on the IP address? Remember, this is in the daemon, so it must
not consume store.
. Change the names of all the pcre_ stuff to, say, PCRE_ so that Exim can be
linked with libraries or whatever that also use an external PCRE library.
. Look at code in pidentd for running Exim in wait mode from inetd and re-using
the socket. This would allow it to run more tidily as non-root.
. Think up some scheme for checking for orphan files in the spool directories.
Perhaps -bp should always do it, but it would be nice to have it done
automatically now and again. Maybe we just leave this for a cron job? Perhaps
a new -bx, e.g. -bpck or something. Better, perhaps, is a separate Perl
script. Orphan = a file that is over 24h old (or 1s when test harness) and
either doesn't end in -D or -H, or is a -D without a matching -H (or vice
versa).
. Make set_process_info buffer bigger, and put the overflowed message at the
end, thereby leaving the start.
. Swamping with delays in checking for reserved hosts - the connections are
counted in the total allowed. Can we improve on this somehow? Maybe shared
memory can help here. Think about different states and different limits.
. Lists that must use colons: can we check for other cases, and fix them up
before passing them on? Is it worth it?
. Linux for S/390 - create configuration?
. Process receiving error message fails - can we get more info, such as the
stdout/stderr?
. dbmbuild - if renaming one of .dir/.pag fails, reinstate the other. Should
there be a lock?
. Write a script to check for format problems in the source - formats that are
not fixed strings and are built from outside code.
. freeze_tell: Don't if message is a bounce message containing From: the local
machine - even if the bounce comes from another host.
. Add additional data into the "frozen" log message at end of delivery, e.g. if
remote host was the local host or whatever. At least some cross referencing.
. Someone had a requirement to install the Exim binary in a different place to
the utilities, etc. Also, for different builds on the same host and
architecture.
. Include (part of?) the ppid in the message id? Or a random number?
. Re-implement the code in readconf that reads error names for retry rules.
Make it use a table for most of the error types. Then see if we can usefully
add any additional error types.
. Should there be "exim -bP acls" etc? It would mean inventing some kind of
"hide" facility within the ACL syntax.
. VERY LONG TERM: the message ID is too small now, with the recent changes to
cram in the sub-second time. It would be a big project to extend it; Exim
would have to recognize both forms for a while, and become stable, before
generating the new form. Probably a runtime switch needed. The new form needs
at least microsecond time (or more?) and should probably cope with 64-bit
pids, just to be safe (or leave expansion space that could be used for that).
It should also be able to hold big enough things in base 36.
. Take a look at libexec.
. Sort out the stcncpy/strlcpy issue once and for all. Time things.
. Error in transport filter. See test 407. All 3 processes see errors - which
one should be noticed? Transport_filter_temp_errors may be needed.
. Think about 5xx thresholds -- too many and you're out. What about 4xx?
. autoreply - should it call /usr/sbin/sendmail? Provide a way of not passing
-C and -D when creating the message ('cause it won't be privileged).
. Strings containing \000 - anything we can do?
. OpenSSL - can we pass an opened file for certificate? Repeatedly?
Otherwise pre-initialize while root? There do seem to be functions for
manipulating certificates, but documentation is scarce. Can we just load the
certificate in as root in the server?
. Consider using poll() to close unwanted fds. Is this efficient? Perhaps it
doesn't matter for the daemon.
. On a 64-bit system there are some cast warnings for casting addresses to
ints. Either we must find a way of not warning, or we'll have to use unions
to get round it.
. Run splint on the source?
. It has been suggested that rejection because not authenticated should use
530 and not 550, but this is hard to detect because of the way ACLs work.
. When there is a sender verify failure, $acl_verify_message contains "sender
verify failed", not the details of the failure. Should this change? Some of
the waffly details are added later in smtp_in.c. In the ACL that text is in
sender_verified_failed->user_message.
. An empty string for a transport filter currently causes an error. Should it
ignore? Tricky because of special expansion rules for commands.
. GFDL for documentation (www.gnu.org/licenses/fdl.html)? The 1.2 version of
this licence is still quite new (it is dated November 2002) so I think
waiting for reaction/opinion is the best plan. There are Debian concerns
about this licence. At very least, no Invariant Sections and no Cover Texts
can be used.
. Allow $recipients in other places. Not clear what this value should be if,
say, the system filter has overridden them. Default would be envelope
recipients, as now.
End
Index: RFC.conform
====================================================================
$Cambridge: exim/exim-doc/doc-misc/RFC.conform,v 1.1 2004/10/08 10:38:47 ph10 Exp $
Conformance with RFCs
---------------------
Exim is written to follow the rules laid down in the RFCs. However, there are
some circumstances where it either extends what is specified, or chooses not to
follow them strictly, for various reasons. Sometimes variations are controlled
by an option, which may default on or off. This document lists the variations
from the latest email RFCs, and discusses their background and implications.
Last Updated: 25 January 1999
1. RFC 822
----------
The original specification of the format of Internet mail messages is RFC 822,
later clarified and modified by RFC 1123. At the time of writing (January 1999)
a new RFC (currently known as draft-ietf-drums-msg-fmt-07) which updates and
consolidates all the material related to the message format is at a late stage
of drafting, and is expected to become an Internet Standard in due course.
The following is (I hope) a complete list of major variations from the draft
RFC. References in square brackets are to the -07 draft.
1.1 Line termination [2.1, 2.3]
-------------------------------
[Lines are terminated by CRLF; isolated CR and LF are not permitted.]
The CRLF requirement has to be interpreted carefully, because the RFC also says
that it does not cover the internal format "used by sites". Exim keeps messages
on its spool in Unix format, using only LF as the line terminator, and also
does local deliveries using only LF. I believe this is compliant with the RFC,
as these are both "internal formats".
Messages sent out by SMTP have CRLF line terminators. However, isolated CR
characters are treated as any other data characters, because Exim is eight-bit
clean (see 1.2 below).
See 2.1 below for a discussion of line terminators in incoming messages.
1.2 Eight-bit characters [2.1]
------------------------------
[Messages consist of 7-bit characters.]
Exim is eight-bit clean. It does not do any processing of the characters in the
body of a message.
1.3 Maximum line length [2.1, 2.3]
----------------------------------
[The maximum length of a line is 998 characters.]
Exim does not enforce any limit on line length.
1.4 The "phrase" part of an address [3.4]
-----------------------------------------
[The phrase is a sequence of "words"; a word is an "atom" or a quoted string.]
The characters that can be used in an "atom" do not include the full stop
(dot, period). Thus a header line such as
To: John Q. Public <jqp@???>
is syntactically invalid under a strict interpretation of the RFC because the
dot in the phrase part is not quoted. However, many MTAs do not enforce this
restriction, so Exim was changed to be relaxed about it as well. In fact, the
draft RFC is moving towards allowing this. In section [4.1], which is defining
"obsolete" syntax that programs must accept (but not generate), it says this:
The period character is added to obs-phrase.
Note: The period character in obs-phrase is not a form that was allowed
in earlier versions of this or any other standard. Period (nor any other
character from specials) was not allowed in phrase because it introduced
a parsing difficulty distinguishing between phrases and portions of an
addr-spec (see section 4.4). It appears here because the period
character is currently used in many messages in the display-name portion
of addresses, especially for initials in names, and therefore must be
interpreted properly. In the future, period may appear in the regular
syntax of phrase.
1.5 Source routed addresses [4.4]
---------------------------------
[Source routed addresses are always enclosed in <>.]
Source routed addresses are declared obsolete in the draft RFC, but MTAs are
still required to handle them. Strictly, a source-routed address must be
enclosed in <> characters, so a header such as
From: @a,@b:c@d
is syntactally invalid. Exim does not enforce this restriction.
1.6 Local parts [3.4.1]
-----------------------
[Dots in unquoted local parts may not be consecutive or at either end.]
Exim allows unquoted local parts to begin or end with a dot (period, full
stop), and it also permits two consecutive dots in a local part.
2. RFC 821
----------
The original specification of SMTP is RFC 821, later clarified and modified by
RFC 1123. Domain name system requirements and their implications for mail are
covered in RFCs 1035 and 974. A scheme for extending the SMTP protocol is
described in RFC 1869, and there are subsequent RFCs specifying particular
extensions.
At the time of writing (January 1999) a new RFC (currently known as
draft-ietf-drums-smtpupd-09) which updates and consolidates all the material
connected with SMTP message transmission is at a late stage of drafting, and is
expected to become an Internet Standard in due course.
The new draft is written using the terms MUST, SHOULD, and MAY, which, when
written in capital letters, have precise meanings. To quote from the draft:
"MUST" or "MUST NOT" identify absolute requirements for conformance to
this specification. Implementations that do not conform to them lie
outside the scope of this specification and often will not
interoperate properly with SMTP implementations that do conform.
Implementations that are fully conforming also adhere to all "SHOULD"
and "SHOULD NOT" requirements. Implementations that adhere to all
"MUST" ("MUST NOT") but not to all of these are considered to be
partially conforming. Such implementations may interoperate properly
with fully conforming ones and with each other, but this will
typically be the case only if great care is taken. Consequently, an
implementation should violate "SHOULD" ("SHOULD NOT") requirements
only under exceptional and well-understood circumstances.
The implementation of Exim is intended to conform to the spirit of this
paragraph. The following is (I hope) a complete list of major variations
from the draft RFC. In addition to the items listed here, there are other minor
extensions such as the tolerance of white space in places where it is not
strictly permitted by the RFC. References in square brackets are to the -09
draft sections, and brief summaries of the RFC requirement are also given in
square brackets.
2.1 Line termination [2.3.7, 4.1.1.4]
-------------------------------------
[SMTP lines are terminated by CRLF.]
Exim recognizes LF without CR as a line terminator in all forms of input. For
SMTP input, any preceding CR is discarded. An early version of Exim followed
the RFC strictly, and did not recognize LF without CR in SMTP input. However,
it seems that sites on the net send out messages with just LF terminators,
despite the warnings in the RFCs, and other MTAs handle this, so Exim was
changed. However, there is a compile time macro called STRICT_CRLF which can be
set to restore the strict behaviour, though this is undocumented.
2.2 Eight-bit characters [2.4.1]
--------------------------------
[SMTP transmits only 7-bit characters.]
Exim is eight-bit clean, and makes no attempt to modify the data in a message
in any way. In particular, for messages containing characters with the top bit
set, it neither tries to negotiate 8-bit transmission, nor converts such
characters into an encoded form. In other words, it adopts the "just send 8"
strategy. It can be configured to send out 8BITMIME in its response to EHLO
(which it does not do by default), and it recognizes the 8BITMIME keyword on
incoming messages, but neither of these affect its handling of message data.
"Just send 8" is the strategy of a number of MTAs; it is argued that it
achieves what the user wants more often than other strategies.
2.3 Use of EHLO/HELO [3.2]
--------------------------
[Client MTAs should always start with EHLO, not HELO.]
Exim sends EHLO only when it finds the string "ESMTP" in an SMTP greeting
message. If EHLO is refused with a 5xx return code, it then reverts to HELO as
required, but it does not contain logic for converting to HELO on other errors
such as loss of connection or timeout after EHLO. That is one reason why it
doesn't always send EHLO; there are reported to be ancient SMTP servers out
there which collapse on receiving EHLO. (There is also at least one server
whose banner reads "<host name> ignores ESMTP", but it is RFC 821 compliant in
that it responds with 5O0 to EHLO, so Exim successfully reverts to HELO.)
2.4 Closing the connection [4.1.1.10]
-------------------------------------
[Client must wait for response to QUIT before closing the connection.]
Exim closes the connection immediately after sending QUIT, without waiting for
the reply. There was a lot of discussion about this on one of the mailing
lists. The conclusion was that this behaviour is fine on Unix systems, which
have TCP/IP implementations that close down the underlying channel tidily even
when the associated process has terminated. Indeed, not waiting may be
beneficial, as it moves the TIME_WAIT state (waiting to ensure there's no more
data in transit) from the server to the client system. On some other operating
systems (I understand) it is a disaster to terminate the sending process
without waiting for the QUIT response, because all the data about the
connection lives in the client's process space, and is therefore thrown away
before the response arrives. The subsequent arrival of the response then causes
bad behaviour.
2.5 IPv6 address literals [4.1.2]
---------------------------------
[IPv6 address literals are introduced by "IPv6".]
Exim recognizes IPv6 literals as just the colon-separated hexadecimal form of
an IPv6 address, for example 1080:0:0:0:8:800:200C:417A, without the need for a
prefix. At present, it does not even recognize the prefix. When IPv6 becomes
more widespread, Exim will follow whatever the common usage is.
2.6 Underscores in domain names [4.1.2]
---------------------------------------
[Underscores are not legal in domain names.]
RFC 822 allows all characters except specials, space, and controls in domain
names, but the SMTP RFCs are stricter, allowing only letters, digits, and
hyphen. Exim is compliant when checking incoming addresses in SMTP commands,
but it is more relaxed by default when checking domain names that are supplied
by EHLO or HELO commands, because many client workstations get set up with
underscores in their names. There is an option that can be set to cause Exim to
refuse underscores. (There are also options to specify certain hosts from which
it will accept any old junk after EHLO or HELO. Such is the woeful state of
some SMTP clients.)
2.7 Removal of return-path headers [4.4]
----------------------------------------
[Relaying MTAs should not remove return-path.]
Exim removes Return-Path: headers from all messages, if return_path_remove is
set (the default). It does not attempt to determine if it is being a relay or
not. Indeed, for some messages it might be both a relay and a final destination
MTA for the same message.
2.8 Randomizing the order of addresses of multihomed hosts [5]
--------------------------------------------------------------
[Multihomed host addresses should not be randomized.]
Exim does randomize a list of several addresses for a single host, because
caching in resolvers will defeat the round-robinning that many namerservers
use. (Note: this is not the same as randomizing equal-valued MX records. That
is required by the RFC.)
2.9 Handling "MX points to self" [5]
------------------------------------
[MX points to self must be treated as an error.]
The RFC doesn't allow for the possibility of special-purpose routing in the
case when the lowest numbered MX record points to the local host. The default
Exim configuration is compliant, but it is possible to configure Exim to behave
differently, and there are several situations where this can be useful.
2.10 Source routing [6.1]
-------------------------
[Source routes should be stripped.]
The new RFC has moved forward in deprecating source-routed email addresses.
Exim does not strip them down by default, but can be made to do so by setting
collapse_source_routes. However, even when it is not stripping them down, it
does not add host routing to reverse-paths when processing a source-routed
forward-path.
2.11 Loop detection [6.2]
-------------------------
[Loop count for Received: headers should be at least 100.]
Exim's default setting of the received_headers_max option is 30. Most messages
these days seem to accumulate less than half a dozen Received: headers, and
even a couple of forwardings don't bring this anywhere near 30.
2.12 Addition of missing headers [6.3]
--------------------------------------
[Missing headers may be added, and domains qualified, only if client is
identified.]
Exim always adds Message-Id: and Date: headers if these are missing, whatever
the source of the message, and likewise when it expands non-fully-qualified
domains, it does so independently of the message's source.
2.13 Syntax of MAIL and RCPT commands [4.1.1.2, 4.1.1.3]
--------------------------------------------------------
Exim is more relaxed than the RFC requires:
(1) Trailing white space is ignored.
(2) It permits white space after the "FROM" and "TO" keywords.
(3) It does not insist on the address being enclosed in <> characters. In fact,
it recognizes addresses in RFC 822 format here, except that domain
components are restricted to containing only letters, digits, and hyphens.
(4) Local parts are permitted to contain null components, that is, may start or
end with an unquoted full stop (period) or contain two consecutive
unquoted full stops.
2.14 Non-fully-qualified domains [2.3.5]
----------------------------------------
[All domains must be fully qualified.]
A domain that is not fully qualified has some of its trailing components
missing, and is normally a local alias of some sort, for example, just a
single-component host name.
Exim can be configured to "widen" non-fully-qualified domains, either by using
the facilities of the DNS resolver, or by an explicit list of widening strings.
When this is done, it applies to addresses received by SMTP from other hosts,
as well as to locally-originated addresses. Address re-writing could also be
used for this purpose.
2.15 Unqualified addresses [4.1.2]
----------------------------------
[Addresses in SMTP commands must include domains.]
An unqualified address consists of a local part without a domain. Do not
confuse "qualified address" and "qualified domain". A qualified address may
include a non-fully-qualified domain.
There is one exception to the RFC rule: it is required that the unqualified
address "<postmaster>" always be accepted. Apart from this, Exim rejects
domainless addresses in SMTP commands by default, but it can be configured with
a list of hosts and/or networks that are permitted to send addresses without
domains in SMTP commands. Any such address that is accepted (including
<postmaster>) is qualified by adding the value of the qualify_domain option.
2.16 VRFY and EXPN [3.5.1, 3.5.2, 3.5.3, 7.3]
---------------------------------------------
[VRFY and EXPN should be supported.]
Exim does not support VRFY and EXPN by default, but a list of hosts and
networks for which they are permitted can be given.
2.17 Checking of EHLO/HELO commands [4.1.4]
-------------------------------------------
[Client must send EHLO. Server must not refuse message if EHLO/HELO check
fails.]
Exim, as a client, always sends EHLO or HELO (see 2.3 above). As a server, it
does not insist on there having been a valid EHLO or HELO command before the
start of a message transaction. Any EHLO or HELO command that is received is
rejected only if it contains a syntax error. That is, it is never rejected on
the basis of any validation checking that may be performed on the data it
contains.
However, Exim can be configured to insist that (a) there is valid EHLO/HELO
command before any message transaction and (b) the domain in that command
matches the domain obtained by looking up the IP address of the sending host.
It is possible to specify exception lists of hosts and/or networks for which
this check does not apply.
2.18 Format of delivery error messages [3.7]
--------------------------------------------
[Standard report formats should be used if possible.]
Exim's delivery failure reports do not conform to the format described in RFC
1894.
## End ##
Index: TexiNotes
====================================================================
$Cambridge: exim/exim-doc/doc-misc/TexiNotes,v 1.1 2004/10/08 10:38:47 ph10 Exp $
Notes for conversion of sgcal input into Texinfo input
------------------------------------------------------
(Dated 6 August 1996)
The escape character is @. Only @ and curly brackets are sensitive. Get them in
by @@ @{ and @} if required.
@: after a dot that is not a sentence end.
@. instead of . if sentence ends with capital letter
@copyright{} for copyright
@minus{} is a slighly longer minus sign
Input file ends with .texinfo usually.
MUST start the file with
\input texinfo
@c %**start of header
@setfilename INFO-FILE-NAME
@settitle NAME_OF_MANUAL
$c %**end of header
Then, typically
@ifinfo
summary and copyright
@end ifinfo
Followed by
@titlepage
title and copyright
@end titlepage
Then the top node and master menu - for info file only
@node Top, First Chapter, (dir), (dir)
@comment node-name next, previous, up
@top
@menu
* First Chapter:: The first chapter is the
only chapter in the sample
* Concept Index:: An index
@end menu
Then the body
@node First Chapter, Concept Index, Top, Top
@comment node-name next, previous, up
@chapter First Chapter
@cindex Sample index entry
This is the contents of the first chapter
@cindex Another sample index
Then stuff about indexes and tables of contents
@node Concept Index, , First Chapter, Top
@unnumbered Concept Index
@printindex cp
@contents
MUST end the file with
@bye
. NEWLINE AND NO-FILL MODE
@page for new page
@* forces a line break
. LINE CENTERING
@center stuff
. ROMAN, ITALIC, BOLD ITALIC, SMALL CAPS
@code{...} for 'code' => `...' in info
@file{...} for file names => `...' in info
@samp{...} for sample text => `...' in info
@var{...} for variable => caps in info
@dfn{...} defining a term => double quotes in info
@emph{...} produces italic
@strong{...} produces bold
@sc{...} small caps but with letters in lower case.
@i italic )
@b bold ) no effect on info file
@r roman )
. TABBING
. CHAPTERS & SECTIONS
@chapter <title>
@unnumbered <title> is an unnumbered chapter
@section
. SECTION
. FANCY VS PLAIN
@iftex ... @end iftex for printed only; likewise @ifinfo ... @end ifinfo
. LEAVING BLANK SPACE
@sp 10
. EM & NEM
no can no
. DISPLAY ASIS
@example ... @end example
@display ... @end display no change of font => rm
. COMMENTS
@comment or @c introduces comment lines
. NUMBERED LISTS
@enumerate
@item
first item
@item
second
@end enumerate
. BULLETED LISTS
@itemize @bullet
...
. CROSS REFERENCES
@xref start sentence
@ref{name}
@pxref (parenthesized)
5 args: node name (required), cross-ref name, topic description, name of
info file, name of printed manual.
. TABLES
@table for two-column tables
@table @asis
@item first column
second column
@item ...
. INDEX
@cindex concept index
@findex function index
@vindex variable index
@kindex key index
@pindex program index
@tindex data type index
***
Index: WishList
====================================================================
$Cambridge: exim/exim-doc/doc-misc/WishList,v 1.1 2004/10/08 10:38:47 ph10 Exp $
EXIM 4 WISH LIST
----------------
Even when it was first released, Exim 4 had a Wish List because not all the
things suggested for it were implemented. The list has not stopped growing...
Another reason it is so long is that I have retained some items from the Exim 3
Wish List that never got implemented, but which seem reasonable possibilities
for later addition to Exim 4.
I have guessed at the amount of work involved, and categorized the items as
Tiny, Small, Medium, Large, or Unknown. The guesses are not based on any
detailed investigation, so must be taken as very rough.
------------------------------------------------------------------------------
------------------------------------------------------------------------------
----- Retained from the Exim 3 Wish List ------
------------------------------------------------------------------------------
------------------------------------------------------------------------------
(10) 13-Jul-98 M more flexibility for pipe returns
Ben Smithurst
The ability to specify more precisely what happens concerning the return code
from the pipe and the presence/absence of STDOUT/STDERR is requested. The
particular configuration that was requested was:
> if the command exited EX_OK, *and* produced nothing on STDOUT or
> STDERR, it succeeded...
> if the command exited EX_TEMPFAIL, defer, regardless of
> STDOUT/STDERR...
> otherwise freeze the message (this will get my attention by way of
> freeze_tell_mailmaster)...
------------------------------------------------------------------------------
(11) 17-Jul-98 G support for DSN
Andy Mell
It is unclear to me how this should work in the presence of aliases and
forwarding. Local deliveries would have to explicitly configured as deliveries
or relaying or whatever. A substantial amount of code is probably needed.
Jeffrey Goldberg
I have nothing to add except to say that for many of the reasons you've
stated, I don't think that DSN is coherent enough to be worth the effort
to implement.
Another comment:
I thought the RFC was pretty clear on this. In a nutshell, if the
delivery rewrites the envelope from address, it's considered a
terminal delivery (i.e. delivery to a mailing list exploder), otherwise
treat it as a forwarding operation (the /etc/aliases case). I would
treat a .forward expansion as a final delivery event (it got to the
user as far as the MTA is concerned).
Yes, we need the DSN syntax. We also require the complete semantics of
NOTIFY=SUCCESS,FAILURE for our application to work.
Electronic Bill Presentment is really going to push the need for
DSN support in MTAs. We just don't want to get stuck in a situation
where we're faced with a non-DSN-aware MTA when we go to install
our bill/statement engine, thus our interest in what the MTA vendors
are planning to do about DSN.
------------------------------------------------------------------------------
(41) 14-Oct-98 M Find a way of modifying header lines
Oliver Smith
The problem with header_remove followed by header_add is that you can't refer
to the previous value of the header when adding a replacement. This could be
solved with a replace_header option.
------------------------------------------------------------------------------
(43) 15-Oct-98 M Sender rewrite *after* SMTP incoming checks
Andreas Edler
The anti-relaying check happens after the sender has been rewritten; there are
times when it would be helpful to do the check on the original sender, not on
the rewritten one. Quite how to configure this I'm not sure.
A related suggestion (from Steve Sargent) is to retain the original sender
address and make it accessible somehow.
------------------------------------------------------------------------------
(46) 20-Oct-98 L SMTP protocol hooks
Malcolm Ray
"But there are enough broken SMTP implementations to make me wonder whether
there isn't a case for providing hooks for tweaking the SMTP transport's
protocol exchange. Something which would allow me to say things like 'if, when
talking to lame.example.com, you get a 251 response to a MAIL command, rewrite
the response to 501 before continuing'."
------------------------------------------------------------------------------
(50) 13-Nov-98 M A "Focus" option for eximon
Frank Elsner
This is the opposite of "Hide"; it just displays a certain subset. Hmm. Could
something clever be done with regular expressions?
------------------------------------------------------------------------------
(61) 22-Dec-98 M Send failed error messages to somebody
Harald Meland
With sendmail, the failed error message is made into a error message,
with both envelope sender and recipient set to MAILER-DAEMON. The
original, bogus-envelope-sender message is then available to whoever
receives MAILER-DAEMON's mail. A more flexible approach would be to
specify a specific recipient.
------------------------------------------------------------------------------
(81) 01-Mar-99 M Addition of Content-MD5 support
Martin Hamilton
Martin supplied a suggested patch at
http://www.net.lut.ac.uk/~martin/antispam/exim-hacks/
------------------------------------------------------------------------------
(85) 15-Mar-99 M ability to rewrite addresses in non-standard headers
Dave Lewney
John Holman
Such as "return-receipt-to". See also 41.
------------------------------------------------------------------------------
(90) 21-Apr-99 M change wild prefix/suffix greediness
Ben Smithurst
Currently, when prefix or suffix containing * is set on a director, and the
fixed part occurs more than once in a local part, the length of the prefix or
suffix is maximized. For example, with suffix = -* and a local part of
foo-bar-baz the suffix is taken as bar-baz, leaving the local part as foo.
An option is proposed to invert this rule.
------------------------------------------------------------------------------
(91) 26-Apr-99 S make queue_run_in_order to newest first
"Andreas M. Kirchwitz"
The tidiest thing would be to have queue_run_order={random,oldest,newest},
and make queue_run_in_order obsolete.
------------------------------------------------------------------------------
(93) 04-May-1999 L fallback_transport
This would be a generic transport option, specifying a different transport to
be used if the first one failed. Failed hard, or failed soft? Or an option?
And if failed hard, is a bounce message sent as well, or not? There are uid
issues. Remote delivery would have to be done always in a subprocess so that
the main process could retain privilege in case the fallback transport was
local. That could be conditional. That's why this is labelled "Large". Some of
the things people want to do with this can be done by variations in the
routers, e.g. use $message_age to switch routers.
------------------------------------------------------------------------------
(94) 13-May-1999 M message to go with -Mg
Dave Holland
Alan Thew
So the admin can pass back a reason.
------------------------------------------------------------------------------
(99) 28-May-1999 M header to list failures for syntax_errors_to
mark david mcCreary
"I use the syntax_errors_to feature to email a copy of the error message.
It would be helpful to have the X-Failed-Receipients header in there,
identifying which addreses(s) are the problem, so that I don't have to
parse the body of the email message to figure out which addresses."
------------------------------------------------------------------------------
(100) 04-Jun-1999 S admin_users option, like trusted_users
Paul Mansfield
------------------------------------------------------------------------------
(102) 21-Jun-1999 M expanded basic variables
Julian King
Oh, and a wishlist entry, qualify_domain, and preferably other variables
can be set with a $lookup in the first part of the exim configuration
file, perhaps by an equivalent to backticks in shell script ("`command`")?
------------------------------------------------------------------------------
(105) 28-Jun-1999 M MIME-format bounce messages
Paul Makepeace
"Is there any work going/gone on/planned to enable exim to report delivery
status notifications using RFC1892 multipart/report MIME messages? It would be
great to have errors reported in a message/rfc822 attachment."
Jeffrey Goldberg
"I like plain bounces, so would hope that if you do this, that it be
configurable. I think that even for those who want it, it shouldn't be very
high on the wish list priority."
Other suggestions: toggle for bounces/warnings; override max_return for
certain addresses; use plain text if original not MIME. See Paul's hack
for background of what to do.
Nigel suggests using a specially named autoreply transport to generate bounces;
people could then replace this with another transport (e.g. pipe) if they want
to customize it themselves.
Eli Chen posted an unconditional patch for 3.32 that does some of this work.
That could form a basis.
------------------------------------------------------------------------------
(107) 12-Jul-1999 S defer transport at given load level
Marc Haber
------------------------------------------------------------------------------
(108) 16-Jul-1999 S remote sort by numbers of recipients
mark david mcCreary
In the absence of remote_sort, sort remote domains by the number of recipients
in each.
------------------------------------------------------------------------------
(114) 11-Nov-1999 S List of possible outgoing interfaces
Allow the smtp "interface" option to be a list: try them in turn until one
is found to work. Also allow masks to specify a range of addresses.
------------------------------------------------------------------------------
(123) 23-Dec-1999 L Use AUTH + TURN for dial-in hosts
Andrew Tverdokhleb
The way to do this would be to have Exim deliver messages into per-host
directories in, say, BSMTP format. Accept TURN if authenticated, and cause it
to run a helper program that is passed the socket in order to deliver the mail.
Provide a helper program!
------------------------------------------------------------------------------
(125) 04-Jan-2000 L Use shared memory segment for queue list
Theo Schlossnagle
The idea is that a queue-runner that finds no existing shared segment should
create one (if configured - possibly some fixed size) and all Exim processes
should maintain a list of messages in it, thereby saving on directory scans
when there are lots of messages. This needs a lot of careful thought to try to
eliminate any possibility of data loss. The interlocking could be quite tricky.
Further posters suggested using a db file to hold the list. See also 127.
------------------------------------------------------------------------------
(129) 14-Jan-2000 L Dynamically loadable lookup modules
Steve Haslam
Suggested patch provided.
------------------------------------------------------------------------------
(131) 17-Jan-2000 T Facility for assuming existence for EACCES
Peter Radcliffe
The opposite option for "+" in require_files: assume existence if cannot
peer into the directory (+ assumes non-existence).
------------------------------------------------------------------------------
(131) 29-Feb-2000 M? Control total number of outgoing SMTP calls
Brian White
This is for hosts with slow connections. Could some modification of
serialize_hosts be used for this? Or maybe use a semaphore? They seem to
be quite widely available.
------------------------------------------------------------------------------
(132) 01-Mar-2000 S Lookup host name from outgoing interface
Vadim Vygonets
Instead of primary_hostname, look up the name for the interface that is being
used for sending. Suggested patch supplied, but this should be an option of the
smtp transport.
------------------------------------------------------------------------------
(133) 06-Mar-2000 S Filter option not to log "previously sent"
Bruce Bowler
This is when using the "log" option of the autoreply driver.
------------------------------------------------------------------------------
(134) 09-Mar-2000 S Option to remove attachments when bouncing
------------------------------------------------------------------------------
(136) 13-Mar-2000 S/M Option for aliasfile to suppress "me too"
Could be tricky determining who "me" is.
------------------------------------------------------------------------------
(143) 08-May-2000 S Make quota_warn_threshold into a list
David Carter
So several warnings could be generated as the mailbox got bigger and bigger.
------------------------------------------------------------------------------
(146) 15-May-2000 M Allow SMTP error codes in retry rules
This would allow special handling of certain errors from certain hosts. In
particular, it would allow failing of certain 4xx codes.
This is now available for 4xx responses to RCPT commands. Is anything more
needed?
------------------------------------------------------------------------------
(148) 15-May-2000 S Warn recipient if message rejected for quota excession.
Heinz Ekker
Maybe not all that small, because the possibility of retrying must be taken
into account.
------------------------------------------------------------------------------
(149) 19-May-2000 L Make added headers visible in filters and other places
Hans Morten Kind
Headers added by directors/routers are not visible in subsequent processing.
This is a request to make them visible. What about removed headers? This could
be tricky to specify, hence the L.
A separate but related issue is the effect of headers added by "unseen"
directors. These are documented in chapter 19 as not being accumulated. Should
any change be made?
------------------------------------------------------------------------------
(155) 16-Jun-2000 M Special handling for certain hosts
mark david mcCreary
A means of changing the transport depending on the host name/IP of the most
preferred MX record so that all domains that route to certain hosts can be
handled specially. Maybe this could be a variable that is available in the
expansion of the "transport" option.
------------------------------------------------------------------------------
(158) 29-Jun-2000 S Configure "From" in bounces
Ben Parker
Cf Reply-To.
------------------------------------------------------------------------------
(159) 07-Jul-2000 M Keep messages for fixed time
Gary Palmer
An option to keep messages on the queue for a specified time, even if all their
destination hosts have timed out.
------------------------------------------------------------------------------
(164) 17-Aug-2000 S sender_unqualified_auth_hosts
To allow authenticated hosts to send unqualified addresses. Presumably it
needs received_... as well.
------------------------------------------------------------------------------
(167) 05-Sep-2000 L Support for ODBC
This would allow access to databases that don't have native support built into
Exim. See
http://www.openlinksw.com/info/docs/rel3doc/unix/odbcsdk.htm
------------------------------------------------------------------------------
(168) 06-Sep-2000 M Deliver messages that alias to nothing to a given address
Dr ZP Han
If other people are managing alias lists, and one is empty, bounce that
delivery to a given address rather than freezing the message. Use the errors_to
address?
------------------------------------------------------------------------------
(172) 11-Sep-2000 S Allow file/directory in appendfile to override
"Michael J. Tubby"
When appendfile is called from forward or filter files, it ignores file or
directory settings. Maybe they should override. The path set by the forward or
filter is available in $address_file these days, so it could be used to create
a longer path.
------------------------------------------------------------------------------
(173) 18-Sep-2000 S A way of doing lsearches with EOL terminated keys
Jason Robertson
This is for looking up things like subject contents. Probably need an option to
exim_dbmbuild to make them into DBM files.
------------------------------------------------------------------------------
(174) 19-Sep-2000 S A way of using a different port for fallback hosts.
Dean Brooks
------------------------------------------------------------------------------
(181) 10-Nov-2000 S Compile-time options for ignoring Sendmail options
So that new ones could be accommodated easily.
------------------------------------------------------------------------------
(183) 04-Dec-2000 L dns_means_nonexist_after
Dave C.
In other words, wait a bit before giving up. This needs a mechanism for
remembering, which is not currently available. To be borne in mind for the
future.
------------------------------------------------------------------------------
(184) 04-Dec-2000 M Log more details of local caller
J. Nick Koston
"I was wondering if it was possible for exim to log the parent pid's cwd and
exe when it is called from a script/invoked by actually running /usr/sbin/exim
or /usr/sbin/sendmail." Question: is this information actually/easily
available to Exim? Needs investigation.
------------------------------------------------------------------------------
(186) 19-Dec-2000 S A simple utility to reset a retry time
Marc Haber
Basically, to do what exim_fixdb "delete" can do, but straightforwardly. There
could be an interface from eximon.
------------------------------------------------------------------------------
(187) 02-Jan-2001 M Wildcarding in headers_remove
Tamas TEVESZ
What I'd like to see is it to handle globs (or regexps, but i'm not sure this
latter would worth the hassle), in a way like:
headers_remove = "X-*:Additional-header"
------------------------------------------------------------------------------
(188) 02-Jan-2001 S Make pipe timeout a temporary error
Georg v.Zezschwitz
A way to make a timeout into a temporary error.
------------------------------------------------------------------------------
(190) 03-Jan-2001 M Multiple message operations in eximon
------------------------------------------------------------------------------
(195) 19-Mar-2001 T TCP window size
TCP window size for receiving/sending, SMTP client/server.
------------------------------------------------------------------------------
------------------------------------------------------------------------------
------------------------------------------------------------------------------
------------------------------------------------------------------------------
----- Things that didn't make it into Exim 4 ------
------------------------------------------------------------------------------
------------------------------------------------------------------------------
. An option to send messages to postmaster when ignore_errmsg_errors_after
times out.
. When an address is being routed, its constituents are in $local_part and
$domain, but there is currently no variable that contains the whole thing. It
could be put into $recipient, but that risks confusion with $recipients
(which is available in system filters). Maybe $address could be used?
. The ability to relay to host X without knowing all the domains that host X
might have. At ACL time, one would need to verify the recipient, and determine
that it routed to host X.
. A new lookup library that operates on a specially prepared file of IP
addresses and masks so that a single "lookup" yields a yes/no answer. This
should be a freestanding thing - needs a utility to build the file from a list.
. People want to change the wording of messages; can we find an efficient way
of allowing this? (Maybe put all messages into a separate module?) The problem
is not in the messages themselves, but in the values that get inserted into
messages. Would have to invent a new kind of function that used identified
values rather than positional ones. Use GNU gettext?
. Invent lf_hosts for those that may use LF without CR. Any other RFC
things we need to worry about?
. A user would really like to see something similar, perhaps with
"ID=$authenticated_id", similar to "helo=" and "ident=" in the default received
header. BUT there are security issues. Maybe give it as a commented out option
in the default configuration?
. Consider expanding further options that take integer values. What about
smtp_xxx options for different limits at different times of day (for example)?
What about tls_advertise_hosts (so can look at incoming IP/port)?
. How about a "hold hosts" option (cf hold_domains) to hold delivery to certain
hosts?
. Allow user filters to use "headers add", but probably not remove. Or maybe
just implement "allow" options for both of these features.
. Have the return from pipe in a variable, so that (e.g. error_message_file)
can make use of it.
. Implement randomize for ldap/sql servers.
. Add an option for ETRN that says "wait for the command to finish, and use its
stdout as the SMTP response."
. -odsomething for "ignore retry when doing immediate delivery".
. Add an option to the smtp transport to make it treat 5xx on connection as if
it were 4xx. Or possible add a sophisticated "after command X, treat xxx as
yyy".
. A way of rewriting addresses in non-standard header lines such as
Mail-Followup-To.
. Global option to enable initgroups() for exim uid. Default off.
. When verifying a sender, should it be rewritten with any T rewrites, because
it would be so rewritten if it actually was a recipient in a message?
. Sean Witham wants a way of defining macros that are not privileged, and a
sort of #ifdef structure that allows for different configurations in the same
file.
. Allow :fail: to specify that 551 be used instead of 550. Maybe allow a code
at the start, optionally? What about :defer:?
. SMTP timeout in middle of receiving message: log sender address if known, and
possibly message_id if known.
. Make -brw show rewrites for transports too.
. Have the MTA log destinations that have timed-out on a ident request and
no longer send rfc1413_queries to them. Add an option for how not to cache
these entries.
. Options and/or a utility to enable non-privileged users to view the queue
(e.g. -bpp), manipulate their own messages, etc.
. Specify a port along with a host in a route_list.
. A generalized "From" escaping scheme that also escapes >From so that the
whole thing can be reversed.
. There was a request for the \dns_again_means_nonexist\ option not to be
instantaneous, but to operate only after the DNS has been giving "try again"
for some time. Use the misc hints database.
------------------------------------------------------------------------------
------------------------------------------------------------------------------
----- The Exim 4 Wish List ------
------------------------------------------------------------------------------
------------------------------------------------------------------------------
(1) 01-Jan-02 U Use of dynamically loaded libraries.
People want Exim to use dynamically loaded modules for a variety of reasons.
When I started to create Exim, I never expected anything other than source
distribution; the RPMs and inclusions in OS distributions caught me by
surprise. I know very little about the mechanics of dynamic loading, but I'm
aware that not all operating systems support it. I'm also aware that not all
people support it!
Furthermore, a way round this might be to supply more hooks along the lines of
local_scan(). Then people can write their own dynamic loaders if they want.
------------------------------------------------------------------------------
(3) 01-Jan-02 U Test for over-quota at SMTP time
This is a hard one, because the only way to test for over quota is to try to
deliver a message, certainly if system quotas are being used. And also, the
only available size at RCPT time is the SIZE option, though of course the test
could be run at DATA time. I think maybe we leave this one to an external
program, and require people to use ${run} to access the data. Let someone else
figure out how to extract the current mailbox size!
One suggestion is to implement
${file_size:/path/to/file}
${directory_size:/path/to/directory}
so that explicit checks can be done. It may be necessary to have four
operators, two being based on the block count, and two showing the "visible"
size. Directory scanning is expensive; is there any scope for caching? It would
seem not (you don't often get two addresses to the same user).
------------------------------------------------------------------------------
(4) 01-Jan-02 S Option to reject if no From: or Date: header line
Exim, in common with many other MTAs, inserts a From: or Date: header line if
one is missing. (It also inserts a blank Bcc:, but that is no longer needed by
RFC 2822 - it was by 822.) The suggestion is an option to give an error
instead. This could be done by making it possible to detect these insertions in
the acl_smtp_data ACL.
------------------------------------------------------------------------------
(6) 01-Jan-02 S Option to disable the use of -t
Dave C.
Would require work so that Exim itself doesn't use -t.
------------------------------------------------------------------------------
(7) 01-Jan-02 M Avoid showing LDAP passwords in log lines for LDAP errors
John W Baxter
May be tricky, because at the higher levels, the format of the query is not
understood.
------------------------------------------------------------------------------
(8) 01-Jan-02 S Expand once_repeat in autoreply
John Jetmore
------------------------------------------------------------------------------
(9) 01-Jan-02 S Headers as well as body in file for autoreply
Florian Laws
------------------------------------------------------------------------------
(10) 01-Jan-02 T Make "true" and "false" valid expansion conditions
This might help with "and" and "or" when one of the sub-conditions is, for
example, a lookup.
------------------------------------------------------------------------------
(11) 01-Jan-02 S Allow a filter to include another file.
------------------------------------------------------------------------------
(12) 01-Jan-02 M Support for different SQL servers per query
In other words, the global mysql_servers etc. is too restrictive.
------------------------------------------------------------------------------
(14) 01-Jan-02 M? Support for Sendmail milters
This could perhaps be done by extending the local_scan() idea and providing a
"standard" module which interfaced to milter.
------------------------------------------------------------------------------
(15) 01-Jan-02 M More hooks like local_scan()
One request has been for a similar hook at logging time. For other SMTP
interactions, maybe a hook into the ACL? See also 79 and 218.
------------------------------------------------------------------------------
(17) 11-Jan-02 M The construction of config.h needs refactoring
This has been hacked about substantially since the original implementation.
Given that there is a program (buildconfig), the messing around with the
environment could be abolished. Also, the distinction between "yes" and "no"
isn't always properly made (tests for #ifdef don't care about the value).
------------------------------------------------------------------------------
(18) 24-Jan-02 S Make $value retain its value after a top-level expansion
This was specifically for use in filter files. Currently it reverts to empty
as a consequence of save/restore for every lookup. It might be confusing to
do otherwise, however.
------------------------------------------------------------------------------
(19) 29-Jan-02 L Use of multiple DBM libraries
The problem is how to handle conflicting function names. Much research is
needed.
------------------------------------------------------------------------------
(20) 29-Jan-02 S Make system filter refreeze after manual thaw
Currently, a "freeze" in a system filter doesn't freeze after a manual thaw.
------------------------------------------------------------------------------
(21) 12-Feb-02 S Expand return_size_limit
Joachim Wieland
Is this really worth it? A per-transport value is also suggested - that would
mean remembering the value with each failed address and taking a minimum or
a maximimum (which?).
------------------------------------------------------------------------------
(24) 21-Feb-02 ? A way of testing TLS using -bh
------------------------------------------------------------------------------
(27) 06-Mar-02 M Distinguishing between different temporary callout errors
The request was to distinguish between a 4xx error and a failure to connect.
Problem is: how to cope when there is more than one host? Maybe only if ALL
fail to connect. An option like /callout_no_connect_ok.
------------------------------------------------------------------------------
(30) 12-Mar-02 S Add "recipients" precondition to routers.
This would avoid having to use "condition". (See also requirement for $address
mentioned above.) However, it would also require adding a caching feature, and
probably $recipient_data (cf $domain_data).
------------------------------------------------------------------------------
(31) 21-Mar-02 S Variables that indicate 8-bit message and 8-bit host, and
a way of using them to suppress a transport filter
A variable that is set if the message contains 8-bit characters, and another
that is set during the smtp transport if the host supports 8-bit. Then we also
need a condition that's expanded in the transport to control whether the filter
is run or not (e.g. transport_filter_condition).
------------------------------------------------------------------------------
(32) 22-Mar-02 M More info about callout fails for header sender verify
When there's a callout failure for an envelope address, the error message
contains details (by default) of the callout commands. This doesn't happen
for addresses in the header because there may be more than one of them, and
deciding how to give that information is tricky. Can we do better?
------------------------------------------------------------------------------
(33) 25-Mar-02 S Option to assume nomatch in dnslist lookups that time out
Currently this causes a DEFER.
------------------------------------------------------------------------------
(34) 26-Mar-02 S Access to DNS lookup functions via local_scan() API
This would make local_scan() writers lives easier for DNS usage.
------------------------------------------------------------------------------
(36) 02-Apr-02 ? A way of throttling, but allowing, relaying that would
otherwise be denied
This was suggested in connection with anonymizing messages. The "wait" command
in ACLs goes some way towards this. Is it enough?
------------------------------------------------------------------------------
(41) 17-Apr-02 T Make config.samples available as a directory for ftp
This is so that people can browse individual samples directly.
------------------------------------------------------------------------------
(42) 23-Apr-02 T An option not to flatten newlines in $message_body.
Or maybe better to provide $message_body_nl so as to have both.
------------------------------------------------------------------------------
(43) 23-Apr-02 T An option to treat 4xx as 5xx from STARTTLS
This would make Exim retry in clear unless the host is in hosts_require_tls.
------------------------------------------------------------------------------
(44) 24-Apr-02 ? Use errors_to for timeouts after redirect syntax errors
A syntax error in redirection data (with skip_syntax_errors false) causes a
defer. Eventually, the address may time out. This suggestion is that, when it
does, the bounce is sent to errors_to rather than to the sender.
------------------------------------------------------------------------------
(45) 13-May-02 T smtp_etrn_serialize_id = ....
The default behaviour would be equivalent to
smtp_etrn_serialize_id = $smtp_command_argument
------------------------------------------------------------------------------
(47) 16-May-02 S Access to all addresses in batched local delivery
Miquel van Smoorenburg
In a batched local delivery with more than one recipient, there's no way to
access the list of recipients for doing custom things, such as stuffing them
all into a header. (BSMTP is the only approach; not everybody can use it.)
Suggested patch supplied.
------------------------------------------------------------------------------
(48) 21-May-02 M Support for ATRN (server and client)
Brian Candler
Server: If Exim had the ability to accept an ATRN command and then simply
invoke an external program, passing the SMTP stream on stdin and stdout and
the authenticated id as a parameter, that would do the job nicely.
Client: We need a variant of 'exim -bs' which would connect to a specified
host, send AUTH/ATRN, and then accept incoming messages as usual.
------------------------------------------------------------------------------
(50) 22-May-02 T Add comment (duplicate address) to Envelope-To:
This is just to minimize the confusion some people have.
------------------------------------------------------------------------------
(51) 07-Jun-02 S Option to use another address in callout MAIL FROM
This would be an address to try if MAIL FROM:<> failed. Is this actually going
to be helpful? See also 101.
------------------------------------------------------------------------------
(53) 11-Jun-02 S Make local_scan() dynamically loadable
David Woodhouse sent a patch. There's a more sophisticated one from Marc
Merlin. (See also Peter Benie's comments.) But should the base Exim have all
this in it?
------------------------------------------------------------------------------
(54) 11-Jun-02 S Ignore -Ac if called as mailq
I am not sure if this makes sense. This flag requests a listing of a different
mail queue, but Exim doesn't work like that. Is is not better for people to be
aware of this?
------------------------------------------------------------------------------
(55) 13-Jun-02 M Rewriting whole header lines
Dave C.
Current rewriting rules apply to individual addresses in header lines. This
feature would use a regex to match whole lines and replace them. It could be
useful for patching up syntactically invalid lines from crappy clients, before
the syntax check kicks in. (It might also be useful for hiding local host names
in Received: headers.)
------------------------------------------------------------------------------
(58) 26-Jun-02 ? Extend PAM support
Apparently PAM can do challenge-response authentication. The Exim interface
can't handle this. Investigate and think about how to do this.
------------------------------------------------------------------------------
(59) 26-Jun-02 M A "custom" authenticator
... that is simply a front end to external code. For example, there may be
an external API that hides the user password and does CRAM-MD5 when passed the
details of the challenge and response.
------------------------------------------------------------------------------
(60) 27-Jun-02 S Make trusted_users a local part list
So that it can use lsearch etc.
------------------------------------------------------------------------------
(62) 28-Jun-02 S Remove headers before DATA ACL
Patrice Fournier
"I'd like to be able to give Exim a list of headers that must be removed
from the message at arrival, before data_acl processing (and before the
rcpt_acl warn headers are added to the message)."
------------------------------------------------------------------------------
(63) 28-Jun-02 S Access to ACL-added headers in ACLs
Patrice Fournier
"I'd like also to be able to look at the already added headers by a
rcpt_acl when still checking rcpt_acl (either later in the acl for the same
RCPT TO or for another RCPT TO)."
------------------------------------------------------------------------------
(65) 28-Jun-02 M Expand fallback hosts
See also 174 of the Exim 3 list.
------------------------------------------------------------------------------
(66) 01-Jul-02 M Use Berkeley DB 4 concurrent access features
This might give better performance on very busy sites by reducing the
contention for access to hints databases. Rob Butler points out that this could
also be useful to allow updates of other DB files used by Exim to happen
concurrently. Another thing to think about with BDB is the possible use of
B-trees.
------------------------------------------------------------------------------
(68) 01-Jul-02 S Add sender host to delivery line
"Would it be possible to have a "sending_host_on_delivery" option that
logs the IP of the sending host in the => line?" Also requested was amount of
data transmitted for a non-delivery attempt.
------------------------------------------------------------------------------
(69) 03-Jul-02 T Log selector to log whoson checs
Matt Bernstein
"I'd quite like a log_selector option which could spot you'd done a whoson
lookup in your DATA ACL and maybe log it as W=user."
------------------------------------------------------------------------------
(70) 09-Jul-02 S A way of changing the RCPT address in an accept router
So as to avoid duplication problems when sending multiple addresses in multiple
copies to the same address.
------------------------------------------------------------------------------
(73) 17-Jul-02 M Match a list from within a condition
e.g. ${if matchdomain {$domain}{+domainlist} ...
${if matchhost {$sender_host_address}{1.2.3.4/10:2.3.4.5/16}...
Thought needed about how to handle host names. This may be too messy to specify
cleanly.
22-Apr-04: Implemented for domains, addresses, and local parts. Hosts are
too messy!
------------------------------------------------------------------------------
(74) 22-Jul-02 M Extend -bV to do more semantic checking
For example, diagnose "local_hosts" that should probably be "+local_hosts".
------------------------------------------------------------------------------
(75) 23-Jul-02 S Reference option on command line
The idea here is that a spam scanner that re-injects a message can supply a
reference on the command line that gets logged with R=.
------------------------------------------------------------------------------
(78) 30-Jul-02 S Expand queue_only (and/or queue_only_file)
The requirement is to make it possible to queue messages if certain conditions
are met (e.g. messages from certain local users). See also 93.
This control can now be achieved in the ACL - is this still needed?
------------------------------------------------------------------------------
(79) 31-Jul-02 S Additional info for log lines
An option to set an expanded string to be added to <= lines. And also for the
other delivery lines? See also 15.
------------------------------------------------------------------------------
(84) 09-Aug-02 S Make interfaces available in a variable
Something like $local_interfaces. Maybe limit the max length.
------------------------------------------------------------------------------
(85) 12-Aug-02 S/M Notice database connection failures
The small version of this just removes a server from the list within a single
Exim process when a connection to it fails. The bigger project would use the
retry database - but that has implications for bottlenecking and may not be
helpful. See also item 109. Another suggestion is to randomize the order in
which database servers are tried (randomize_database_servers). And another is
to measure response times and remember which server is fastest.
------------------------------------------------------------------------------
(87) 12-Aug-02 M Partial lookups for query-style lookups
The suggestion is to allow the lookup to contain a keystring (same syntax as
single-key lookups) which is then permuted and place in a suitable variable
each time - $permuted_key or something.
------------------------------------------------------------------------------
(88) 20-Aug-02 S Allow special retrying for forced defer
See also 146 in Exim 3 wish list above.
------------------------------------------------------------------------------
(89) 20-Aug-02 S Also allow retry rules on routers and transports
------------------------------------------------------------------------------
(90) 23-Aug-02 M Macros with arguments, a la C
I don't like this, because of the cost of frequent interpretation.
------------------------------------------------------------------------------
(93) 27-Aug-02 S queue_only_condition
Peter A. Savitch
queue_only_condition global option, expanded string. This contain
condition, which if evaluated to `no' or `false' or `0', behaves like
queue_only (queue_only_load ?). Don't know what to do is the string
expansion fails with DEFER (either force queueing or continue with
immediate delivery). Another option can control Exim behaviour if the
expansion fails. Don't know how the name for it ;-) See also 78.
This control can now be achieved in the ACL - is the new feature now needed?
------------------------------------------------------------------------------
(95) 27-Aug-02 S Log all parents as a router option
So that specific addresses can be logged like this. Should there be more log
selector options per router? Per transport?
------------------------------------------------------------------------------
(99) 28-Aug-02 L Test pre-conditions in order given
This would get round certain problems with require_files. However, it is
totally incompatible, and therefore an "Exim 5" wish.
------------------------------------------------------------------------------
(101) 02-Oct-02 M Callout and <> rejections
Some people don't want to fail the callout if the MAIL FROM:<> command is
rejected. Think of a way of handling this tidily. See also 51.
------------------------------------------------------------------------------
(102) 03-Oct-02 M Log option to suppress message-id logging
M because it would involve a change to eximstats.
------------------------------------------------------------------------------
(106) 09-Oct-02 S Appendfile to create directory not as user
Arrange for the setup entry to appendfile to create the directory under some
other uid (and with given owners/permissions?)
------------------------------------------------------------------------------
(109) 15-Oct-02 M Remember when LDAP (etc) servers are down
The idea would be to use some kind of retry rule, just like for hosts.
See also 85.
------------------------------------------------------------------------------
(110) 18-Oct-02 M errors_to for pipe command in filter
To work in the same was as errors_to for deliver commands.
------------------------------------------------------------------------------
(113) 15-Nov-02 M support for XMLRPC
Patch supplied for 4.10 by Joel Vandal.
------------------------------------------------------------------------------
(114) 04-Dec-02 M local_scan: return message on accept
(This actually dates from earlier.) The problem with this is that the string
currently passes into $local_scan_data. Thus, an incompatible change of some
sort would be required. Possibly a global that local_scan can set?
------------------------------------------------------------------------------
(118) 10-Dec-02 S access to Perl from local_scan
------------------------------------------------------------------------------
(119) 12-Dec-02 M ability to specify additional headers in an autoreply
This is so that vacation messages etc can have MIME headers that specify, for
example, the character set.
------------------------------------------------------------------------------
(125) 02-Jan-03 M Per-host daemon logging
"So what I would like is an option like debug_hosts, that allows to specify
an hostlist, and if the current incoming/outgoing hosts matches, creates a
logfile like $hostname_(in|out).log in my logdirectory."
------------------------------------------------------------------------------
(127) 06-Jan-03 M Different messages for different callout failures
The real requirement here is to detect when a callout "MAIL FROM:<>" failed, so
that a specific warning about that can be sent, different to the message when a
callout "RCPT TO:" fails. I think this is in fact now mostly done.
------------------------------------------------------------------------------
(129) 09-Jan-03 M Keep track of DNSBL timeouts, and refrain from calling
If so configured, keep track of DNSBL timeouts in a hints record, and don't
retry that DNSBL for a while after (a sufficient number of) timeouts. It is
effectively disabled for a while. Log enable/disable, of course. Another
thought is an option not to apply +defer_unknown unless *all* DNSBL lookups in
a list defer.
------------------------------------------------------------------------------
(130) 09-Jan-03 M A number of LDAP-related things
Peter A. Savitch
OpenLDAP 2.1 is going to be more popular (2.1.9 is available with many
bug fixes). TLS-enabled LDAP is an interesting and usefull thing.
I can try to implement some things and send the patches, like with
ldapi.
How do You see:
1) The propagation of TLS options (key, certificate, CA certificate)
to the OpenLDAP library.
2) (was dereferencing; done in 4.23).
------------------------------------------------------------------------------
(131) 09-Jan-03 S Additional variables
Peter A. Savitch
$root_uid Why?
(Some that were previously here are done)
$smtp_accept_count -- used for acl_smtp_connect
$queue_runners -- children of the listening daemon could use this
value for controlling the number of queue runners
I don't like either of these because they cannot be real-time values. They
would be snapshots of the values at the time the process was forked from the
daemon, and I fear they would just be confusing. For processes that were not
forked from the daemon they couldn't be set at all.
------------------------------------------------------------------------------
(131) 09-Jan-03 S Additional options
Peter A. Savitch
exim_processes_max
exim_file_descriptors_max
queue_run_condition -- to deprecate queue_run_max, better system
load control
Given Exim's distributed nature, I'm not at all sure how the first two of these
can usefully be implemented.
------------------------------------------------------------------------------
(132) 16-Jan-03 M Option for when a transport filter fails (crashes)
Freezing is one obvious option. Currently, Exim just retries. Another user
wanted to retry without the filter, but that is much harder.
------------------------------------------------------------------------------
(136) 24-Jan-03 M Make "personal" available as a condition for use in routers
------------------------------------------------------------------------------
(138) 28-Jan-03 M A variable containing what was matched in a host list
Or, presumably, other lists. This is so that ACL messages can say things like
"your host name matches xxxx". Note: not the same as $domain_data. Also, this
could be tricky with lookups and things that match in files.
------------------------------------------------------------------------------
(143) 06-Mar-03 L Ability to have multiple authenticators of same type
For example, to have two PLAIN authenticators; if the first fails, try the
second.
------------------------------------------------------------------------------
(144) 07-Mar-03 T ACL control = local_scan_skip to skip the local scan
A bigger project would be control = local_scan <xxx> where xxx could select
different local_scan functions (possibly by dynamic loading).
This can now be simulated using the fact that ACL variables are preserved,
so it doesn't look as it once did.
------------------------------------------------------------------------------
(145) 07-Mar-03 T Export string_cat() to local_scan()
------------------------------------------------------------------------------
(147) 17-Mar-03 T Option to treat 5xx as 4xx if received on initial connection
This issue is controversial. That may be a good reason for not changing
anything.
------------------------------------------------------------------------------
(153) 25-Apr-03 S A way of making log_as_local apply to the smtp transport
Either an option on the transport, or log_remote_as_local for the router.
Messy, either way. Maybe log_local_as_local and log_remote_as_local, and
deprecate log_as_local?
------------------------------------------------------------------------------
(154) 01-May-03 M Teergrubing at the CR/LF level
It is believed that the most effective way to teergrube is to insert a delay
between transmitting CR and LF in the SMTP response. Furthermore, this is also
the best place to test for bad synchronization (i.e. at the last possible
time).
------------------------------------------------------------------------------
(155) 01-May-03 S "control=no_pipelining" for connect and EHLO ACLs
Yet more flexibility! Maybe this should be a more general control for what is
sent in response to EHLO.
------------------------------------------------------------------------------
(156) 06-May-02 M Finer-grained synchronisation checking
On operating systems that can be asked whether any sent bytes have not yet been
ACK'd at the TCP/IP level, a finer-grained check for proper synchronisation can
be done. All bytes must have been ACK'd if the client has received the previous
response before sending the next command. See also 293.
------------------------------------------------------------------------------
(157) 07-May-03 M Newline as a list item separator
This will make life easier for lists obtained form databases where the
separator is naturally a newline.
------------------------------------------------------------------------------
(158) 13-May-03 M Ability to add to OK message for SMTP commands
For sending reasons for slow response, etc.
------------------------------------------------------------------------------
(160) 19-May-03 M Remove headers using wild cards
------------------------------------------------------------------------------
(162) 28-May-03 M/L Use of real numbers in filters, expansions, and options
The motivation for this is for handling spam scores that are real numbers. The
questions are (a) how widely should it spread and (b) whether floating point or
fixed point representations should be used. And what about the eval operator?
------------------------------------------------------------------------------
(164) 02-Jun-03 S Set variables for interface and port in smtp transport
These could be useful for varying HELO data etc. See also several other
items about interfaces above.
------------------------------------------------------------------------------
(166) 18-Jun-03 S CN verification in client TLS code
A tls_verify_cn option is suggested by Sven Geggus.
------------------------------------------------------------------------------
(168) 19-Jun-03 S Ability to add a header recording envelope rewrites
Current code adds a deleted header with only some information. Maybe what is
needed is a flag for a rewrite rule.
------------------------------------------------------------------------------
(169) 19-Jun-03 M A way of detecting timeouts in callout returns
------------------------------------------------------------------------------
(170) 23-Jun-03 S Option to accept rather than defer after local scan timeout
Suggested patch supplied.
------------------------------------------------------------------------------
(171) 23-Jun-03 S Option to make timeout a soft failure on pipe transport
------------------------------------------------------------------------------
(172) 23-Jun-03 M Option to make SQL query to specific server
------------------------------------------------------------------------------
(175) 04-Jul-03 S show_all_ancestors_in_errmsg for the redirect router
This is the opposite of hide_child_in_errmsg in effect.
------------------------------------------------------------------------------
(180) 14-Jul-03 M Extend never_users to be more flexible
e.g. never_users = ! mailnull : ! cyrus : !mailman : 0-100
------------------------------------------------------------------------------
(183) 16-Jul-03 S freeze_tell_text to add custom text to the message
------------------------------------------------------------------------------
(185) 24-Jul-03 S An expansion operator that decodes RFC 2047 strings
------------------------------------------------------------------------------
(188) 13-Aug-03 T batch_max=0 to mean unlimited
------------------------------------------------------------------------------
(189) 22-Aug-03 S Allow filter "logwrite" to write to syslog
I feel this is a dangerous facility, and also of very minority interest, at
least for user's filters. Allowing a system filter to write to mainlog or
syslog may be different. However, writing the main log would only be possible
if the filter runs as root or exim.
------------------------------------------------------------------------------
(190) 22-Aug-03 S A way of testing "forced delivery" in filter and routers
------------------------------------------------------------------------------
(191) 26-Aug-03 M Preserve $address_data for a verified recipient
The idea is to preserve it in the recipients data structure so that local_scan
can have access to it. The value could also be used as the initial value of
$address_data while routing.
------------------------------------------------------------------------------
(192) 05-Sep-03 M Better handling of TXT records for dnslists
When multiple lists are accessible via a merged lookup, handling TXT records
is difficult. An option for doing the TXT lookup in a sub-list has been
suggested, with syntax such as
dnslists = list.example.org=127.0.0.2%dialups \
,127.0.0.3%relays \
,127.0.0.5%spews
------------------------------------------------------------------------------
(194) 10-Sep-03 M $addresslist_data to be like $host_data/$domain_data
------------------------------------------------------------------------------
(195) 29-Sep-03 M A variable containing the error for verify = header_syntax
Maybe there should always be a variable with the error message for all the
different kinds of verify failure.
------------------------------------------------------------------------------
(196) 30-Sep-03 S A way of detecting whether it was HELO or EHLO in the ACL
$received_protocol isn't reset until after the command is accepted (which
seems right), and $smtp_data shows only the arguments. Maybe $smtp_command?
------------------------------------------------------------------------------
(197) 30-Sep-03 S MACROS_DROP_PRIVS and ALT_CONFIG_DROP_PRIVS
Now that alternative configurations can be restricted to certain directories,
some more flexibility can be allowed. Not by default, though.
------------------------------------------------------------------------------
(198) 01-Oct-03 M Accept mail after local_scan() crash instead of defer
This may not be as easy to implement as it sounds; one is never sure of the
environment after a crash. Is is actually a good idea? The crashing local_scan
may have wrecked the memory in arbitrary ways; for example, screwing up the
recipients list...
------------------------------------------------------------------------------
(199) 01-Oct-03 M ${pipe which will pipe the message to a script ...
... and otherwise behave as ${run. Probably needs to have locking out features
so that it can be turned off for users .forwards if the sysadmin so desires.
------------------------------------------------------------------------------
(200) 07-Oct-03 L Alternative ways of storing hints
People want to store hints in databases. Some assert that SQL databases can
be made to perform satisfactorily. If a general interface could be worked on,
people could at least try different strategies. See also 66 above, which is
specifically concerned with Berkeley DB. Another possible option is a switch to
disable smtp-wait hints - to avoid contention problems.
------------------------------------------------------------------------------
(201) 07-Oct-03 M A "soft bounce" feature
This is an option that turns all hard bounces into soft bounces. The idea is
that it can be used as a safety-net while testing configurations. Instead of a
local bounce, the message stays on the queue; instead of 5xx SMTP responses,
4xx ones are given.
The ability to do the opposite - turn 4xx into 5xx under certain circumstances
might also be useful (e.g. after a certain time). This might best be done by
extending the retry logic to recognize 4xx as a special error. (This is now
done.)
------------------------------------------------------------------------------
(202) 10-Oct-03 S -bvsomething to do a callout after the verify
------------------------------------------------------------------------------
(203) 14-Oct-03 S verify=something to easily check for header presence
This is purely cosmetic; "condition" can already be used.
------------------------------------------------------------------------------
(204) 27-Oct-03 S an inverted queue_only_file
That is, queue if a file does NOT exist.
------------------------------------------------------------------------------
(205) 27-Oct-03 S expand smtp_accept_queue_per_connection
------------------------------------------------------------------------------
(206) 27-Oct-03 S appendfile: a variable containing the maildir base name
------------------------------------------------------------------------------
(207) 29-Oct-03 S ability to keep trusted users in a file - expand it.
------------------------------------------------------------------------------
(208) 31-Oct-03 M cache temporary verification errors and fail after a time
This request was for a way of turning temporary verification failures into
permanent ones after some fixed time.
------------------------------------------------------------------------------
(209) 31-Oct-03 S a way of making crashes in pipe commands temporary errors
------------------------------------------------------------------------------
(210) 31-Oct-03 S runtime option to change the daemon name used for tcprwappers
A patch for compile time was supplied, but this seems better as a runtime
option, for use with multiple Exim daemons.
------------------------------------------------------------------------------
(211) 31-Oct-03 S ability to disable debugging output from -bh & -bhc
------------------------------------------------------------------------------
(212) 31-Oct-03 M specify headers lines in HELO ACL to be added to all msgs
------------------------------------------------------------------------------
(214) 05-Nov-03 S Put the wild part of local part prefix/suffx in variables
Unfortunately, this isn't quite as trivial as it seems.
------------------------------------------------------------------------------
(215) 14-Nov-03 S A way of turning off message-submission fix-ups
Globally, and perhaps also via an ACL control so that it can be done on a
per-message basis.
------------------------------------------------------------------------------
(215) 26-Nov-03 M/L Conversion of IDNA domain names for logging
IDNA (RFCs 3490-3492) converts domains names containing non-ASCII characters
into ASCII strings of a special form. Exim will of course handle these.
However, it might be nice to convert them to a local code for logging. This
might be quite a big project: there's also output from -bp and eximon queue
display and no doubt other places as well. (Utilities that process the logs,
e.g. exigrep, eximstats, will be automatically handled if the logs are
changed.)
------------------------------------------------------------------------------
(216) 27-Nov-03 S Option to bounce if required TLS doesn't happen
This is for the smtp transport with hosts_require_tls set. Currently, it
defers. Possibly the best approach is to make the error one that can be seen by
the retry logic.
------------------------------------------------------------------------------
(217) 27-Nov-03 M A function to pass back variables from Perl
This is a function that can be called from Perl, to take a name and a value and
put that value into an Exim variable.
------------------------------------------------------------------------------
(218) 01-Dec-03 M A local_scan-like hook at system filter time
That is, make a C API available for custom filtering at this point.
------------------------------------------------------------------------------
(221) 18-Dec-03 U Merge routers and ACLs - or at least make more similar
"It will be very useful to be able to use most of the ACL conditions
(authenticated, hosts, senders, sender_domains, ... ) in routers and also the
possibility to have multiple conditions in routers. It will be great to also
be able to set variables in routers like in acl's." This is effectively a
radical suggestion for a complete re-design, and is therefore BIG.
------------------------------------------------------------------------------
(222) 19-Dec-03 S Iterative option for dnsdb
A way of getting a dnsdb lookup to chop off components until something is
found: e.g. ${lookup dndsb-i{ns=a.b.c.d}} would look for nameservers for
a.b.c.d, then b.c.d, etc.
------------------------------------------------------------------------------
(223) 22-Dec-03 S Support SOA lookup in dnsdb lookups
------------------------------------------------------------------------------
(225) 22-Dec-03 M Add acl= to routers
This would use an ACL to "control access" to a router, opening up a number
of interesting possibilities. Details of possible limitations need to be
investigated.
------------------------------------------------------------------------------
(226) 23-Dec-03 S A way of treating DEFER as fail in dnsdb lookups
(i.e. the dnsdb lookup failed, so accept the message)
------------------------------------------------------------------------------
(227) 30-Jan-04 M A configuration .if facility
"Second with the .ifdef and such, it would be nice to have a base .if,
so I could do something like
.if DEFINED_DATA == xyz
configuration here
.elseif DEFINED_DATA == abc
configuration here
.else
configuration here
.endif
also this would be nice at least in my case in the system filters, but
isn't required but you could pass the defined data to the system, in
variables."
------------------------------------------------------------------------------
(229) 30-Jan-04 M New expansion mechanism: {list ...}
"Proposed syntax: {list {separator}{item}{item}...}
This first expands the contents of {separator} and all of the {item}s,
then constructs a separator-delimited list. The twist is: if an {item}
generates the empty string, no separator will be generated for it.
The entire construct will fail is {separator} fails, or all {item}s
fail. If just some {item}s fail, they will be treated as if they
generated empty strings.
Examples:
{list {,}{aaaaaa}{bbbbbb}{cccccc}} -> aaaaaa,bbbbbb,cccccc
{list {,}{:fail:}{bbbbbb}{cccccc}} -> bbbbbb,cccccc
{list {,}{aaaaaa}{:fail:}{cccccc}} -> aaaaaa,cccccc
{list { }{aaaaaa}{bbbbbb}{}} -> aaaaaa bbbbbb
{list { }{:fail:}{:fail:}{:fail:}} -> :fail:
{list {:fail:}{aaaaa}{bbbb}{cccc}} -> :fail:
See particularly examples 2-4, which handle the case of a missing first
and last item with ease; doing this using {if ...} would be quite difficult!"
------------------------------------------------------------------------------
(230) 30-Jan-04 M Find IP addresses of a domain's nameservers
This needs some way of processing a list of things in a similar way, which
should perhaps be a more general facility.
------------------------------------------------------------------------------
(231) 30-Jan-04 ? -C has a number of problems when used for real
-C was intended for testing; people are using it for "alternate"
configurations, and it doesn't work too well. Can a better way of doing this be
invented?
------------------------------------------------------------------------------
(232) 02-Feb-04 ? Make parts of the code loadable
The idea being that drivers, etc. could be compiled separately. There are, of
course, security issues. This is not something I want to go into at present.
------------------------------------------------------------------------------
(235) 02-Feb-04 T Make smtp_accept_count available as a variable
This is for use in ACLs. Of course, it is a snapshot of the count at the
start of the receiving process.
------------------------------------------------------------------------------
(236) 02-Feb-04 S String in local_scan that's added to the binary version string
------------------------------------------------------------------------------
(237) 02-Feb-04 M Add_header in ACLs because "message" is overloaded
This would be useful for verbs where "message" is an error message.
------------------------------------------------------------------------------
(238) 05-Feb-04 S ${address to handle multiple addresses
At present, ${address expects to see just one address. An extension would let
it handle header lines with multiple addresses, just retaining the actual
addresses. Or perhaps a new operator is needed?
------------------------------------------------------------------------------
(239) 23-Feb-04 ? Expansion items for encryption/decryption
Perhaps for some kind of cookie handling? This would need an external crypto
library, because there's no crypto code in Exim itself.
------------------------------------------------------------------------------
(240) 23-Feb-04 ? Some way to know if a ip is a mx for a given domain
Some kind of iterative operation for dnsdb might be a general way of providing
this.
------------------------------------------------------------------------------
(242) 01-Mar-04 ? Run a filter from an expansion condition
This would add a lot of power to ACLs, but its implementation might be tricky
because of the possibility of recursion.
------------------------------------------------------------------------------
(243) 01-Mar-04 ? Run an ACL from an expansion condition
The problem here is knowing what data is available at an arbitrary time.
------------------------------------------------------------------------------
(244) 01-Mar-04 ? Add an on-success event to transports
This could just be an expansion string, whose value is either ignored or
logged, but it could be used to run SQL updates or run programs etc.
However, what is "success" when a transport has multiple recipients?
------------------------------------------------------------------------------
(245) 01-Mar-04 M Add all the string expansion conditions to filters
Some thought would be needed on how to design the syntax for this.
------------------------------------------------------------------------------
(247) 09-Mar-04 S IP addresses that are never looked up
It would be nice if we could prevent this for certain IP addresses for
which we _know_ we'll never get a valid PTR record, like 2002::/16.
So a new option might reasonably default to:
hosts_never_lookup = <; 2002::/16
------------------------------------------------------------------------------
(253) 05-Apr-04 M Use ESMTP and TLS for recipient callout verification
The best way to do this would involve quite a bit of refactoring so as to
abstract some of the code from the smtp transport into subroutines that could
also be used from the callout code. The tls parameters should probably be
taken from the transport. That might also require some substantial code
refactoring. See also 294.
------------------------------------------------------------------------------
(260) 30-Apr-04 S Respect +tls_cipher +tls_peerdn in rejectlog entries
------------------------------------------------------------------------------
(261) 05-May-04 S Add a "required_version" option
So that configurations can insist on a specific Exim version.
------------------------------------------------------------------------------
(262) 10-May-04 S Add "scratch" ACL variables
The idea is for variables that are flushed at the start of each ACL. I'm not
really convinced that these are worth implementing.
------------------------------------------------------------------------------
(263) 10-May-04 S Add variable $router_name $transport_name
These could be used in debug_print settings, which are output during -bt, and
thus don't need the privilege to run with -d.
------------------------------------------------------------------------------
(265) 25-May-04 M An init.d script for exim is needed
The old sendmail script used to "just work" because it just did -bd -q 20m or
whatever. Newer versions start more than one sendmail daemon, so do not work.
------------------------------------------------------------------------------
(267) 25-May-04 S tarpitting delay option
A modifier that sets a delay between lines for multiline responses.
------------------------------------------------------------------------------
(268) 25-May-04 S? Add a PID to every log line
Given that pids are reused non-cyclically these days, is this actually useful?
------------------------------------------------------------------------------
(269) 26-May-04 U Run both a system and a user filter in test mode
exim -bF systemfilter -bf userfilter -f sender@dom < message
This would allow testing the way the userfilter handles the system
variables set by the systemfilter.
------------------------------------------------------------------------------
(270) 01-Jun-04 M Add headers at top and middle
Various initiatives like SPF and DomainKeys require header lines to be added
above or in the middle of existing headers. Exim always adds at the bottom.
When these requirements are more standard and clearer, some way of controlling
where header lines are added will probably become necessary. Some new syntax
will be required.
This can now be done fairly generally from local_scan(), and at the start and
after the Received: block from an ACL. Is anything more needed?
------------------------------------------------------------------------------
(271) 02-Jun-04 L Callouts at routing time
From a user's message:
> I would like to be able to:
>[...]
> 2) Forcing callouts as address verification at router level
> [ check_callout just like check_local_user ]
>
> I would like to redirect messages in some domain to "domain with callout
> verification" and to "domain without callout verification"
>
> e.g.
> userA@??? -> userX@??? (use callout to verify)
> userB@??? -> userY@??? (do not use callout verify)
>
> [both out-* domains delivered via "callout ready" transports]
Other versions of the wish:
* limiting callouts in acls to specific transport
verify = recipient/callout=5s,transport:intranet_smtp
* adding "select transport" to ACL conditions
accept domains = +local_domains
transport = cyrus_ltcp
verify = recipient/callout=5s
------------------------------------------------------------------------------
(272) 07-Jun-04 S Expand hosts_randomize
It occurs in manualroute and in smtp.
------------------------------------------------------------------------------
(278) 21-Jun-04 M quota_warn_message_file option
Similar to the bounce and delivery warn message files.
------------------------------------------------------------------------------
(280) 23-Jun-04 M A way of adding a header line after callout defer_ok
This would record that, e.g., a sender domain verified, but the callout
could not be done.
------------------------------------------------------------------------------
(285) 16-Jul-04 M Separate and independent log_selector for rejectlog
For example: mainlog_selector and rejectlog_selector, with log_selector setting
both of them.
------------------------------------------------------------------------------
(286) 21-Jul-04 M Distinguishing a larger number of errors
For instance, detecting "connection reset by peer" (ENETRESET or ECONNRESET)
might be useful.
------------------------------------------------------------------------------
(288) 10-Aug-04 M Option for verify to require MX
e.g. verify=sender/require_mx
I'm not too keen because this is rather special purpose, and of course could
only apply if the verification happened to hit a dnslookup router.
------------------------------------------------------------------------------
(289) 10-Aug-04 L Option to treat defers in database lookups as "not found"
This is so that alternatives can be coded for when databases are down. A
suggested patch has been sent, but it just catches all instances of "defer"
from a lookup in an expansion string. These can occur for a number of different
reasons, not just connection failures. I think that we need a specific
"connection failed" indicator. Also, what about lookups in lists?
------------------------------------------------------------------------------
(291) 13-Aug-04 M An ACL or "local_scan()" to be run on size excession
The idea is to give something a chance to look at the data so far received when
more than message_size_limit (or some other limit) has arrived. I am not sure
how useful this would actually be in practice.
------------------------------------------------------------------------------
(292) 13-Aug-04 M Overall timeout for message reception
A client could in priciple keep an SMTP connection open for a very long time by
trickling in data very slowly. Also, after message_size_limit is exceeded, Exim
continues to swallow the data (though it does not write it to disk) until the
end is reached. Again, the connection could be held open for a very long time.
Some kind of overall time limit for an SMTP connection, possibly reset at the
start of each message, might be helpful in these situations.
------------------------------------------------------------------------------
(294) 23-Aug-04 L Callouts and AUTH and LMTP
People want to do callouts using LMTP as well as SMTP, and that would also
include sockets as well as TCP/IP connections. Also, people want to make use of
AUTH during the callout checking, on all types of connection. I suppose that
means making TLS available as well. This probably means a rewrite of the code
that actually does the callout. Should we use the relevant transport in a new
"callout" mode instead of keeping things separate? See also 253.
------------------------------------------------------------------------------
(296) 09-Sep-04 S Make deliver_time work for == lines as well as =>
What about ** lines?
------------------------------------------------------------------------------
--- HWM 297 ------------------------------------------------------------------
---------------------------- End of WishList ---------------------------------
Index: ABOUT
====================================================================
$Cambridge: exim/exim-doc/doc-scripts/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $
CVS directory exim/exim-doc/doc-scripts
---------------------------------------
This directory contains various scripts that are used to build the distributed
documentation from its source files.
End
Index: ABOUT
====================================================================
$Cambridge: exim/exim-doc/doc-src/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $
CVS directory exim/exim-doc/doc-src
-----------------------------------
This directory contains documentation files that are processed in some way in
order to make the documentation files that form part of Exim distributions. A
non-standard document processor is currently in use (October 2004), but in the
long term something more standard will have to take over.
End
Index: ABOUT
====================================================================
$Cambridge: exim/exim-doc/doc-txt/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $
CVS directory exim/exim-doc/doc-txt
-----------------------------------
This directory contains various documentation files that exist only as plain
text files, and are distributed in that format.
End
Index: ABOUT
====================================================================
$Cambridge: exim/exim-src/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $
CVS directory exim/exim-src
---------------------------
This directory contains everything that is included in an Exim distribution
tarball, with the exception of the doc directory and an empty Local directory.
You can build Exim from the contents of this directory by adding a Local
directory that contains appropriate configuration files.
End
