[exim-cvs] cvs commit: exim/exim-doc/doc-misc ABOUT Ext-mail…

Góra strony
Delete this message
Reply to this message
Autor: Philip Hazel
Data:  
Dla: exim-cvs
Temat: [exim-cvs] cvs commit: exim/exim-doc/doc-misc ABOUT Ext-maildir Ext-maildir++ Ext-mbx-locking LongTermIssues RFC.conform TexiNotes WishList exim/exim-doc/doc-scripts ABOUT exim/exim-doc/doc-src ABO
ph10 2004/10/08 11:38:48 BST

  Added files:
    exim-doc/doc-misc    ABOUT Ext-maildir Ext-maildir++ 
                         Ext-mbx-locking LongTermIssues 
                         RFC.conform TexiNotes WishList 
    exim-doc/doc-scripts ABOUT 
    exim-doc/doc-src     ABOUT 
    exim-doc/doc-txt     ABOUT 
    exim-src             ABOUT 
  Log:
  Start


  Revision  Changes    Path
  1.1       +11 -0     exim/exim-doc/doc-misc/ABOUT (new)
  1.1       +109 -0    exim/exim-doc/doc-misc/Ext-maildir (new)
  1.1       +394 -0    exim/exim-doc/doc-misc/Ext-maildir++ (new)
  1.1       +400 -0    exim/exim-doc/doc-misc/Ext-mbx-locking (new)
  1.1       +200 -0    exim/exim-doc/doc-misc/LongTermIssues (new)
  1.1       +401 -0    exim/exim-doc/doc-misc/RFC.conform (new)
  1.1       +193 -0    exim/exim-doc/doc-misc/TexiNotes (new)
  1.1       +1727 -0   exim/exim-doc/doc-misc/WishList (new)
  1.1       +9 -0      exim/exim-doc/doc-scripts/ABOUT (new)
  1.1       +11 -0     exim/exim-doc/doc-src/ABOUT (new)
  1.1       +9 -0      exim/exim-doc/doc-txt/ABOUT (new)
  1.1       +11 -0     exim/exim-src/ABOUT (new)


Index: ABOUT
====================================================================
$Cambridge: exim/exim-doc/doc-misc/ABOUT,v 1.1 2004/10/08 10:38:47 ph10 Exp $

CVS directory exim/exim-doc/doc-misc
------------------------------------

This directory contains some miscellaneous documentation files that do not form
part of Exim distributions, but are related to its maintenance and development.
Those whose names start with "Ext-" are external documents that won't be
modified (and hence have no local CVS Ids).

End

Index: Ext-maildir
====================================================================
The following information is from the maildir man page of qmail.

  INTRODUCTION
         maildir  is  a  structure for directories of incoming mail
         messages.  It solves the reliability problems that  plague
         mbox files and mh folders.


  RELIABILITY ISSUES
         A machine may crash while it is delivering a message.  For
         both mbox files and mh folders this means that the message
         will  be silently truncated.  Even worse: for mbox format,
         if the message is truncated in the middle of  a  line,  it
         will  be  silently  joined  to the next message.  The mail
         transport agent will try again later to deliver  the  mes-
         sage,  but  it  is  unacceptable  that a corrupted message
         should show up at all.  In maildir, every message is guar-
         anteed complete upon delivery.


         A  machine may have two programs simultaneously delivering
         mail to the same user.  The mbox and  mh  formats  require
         the programs to update a single central file.  If the pro-
         grams do not use some locking mechanism, the central  file
         will  be corrupted.  There are several mbox and mh locking
         mechanisms, none of which work portably and reliably.   In
         contrast,  in  maildir, no locks are ever necessary.  Dif-
         ferent delivery processes never touch the same file.


         A user may try to delete messages from his mailbox at  the
         same  moment that the machine delivers a new message.  For
         mbox and mh formats, the user's mail-reading program  must
         know  what  locking  mechanism  the mail-delivery programs
         use.  In contrast, in maildir, any delivered  message  can
         be safely updated or deleted by a mail-reading program.


         Many sites use Sun's Network Failure System (NFS), presum-
         ably because the operating system vendor  does  not  offer
         anything else.  NFS exacerbates all of the above problems.
         Some NFS implementations don't provide any reliable  lock-
         ing  mechanism.  With mbox and mh formats, if two machines
         deliver mail to the same user, or if  a  user  reads  mail
         anywhere  except  the delivery machine, the user's mail is
         at risk.  maildir works without trouble over NFS.


  THE MAILDIR STRUCTURE
         A directory in maildir format  has  three  subdirectories,
         all on the same filesystem: tmp, new, and cur.


         Each  file  in new is a newly delivered mail message.  The
         modification time of the file is the delivery date of  the
         message.   The message is delivered without an extra UUCP-
         style From_ line, without any >From quoting,  and  without
         an  extra  blank line at the end.  The message is normally
         in RFC 822 format, starting with a Return-Path line and  a
         Delivered-To  line,  but it could contain arbitrary binary
         data.  It might not even end with a newline.


         Files in cur are just like files in new.  The big  differ-
         ence  is  that  files  in cur are no longer new mail: they
         have been seen by the user's mail-reading program.


  HOW A MESSAGE IS DELIVERED
         The tmp directory is used to ensure reliable delivery,  as
         discussed here.


         A program delivers a mail message in six steps.  First, it
         chdir()s to the maildir directory.  Second, it stat()s the
         name  tmp/time.pid.host,  where time is the number of sec-
         onds since the beginning of 1970 GMT, pid is the program's
         process  ID,  and host is the host name.  Third, if stat()
         returned anything other than ENOENT,  the  program  sleeps
         for two seconds, updates time, and tries the stat() again,
         a limited number of times.  Fourth,  the  program  creates
         tmp/time.pid.host.  Fifth, the program NFS-writes the mes-
         sage to the file.  Sixth, the program link()s the file  to
         new/time.pid.host.   At  that instant the message has been
         successfully delivered.


         The delivery program is required to start a 24-hour  timer
         before creating tmp/time.pid.host, and to abort the deliv-
         ery if the timer expires.  Upon error, timeout, or  normal
         completion,  the  delivery program may attempt to unlink()
         tmp/time.pid.host.


         NFS-writing means (1) as usual,  checking  the  number  of
         bytes returned from each write() call; (2) calling fsync()
         and checking its return value;  (3)  calling  close()  and
         checking  its return value.  (Standard NFS implementations
         handle fsync() incorrectly but make up for it  by  abusing
         close().)


  HOW A MESSAGE IS READ
         A mail reader operates as follows.


         It  looks through the new directory for new messages.  Say
         there is a new message, new/unique.  The reader may freely
         display  the contents of new/unique, delete new/unique, or
         rename     new/unique     as     cur/unique:info.      See
         http://pobox.com/~djb/maildir.html   for  the  meaning  of
         info.


         The reader is also expected to look through the tmp direc-
         tory and to clean up any old files found there.  A file in
         tmp may be safely removed if it has not been  accessed  in
         36 hours.


         It is a good idea for readers to skip all filenames in new
         and cur starting with a dot.   Other  than  this,  readers
         should not attempt to parse filenames.
  ###


  Index: Ext-maildir++
  ====================================================================
                                     Maildir++


     In this document:
       * HOWTO.maildirquota
       * Mission statement
       * Definitions and goals
       * Contents of a maildirsize
       * Calculating maildirsize
       * Calculating the quota for a Maildir++
       * Delivering to a Maildir++
       * Reading from a Maildir++
       * Bugs


HOWTO.maildirquota

     The remaining portion of this document is a technical description of
     the maildir quota extension. This section is a brief overview of this
     extension.


    What is a maildirquota?


     If you would like to have a quota on your maildir mailboxes, the best
     solution is to always use filesystem-based quotas: per-user usage
     quotas that is enforced by the operating system.


     This is the best solution when the default Maildir is located in each
     account's home directory. This solution will NOT work if Maildirs are
     stored elsewhere, or if you have a large virtual domain setup where a
     single userid is used to hold many individual Maildirs, one for each
     virtual user.


     This extension to the maildir format allows a "voluntary" maildir
     quota implementation that does not rely on filesystem-based quotas.


    When maildirquota will not work.


     For this quota mechanism to work, all software that accesses a maildir
     must observe this quota protocol. It follows that this quota mechanism
     can be easily circumvented if users have direct (shell) access to the
     filesystem containing the users' maildirs.


     Furthermore, this quota mechanism is not 100% effective. It is
     possible to have a situation where someone may go over quota. This
     quota implementation uses a deliverate trade-off. It is necessary to
     use some form of locking in order to have a complete bulletproof quota
     enforcement, but maildirs mail stores were explicitly designed to
     avoid any kind of locking. This quota approach does not use locking,
     and the tradeoff is that sometimes it is possible for a few extra
     messages to be delivered to the maildir, before the door is
     permanently shot.


     For best performance, all maildir clients should support this quota
     extension, however there's a wide degree of tolerance here. As long as
     the mail delivery agent that puts new messages into a Maildir uses
     this extension, the quota will be enforced without excessive
     degradation.


     In the worst case scenario, quotas are automatically recalculated
     every fifteen minutes. If a maildir goes over quota, and a mail client
     that does not support this quota extension removes enough mail from
     the maildir, the mail delivery agent will not be immediately informed
     that the maildir is now under quota. However, eventually the correct
     quota will be recalculated and mail delivery will resume.


     Mail user agents sometimes put messages into the maildir themselves.
     Messages added to a maildir by a mail user agent that does not
     understand the quota extension will not be immediately counted towards
     the overall quota, and may not be counted for an extensive period of
     time. Additionally, if there are a lot of messages that have been
     added to a maildir from these mail user agents, quota recalculation
     may impose non-trivial load on the system, as the quota recalculator
     will have to issue the stat system call for each message.


    How to implement the quota


     The best way to do that is to modify your mail server to implement the
     protocol defined by this document. Not everyone, of course, has this
     ability. Therefore, an alternate approach is available.


     This package creates a very short utility called "deliverquota". It
     will NOT be installed anywhere by default, unless this maildir quota
     implementation is a part of a larger package, in which case the parent
     package may install this utility somewhere. If you obtained the
     maildir package separately, you will need to compile it by running the
     configure script, then by running make.


     deliverquota takes two arguments. deliverquota reads the message from
     standard input, then delivers it to the maildir specified by the first
     argument to deliverquota. The second argument specifies the actual
     quota for this maildir, as defined elsewhere in this document.
     deliverquota will deliver the message to the maildir, making a best
     effort not to exceed the stated quota. If the maildir is over quota,
     deliverquota terminates with exit code 77. Otherwise, it delivers the
     message, updates the quota, and terminates with exit code 0.


     Therefore, proceed as follows:
       * Copy deliverquota to some convenient location, say /usr/local/bin.
       * Configure your mail server to use deliverquota. For example, if
         you use Qmail and your maildirs are all located in $HOME/Maildir,
         replace the './Maildir/' argument to qmail-start with the
         following:
  '| /usr/local/bin/deliverquota ./Maildir 1000000S'





         This sets a one million byte limit on all Maildirs. As I
         mentioned, this is meaningless if login access is available,
         because the individual account owner can create his own
         $HOME/.qmail file, and ignore deliverquota. Note that in this
         case, you MUST use apostrophes on the qmail-start command line, in
         order to quote this as one argument.


     If you would like to use different quotas for different users, you
     will have to put together a separate process or a script that looks up
     the appropriate quota for the recipient, and runs deliverquota
     specifying the quota. If no login access to the mail server is
     available, you can simply create a separate $HOME/.qmail for every
     recipient.


     That's pretty much it. If you handle a moderate amount of mail, I have
     one more suggestion. For the first couple of weeks, run deliverquota
     setting the second argument to an empty string. This disables quota
     enforcement, however it still activates certain optimizations that
     permit very fast quota recalculation. Messages delivered by
     deliverquota have their message size encoded in their filename; this
     makes it possible to avoid stat-ing the message in the Maildir, when
     recalculating the quota. Then, after most messages in your maildirs
     have been delivered by deliverquota, activate the quotas!!!


    maildirquota-enhanced applications


     This is a list of applications that have been enhanced to support the
     maildirquota extension:
       * maildrop - mail delivery agent/mail filter.
       * SqWebmail - webmail CGI binary.


     These applications fall into two classes:
       * Mail delivery agents. These applications read some externally
         defined table of mail recipients and their maildir quota.
       * Mail clients. These applications read maildir quota information
         that has been defined by the mail delivery agent.


     Mail clients generally do not need any additional setup in order to
     use the maildirquota extension. They will automatically read and
     implement any quota specification set by the mail delivery agent.


     On the other hand, mail delivery agents will require some kind of
     configuration in order to activate the maildirquota extension for some
     or all recipients. The instructions for doing that depends upon the
     mail delivery agent. The documentation for the mail delivery agent
     should be consulted for additional information.
       _________________________________________________________________


Mission statement

     Maildir++ is a mail storage structure that's based on the Maildir
     structure, first used in the Qmail mail server. Actually, Maildir++ is
     just a minor extension to the standard Maildir structure.


     For more information, see http://www.qmail.org/man/man5/maildir.html.
     I am not going to include the definition of a Maildir in this
     document. Consider it included right here. This document only
     describes the differences.


     Maildir++ adds a couple of things to a standard Maildir: folders and
     quotas.


     Quotas enforce a maximum allowable size of a Maildir. In many
     situations, using the quota mechanism of the underlying filesystem
     won't work very well. If a filesystem quota mechanism is used, then
     when a Maildir goes over quota, Qmail does not bounce additional mail,
     but keeps it queued, changing one bad situation into another bad
     situation. Not only know you have an account that's backed up, but now
     your queue starts to back up too.


Definitions, and goals

     Maildir++ and Maildir shall be completely interchangeable. A Maildir++
     client will be able to use a standard Maildir, automatically
     "upgrading" it in the process. A Maildir client will be able to use a
     Maildir++ just like a regular Maildir. Of course, a plain Maildir
     client won't be able to enforce a quota, and won't be able to access
     messages stored in folders.


     Folders are created as subdirectories under the main Maildir. The name
     of the subdirectory always starts with a period. For example, a folder
     named "Important" will be a subdirectory called ".Important". You
     can't have subdirectories that start with two periods.


     A Maildir++ client ignores anything in the main Maildir that starts
     with a period, but is not a subdirectory.


     Each subdirectory is a fully-fledged Maildir of its own, that is you
     have .Important/tmp, .Important/new, and .Important/cur. Everything
     that applies to the main Maildir applies equally well to the
     subdirectory, including automatically cleaning up old files in tmp. A
     Maildir++ enhancement is that a message can be moved between folders
     and/or the main Maildir simply by moving/renaming the file (into the
     cur subdirectory of the destination folder). Therefore, the entire
     Maildir++ must reside on the same filesystem.


     Within each subdirectory there's an empty file, maildirfolder. Its
     existence tells the mail delivery agent that this Maildir is a really
     a folder underneath a parent Maildir++.


     Only one special folder is reserved: Trash (subdirectory .Trash).
     Instead of marking deleted messages with the D flag, Maildir++ clients
     move the message into the Trash folder. Maildir++ readers are
     responsible for expunging messages from Trash after a system-defined
     retention interval.


     When a Maildir++ reader sees a message marked with a D flag it may at
     its option: remove the message immediately, move it into Trash, or
     ignore it.


     Can folders have subfolders, defined in a recursive fashion? The
     answer is no. If you want to have a client with a hierarchy of
     folders, emulate it. Pick a hierarchy separator character, say ":".
     Then, folder foo/bar is subdirectory .foo:bar.


     This is all that there's to say about folders. The rest of this
     document deals with quotas.


     The purpose of quotas is to temporarily disable a Maildir, if it goes
     over the quota. There is one and only major goal that this quota
     implementation tries to achieve:
       * Place as little overhead as possible on the mail system that's
         delivering to the Maildir++


     That's it. To achieve that goal, certain compromises are made:
       * Mail delivery will stop as soon as possible after Maildir++'s size
         goes over quota. Certain race conditions may happen with Maildir++
         going a lot over quota, in rare circumstances. That is taken into
         account, and the situation will eventually resolve itself, but you
         should not simply take your systemwide quota, multiply it by the
         number of mail accounts, and allocate that much disk space. Always
         leave room to spare.
       * How well the quota mechanism will work will depend on whether or
         not everything that accesses the Maildir++ is a Maildir++ client.
         You can have a transition period where some of your mail clients
         are just Maildir clients, and things should run more or less well.
         There will be some additional load because the size of the Maildir
         will be recalculated more often, but the additional load shouldn't
         be noticeable.


     This won't be a perfect solution, but it will hopefully be good
     enough. Maildirs are simply designed to rely on the filesystem to
     enforce individual quotas. If a filesystem-based quota works for you,
     use it.


     A Maildir++ may contain the following additional file: maildirsize.


Contents of maildirsize

     maildirsize contains two or more lines terminated by newline
     characters.


     The first line contains a copy of the quota definition as used by the
     system's mail server. Each application that uses the maildir must know
     what it's quota is. Instead of configuring each application with the
     quota logic, and making sure that every application's quota definition
     for the same maildir is exactly the same, the quota specification used
     by the system mail server is saved as the first line of the
     maildirsize file. All other application that enforce the maildir quota
     simply read the first line of maildirsize.


     The quota definition is a list, separate by commas. Each member of the
     list consists of an integer followed by a letter, specifying the
     nature of the quota. Currently defined quota types are 'S' - total
     size of all messages, and 'C' - the maximum count of messages in the
     maildir. For example, 10000000S,1000C specifies a quota of 10,000,000
     bytes or 1,000 messages, whichever comes first.


     All remaining lines all contain two integers separated by a single
     space. The first integer is interpreted as a byte count. The second
     integer is interpreted as a file count. A Maildir++ writer can add up
     all byte counts and file counts from maildirsize and enforce a quota
     based either on number of messages or the total size of all the
     messages.


Calculating maildirsize

     In most cases, changes to maildirsize are recorded by appending an
     additional line. Under some conditions maildirsize has to be
     recalculated from scratch. These conditions are defined later. This is
     the procedure that's used to recalculate maildirsize:
      1. If we find a maildirfolder within the directory, we're delivering
         to a folder, so back up to the parent directory, and start again.
      2. Read the contents of the new and cur subdirectories. Also, read
         the contents of the new and cur subdirectories in each Maildir++
         folder, except Trash. Before reading each subdirectory, stat() the
         subdirectory itself, and keep track of the latest timestamp you
         get.
      3. If the filename of each message is of the form xxxxx,S=nnnnn or
         xxxxx,S=nnnnn:xxxxx where "xxxxx" represents arbitrary text, then
         use nnnnn as the size of the file (which will be conveniently
         recorded in the filename by a Maildir++ writer, within the
         conventions of filename naming in a Maildir). If the message was
         not written by a Maildir++ writer, stat() it to obtain the message
         size. If stat() fails, a race condition removed the file, so just
         ignore it and move on to the next one.
      4. When done, you have the grand total of the number of messages and
         their total size. Create a new maildirsize by: creating the file
         in the tmp subdirectory, observing the conventions for writing to
         a Maildir. Then rename the file as maildirsize.Afterwards, stat
         all new and cur subdirectories again. If you find a timestamp
         later than the saved timestamp, REMOVE maildirsize.
      5. Before running this calculation procedure, the Maildir++ user
         wanted to know the size of the Maildir++, so return the calculated
         values. This is done even if maildirsize was removed.


Calculating the quota for a Maildir++

     This is the procedure for reading the contents of maildirsize for the
     purpose of determine if the Maildir++ is over quota.
      1. If maildirsize does not exist, or if its size is at least 5120
         bytes, recalculate it using the procedure defined above, and use
         the recalculated numbers. Otherwise, read the contents of
         maildirsize, and add up the totals.
      2. The most efficient way of doing this is to: open maildirsize, then
         start reading it into a 5120 byte buffer (some broken NFS
         implementations may return less than 5120 bytes read even before
         reaching the end of the file). If we fill it, which, in most
         cases, will happen with one read, close it, and run the
         recalculation procedure.
      3. In many cases the quota calculation is for the purpose of adding
         or removing messages from a Maildir++, so keep the file descriptor
         to maildirsize open. A file descriptor will not be available if
         quota recalculation ended up removing maildirsize due to a race
         condition, so the caller may or may not get a file descriptor
         together with the Maildir++ size.
      4. If the numbers we got indicated that the Maidlir++ is over quota,
         some additional logic is in order: if we did not recalculate
         maildirsize, if the numbers in maildirsize indicated that we are
         over quota, then if maildirsize was more than one line long, or if
         the timestamp on maildirsize indicated that it's at least 15
         minutes old, throw out the totals, and recalculate maildirsize
         from scratch.


     Eventually the 5120 byte limitation will always cause maildirsize to
     be recalculated, which will compensate for any race conditions which
     previously threw off the totals. Each time a message is delivered or
     removed from a Maildir++, one line is added to maildirsize (this is
     described below in greater detail). Most messages are less than 10K
     long, so each line appended to maildirsize will be either between
     seven and nine bytes long (four bytes for message count, space, digit
     1, newline, optional minus sign in front of both counts if the message
     was removed). This results in about 640 Maildir++ operations before a
     recalculation is forced. Since most messages are added once and
     removed once from a Maildir, expect recalculation to happen
     approximately every 320 messages, keeping the overhead of a
     recalculation to a minimum. Even if most messages include large
     attachments, most attachments are less than 100K long, which brings
     down the average recalculation frequency to about 150 messages.


     Also, the effect of having non-Maildir++ clients accessing the
     Maildir++ is reduced by forcing a recalculation when we're potentially
     over quota. Even if non-Maildir++ clients are used to remove messages
     from the Maildir, the fact that the Maildir++ is still over quota will
     be verified every 15 minutes.


Delivering to a Maildir++

     Delivering to a Maildir++ is like delivering to a Maildir, with the
     following exceptions:
      1. Follow the usual Maildir conventions for naming the filename used
         to store the message, except that append ,S=nnnnn to the name of
         the file, where nnnnn is the size of the file. This eliminates the
         need to stat() most messages when calculating the quota. If the
         size of the message is not known at the beginning, append ,S=nnnnn
         when renaming the message from tmp to new.
      2. As soon as the size of the message is known (hopefully before it
         is written into tmp), calculate Maildir++'s quota, using the
         procedure defined previously. If the message is over quota, back
         out, cleaning up anything that was created in tmp.
      3. If a file descriptor to maildirsize was opened for us, after
         moving the file from tmp to new append a line to the file
         containing the message size, and "1".


Reading from a Maildir++

     Maildir++ readers should mind the following additional tasks:
      1. Make sure to create the maildirfolder file in any new folders
         created within the Maildir++.
      2. When moving a message to the Trash folder, append a line to
         maildirsize, containing a negative message size and a '-1'.
      3. When moving a message from the Trash folder, follow the steps
         described in "Delivering to Maildir++", as far as quota logic
         goes. That is, refuse to move messages out of Trash if the
         Maildir++ is over quota.
      4. Moving a message between other folders carries no additional
         requirements.



  Index: Ext-mbx-locking
  ====================================================================
       UNIX Advisory File Locking Implications on c-client
              Mark Crispin, 28 November 1995



      THIS DOCUMENT HAS BEEN UPDATED TO REFLECT THE CODE IN THE
      IMAP-4 TOOLKIT AS OF NOVEMBER 28, 1995.  SOME STATEMENTS
      IN THIS DOCUMENT DO NOT APPLY TO EARLIER VERSIONS OF THE
      IMAP TOOLKIT.


INTRODUCTION

       Advisory locking is a mechanism by which cooperating processes
  can signal to each other their usage of a resource and whether or not
  that usage is critical.  It is not a mechanism to protect against
  processes which do not cooperate in the locking.


       The most basic form of locking involves a counter.  This counter
  is -1 when the resource is available.  If a process wants the lock, it
  executes an atomic increment-and-test-if-zero.  If the value is zero,
  the process has the lock and can execute the critical code that needs
  exclusive usage of a resource.  When it is finished, it sets the lock
  back to -1.  In C terms:


    while (++lock)        /* try to get lock */
      invoke_other_threads ();    /* failed, try again */
     .
     .    /* critical code  here */
     .
    lock = -1;            /* release lock */


       This particular form of locking appears most commonly in
  multi-threaded applications such as operating system kernels.  It
  makes several presumptions:
   (1) it is alright to keep testing the lock (no overflow)
   (2) the critical resource is single-access only
   (3) there is shared writeable memory between the two threads
   (4) the threads can be trusted to release the lock when finished


       In applications programming on multi-user systems, most commonly
  the other threads are in an entirely different process, which may even
  be logged in as a different user.  Few operating systems offer shared
  writeable memory between such processes.


       A means of communicating this is by use of a file with a mutually
  agreed upon name.  A binary semaphore can be passed by means of the
  existance or non-existance of that file, provided that there is an
  atomic means to create a file if and only if that file does not exist.
  In C terms:


                  /* try to get lock */
    while ((fd = open ("lockfile",O_WRONLY|O_CREAT|O_EXCL,0666)) < 0)
      sleep (1);            /* failed, try again */
    close (fd);            /* got the lock */
     .
     .    /* critical code  here */
     .
    unlink ("lockfile");        /* release lock */


       This form of locking makes fewer presumptions, but it still is
  guilty of presumptions (2) and (4) above.  Presumption (2) limits the
  ability to have processes sharing a resource in a non-conflicting
  fashion (e.g. reading from a file).  Presumption (4) leads to
  deadlocks should the process crash while it has a resource locked.


       Most modern operating systems provide a resource locking system
  call that has none of these presumptions.  In particular, a mechanism
  is provided for identifying shared locks as opposed to exclusive
  locks.  A shared lock permits other processes to obtain a shared lock,
  but denies exclusive locks.  In other words:


      current state        want shared    want exclusive
      -------------        -----------    --------------
       unlocked         YES         YES
       locked shared         YES         NO
       locked exclusive     NO         NO


       Furthermore, the operating system automatically relinquishes all
  locks held by that process when it terminates.


       A useful operation is the ability to upgrade a shared lock to
  exclusive (provided there are no other shared users of the lock) and
  to downgrade an exclusive lock to shared.  It is important that at no
  time is the lock ever removed; a process upgrading to exclusive must
  not relenquish its shared lock.


       Most commonly, the resources being locked are files.  Shared
  locks are particularly important with files; multiple simultaneous
  processes can read from a file, but only one can safely write at a
  time.  Some writes may be safer than others; an append to the end of
  the file is safer than changing existing file data.  In turn, changing
  a file record in place is safer than rewriting the file with an
  entirely different structure.



FILE LOCKING ON UNIX

       In the oldest versions of UNIX, the use of a semaphore lockfile
  was the only available form of locking.  Advisory locking system calls
  were not added to UNIX until after the BSD vs. System V split.  Both
  of these system calls deal with file resources only.


       Most systems only have one or the other form of locking.  AIX
  emulates the BSD form of locking as a jacket into the System V form.
  Ultrix and OSF/1 implement both forms.
  ?
  BSD


       BSD added the flock() system call.  It offers capabilities to
  acquire shared lock, acquire exclusive lock, and unlock.  Optionally,
  the process can request an immediate error return instead of blocking
  when the lock is unavailable.



FLOCK() BUGS

       flock() advertises that it permits upgrading of shared locks to
  exclusive and downgrading of exclusive locks to shared, but it does so
  by releasing the former lock and then trying to acquire the new lock.
  This creates a window of vulnerability in which another process can
  grab the exclusive lock.  Therefore, this capability is not useful,
  although many programmers have been deluded by incautious reading of
  the flock() man page to believe otherwise.  This problem can be
  programmed around, once the programmer is aware of it.


       flock() always returns as if it succeeded on NFS files, when in
  fact it is a no-op.  There is no way around this.


       Leaving aside these two problems, flock() works remarkably well,
  and has shown itself to be robust and trustworthy.
  ?
  SYSTEM V/POSIX


       System V added new functions to the fnctl() system call, and a
  simple interface through the lockf() subroutine.  This was
  subsequently included in POSIX.  Both offer the facility to apply the
  lock to a particular region of the file instead of to the entire file.
  lockf() only supports exclusive locks, and calls fcntl() internally;
  hence it won't be discussed further.


       Functionally, fcntl() locking is a superset of flock(); it is
  possible to implement a flock() emulator using fcntl(), with one minor
  exception: it is not possible to acquire an exclusive lock if the file
  is not open for write.


       The fcntl() locking functions are: query lock station of a file
  region, lock/unlock a region, and lock/unlock a region and block until
  have the lock.  The locks may be shared or exclusive.  By means of the
  statd and lockd daemons, fcntl() locking is available on NFS files.


       When statd is started at system boot, it reads its /etc/state
  file (which contains the number of times it has been invoked) and
  /etc/sm directory (which contains a list of all remote sites which are
  client or server locking with this site), and notifies the statd on
  each of these systems that it has been restarted.  Each statd then
  notifies the local lockd of the restart of that system.


       lockd receives fcntl() requests for NFS files.  It communicates
  with the lockd at the server and requests it to apply the lock, and
  with the statd to request it for notification when the server goes
  down.  It blocks until all these requests are completed.


       There is quite a mythos about fcntl() locking.


       One religion holds that fcntl() locking is the best thing since
  sliced bread, and that programs which use flock() should be converted
  to fcntl() so that NFS locking will work.  However, as noted above,
  very few systems support both calls, so such an exercise is pointless
  except on Ultrix and OSF/1.


       Another religion, which I adhere to, has the opposite viewpoint.



FCNTL() BUGS

       For all of the hairy code to do individual section locking of a
  file, it's clear that the designers of fcntl() locking never
  considered some very basic locking operations.  It's as if all they
  knew about locking they got out of some CS textbook with not
  investigation of real-world needs.


       It is not possible to acquire an exclusive lock unless the file
  is open for write.  You could have append with shared read, and thus
  you could have a case in which a read-only access may need to go
  exclusive.  This problem can be programmed around once the programmer
  is aware of it.


       If the file is opened on another file designator in the same
  process, the file is unlocked even if no attempt is made to do any
  form of locking on the second designator.  This is a very bad bug.  It
  means that an application must keep track of all the files that it has
  opened and locked.


       If there is no statd/lockd on the NFS server, fcntl() will hang
  forever waiting for them to appear.  This is a bad bug.  It means that
  any attempt to lock on a server that doesn't run these daemons will
  hang.  There is no way for an application to request flock() style
  ``try to lock, but no-op if the mechanism ain't there''.


       There is a rumor to the effect that fcntl() will hang forever on
  local files too if there is no local statd/lockd.  These daemons are
  running on mailer.u, although they appear not to have much CPU time.
  A useful experiment would be to kill them and see if imapd is affected
  in any way, but I decline to do so without an OK from UCS!  ;-) If
  killing statd/lockd can be done without breaking fcntl() on local
  files, this would become one of the primary means of dealing with this
  problem.


       The statd and lockd daemons have quite a reputation for extreme
  fragility.  There have been numerous reports about the locking
  mechanism being wedged on a systemwide or even clusterwide basis,
  requiring a reboot to clear.  It is rumored that this wedge, once it
  happens, also blocks local locking.  Presumably killing and restarting
  statd would suffice to clear the wedge, but I haven't verified this.


       There appears to be a limit to how many locks may be in use at a
  time on the system, although the documentation only mentions it in
  passing.  On some of their systems, UCS has increased lockd's ``size
  of the socket buffer'', whatever that means.
  ?
  C-CLIENT USAGE


       c-client uses flock().  On System V systems, flock() is simulated
  by an emulator that calls fcntl().  This emulator is provided by some
  systems (e.g. AIX), or uses c-client's flock.c module.



BEZERK AND MMDF

       Locking in the traditional UNIX formats was largely dictated by
  the status quo in other applications; however, additional protection
  is added against inadvertantly running multiple instances of a
  c-client application on the same mail file.


       (1) c-client attempts to create a .lock file (mail file name with
  ``.lock'' appended) whenever it reads from, or writes to, the mail
  file.  This is an exclusive lock, and is held only for short periods
  of time while c-client is actually doing the I/O.  There is a 5-minute
  timeout for this lock, after which it is broken on the presumption
  that it is a stale lock.  If it can not create the .lock file due to
  an EACCES (protection failure) error, it once silently proceeded
  without this lock; this was for systems which protect /usr/spool/mail
  from unprivileged processes creating files.  Today, c-client reports
  an error unless it is built otherwise.  The purpose of this lock is to
  prevent against unfavorable interactions with mail delivery.


       (2) c-client applies a shared flock() to the mail file whenever
  it reads from the mail file, and an exclusive flock() whenever it
  writes to the mail file.  This lock is freed as soon as it finishes
  reading.  The purpose of this lock is to prevent against unfavorable
  interactions with mail delivery.


       (3) c-client applies an exclusive flock() to a file on /tmp
  (whose name represents the device and inode number of the file) when
  it opens the mail file.  This lock is maintained throughout the
  session, although c-client has a feature (called ``kiss of death'')
  which permits c-client to forcibly and irreversibly seize the lock
  from a cooperating c-client application that surrenders the lock on
  demand.  The purpose of this lock is to prevent against unfavorable
  interactions with other instances of c-client (rewriting the mail
  file).


       Mail delivery daemons use lock (1), (2), or both.  Lock (1) works
  over NFS; lock (2) is the only one that works on sites that protect
  /usr/spool/mail against unprivileged file creation.  Prudent mail
  delivery daemons use both forms of locking, and of course so does
  c-client.


       If only lock (2) is used, then multiple processes can read from
  the mail file simultaneously, although in real life this doesn't
  really change things.  The normal state of locks (1) and (2) is
  unlocked except for very brief periods.



TENEX AND MTX

       The design of the locking mechanism of these formats was
  motivated by a design to enable multiple simultaneous read/write
  access.  It is almost the reverse of how locking works with
  bezerk/mmdf.


       (1) c-client applies a shared flock() to the mail file when it
  opens the mail file.  It upgrades this lock to exclusive whenever it
  tries to expunge the mail file.  Because of the flock() bug that
  upgrading a lock actually releases it, it will not do so until it has
  acquired an exclusive lock (2) first.  The purpose of this lock is to
  prevent against expunge taking place while some other c-client has the
  mail file open (and thus knows where all the messages are).


       (2) c-client applies a shared flock() to a file on /tmp (whose
  name represents the device and inode number of the file) when it
  parses the mail file.  It applies an exclusive flock() to this file
  when it appends new mail to the mail file, as well as before it
  attempts to upgrade lock (1) to exclusive.  The purpose of this lock
  is to prevent against data being appended while some other c-client is
  parsing mail in the file (to prevent reading of incomplete messages).
  It also protects against the lock-releasing timing race on lock (1).
  ?
  OBSERVATIONS


       In a perfect world, locking works.  You are protected against
  unfavorable interactions with the mailer and against your own mistake
  by running more than one instance of your mail reader.  In tenex/mtx
  formats, you have the additional benefit that multiple simultaneous
  read/write access works, with the sole restriction being that you
  can't expunge if there are any sharers of the mail file.


       If the mail file is NFS-mounted, then flock() locking is a silent
  no-op.  This is the way BSD implements flock(), and c-client's
  emulation of flock() through fcntl() tests for NFS files and
  duplicates this functionality.  There is no locking protection for
  tenex/mtx mail files at all, and only protection against the mailer
  for bezerk/mmdf mail files.  This has been the accepted state of
  affairs on UNIX for many sad years.


       If you can not create .lock files, it should not affect locking,
  since the flock() locks suffice for all protection.  This is, however,
  not true if the mailer does not check for flock() locking, or if the
  the mail file is NFS-mounted.


       What this means is that there is *no* locking protection at all
  in the case of a client using an NFS-mounted /usr/spool/mail that does
  not permit file creation by unprivileged programs.  It is impossible,
  under these circumstances, for an unprivileged program to do anything
  about it.  Worse, if EACCES errors on .lock file creation are no-op'ed
  , the user won't even know about it.  This is arguably a site
  configuration error.


       The problem with not being able to create .lock files exists on
  System V as well, but the failure modes for flock() -- which is
  implemented via fcntl() -- are different.


       On System V, if the mail file is NFS-mounted and either the
  client or the server lacks a functioning statd/lockd pair, then the
  lock attempt would have hung forever if it weren't for the fact that
  c-client tests for NFS and no-ops the flock() emulator in this case.
  Systemwide or clusterwide failures of statd/lockd have been known to
  occur which cause all locks in all processes to hang (including
  local?).  Without the special NFS test made by c-client, there would
  be no way to request BSD-style no-op behavior, nor is there any way to
  determine that this is happening other than the system being hung.


       The additional locking introduced by c-client was shown to cause
  much more stress on the System V locking mechanism than has
  traditionally been placed upon it.  If it was stressed too far, all
  hell broke loose.  Fortunately, this is now past history.
  ?
  TRADEOFFS


       c-client based applications have a reasonable chance of winning
  as long as you don't use NFS for remote access to mail files.  That's
  what IMAP is for, after all.  It is, however, very important to
  realize that you can *not* use the lock-upgrade feature by itself
  because it releases the lock as an interim step -- you need to have
  lock-upgrading guarded by another lock.


       If you have the misfortune of using System V, you are likely to
  run into problems sooner or later having to do with statd/lockd.  You
  basically end up with one of three unsatisfactory choices:
      1) Grit your teeth and live with it.
      2) Try to make it work:
         a) avoid NFS access so as not to stress statd/lockd.
         b) try to understand the code in statd/lockd and hack it
            to be more robust.
         c) hunt out the system limit of locks, if there is one,
            and increase it.    Figure on at least two locks per
            simultaneous imapd process and four locks per Pine
            process.    Better yet, make the limit be 10 times the
            maximum number of processes.
         d) increase the socket buffer (-S switch to lockd) if
            it is offered.  I don't know what this actually does,
            but giving lockd more resources to do its work can't
            hurt.  Maybe.
      3) Decide that it can't possibly work, and turn off the
         fcntl() calls in your program.
      4) If nuking statd/lockd can be done without breaking local
         locking, then do so.  This would make SVR4 have the same
         limitations as BSD locking, with a couple of additional
         bugs.
      5) Check for NFS, and don't do the fcntl() in the NFS case.
         This is what c-client does.


       Note that if you are going to use NFS to access files on a server
  which does not have statd/lockd running, your only choice is (3), (4),
  or (5).  Here again, IMAP can bail you out.


       These problems aren't unique to c-client applications; they have
  also been reported with Elm, Mediamail, and other email tools.


       Of the other two SVR4 locking bugs:


       Programmer awareness is necessary to deal with the bug that you
  can not get an exclusive lock unless the file is open for write.  I
  believe that c-client has fixed all of these cases.


       The problem about opening a second designator smashing any
  current locks on the file has not been addressed satisfactorily yet.
  This is not an easy problem to deal with, especially in c-client which
  really doesn't know what other files/streams may be open by Pine.


       Aren't you so happy that you bought an System V system?


Index: LongTermIssues
====================================================================
$Cambridge: exim/exim-doc/doc-misc/LongTermIssues,v 1.1 2004/10/08 10:38:47 ph10 Exp $

Exim Long Term Issues
---------------------

I restarted this list from scratch for Exim 4. I amalgamated it with another
list when creating the CVS repository (October 2004). But it still probably
needs a substantial spring clean. Some of it is very old now.


AUTOCONF
--------

Somebody once tried to \(autoconf)\ Exim, but found it too big a job. I now
have some experience with using \(autoconf)\ for PCRE, and I think maybe some
use could be made of it. I don't, however, believe that \(all)\ Exim build-time
configuration should be done that way. The reason is that, unlike something
like PCRE, there is quite a lot of information that is "user choice". Giving it
all as options to a \(configure)\ command does not seem the best way of doing
things.

Whenever I build something that needs more than a couple of obvious options to
\(configure)\, I always save them in a file anyway, so I know what I did for
next time. Therefore, I think it is sensible to retain the current Local file
structure for all the user choice configuration.

However, it might be helpful to use \(autoconf)\ to dig out various bits of
information about the operating system. At present, the \(OS/Makefile-*)\ files
have hard-wired settings, and maybe this information could be figured out by
running \(autoconf)\, which would save having to keep maintaining these files.

I would arrange things so that \(configure)\ is run automatically the first
time that \(make)\ is run, but it would be possible to run it manually first,
to override defaults. (For example, if you have both \(cc)\ and \(gcc)\
installed on your system, as I do, you need to be able to specify which to
use.) I will need to do some experiments to see exactly how this would work.


EXIMON and other utilities
--------------------------

  . Consider optionally making it possible to link with something other than
    Athena widgets - for example, gtk. Or indeed re-write the whole thing!



GENERAL
-------

  . Convert os.c into a directory of separate functions, with the macro
    switches defined elsewhere. Then make it into a library.


  . Use a pointer to an address structure for expanding $domain etc, to make it
    easier to save/restore this collection of variables. But note that $domain
    and $local_part aren't always in an address. Check out when these are set.
    Note also the new $address_data possibility.


. Spool_in and spool_out - speed up by using a table?

  . Find a more compact way of encoding the options interpretation, and also of
    checking for incompatible options.


  . Find a more compact way of passing an open SMTP channel without having
    to use options. What about the TLS state information? Could use a pipe to
    pass more data.


  . Some people have suggested separately loadable modules. But do all systems
    have them? Is this going too far for just a few specialist users? In
    particular, people want to be able to replace the logging with his own code.
    Can we arrange this without going for the separately loaded modules? (cf the
    incoming checking code.)


  . SIGHUP the daemon - don't close the sockets; instead pass a list of them
    somewhere for the new daemon to pick up. Iff started by exim or root, of
    course. There might be quite a long list of them - argv might not be the best
    idea. If this were done, then a non-setuid exim daemon could be SIGHUPped.


  . Parallel deliveries. Currently dead host information doesn't get propagated
    between them very well. Is there anyway this could be improved?


  . In some environments the use of gethostbyname() seems to cause problems.
    Check out its use, and see if having a "force DNS" option could be helpful.
    But people would have to know what they were doing.


  . accept_max_per_host is a slow, linear search. If smtp_accept_max is large,
    this can be very slow. Is there some way we can speed this up? Some kind of
    index based on the IP address? Remember, this is in the daemon, so it must
    not consume store.


  . Change the names of all the pcre_ stuff to, say, PCRE_ so that Exim can be
    linked with libraries or whatever that also use an external PCRE library.


  . Look at code in pidentd for running Exim in wait mode from inetd and re-using
    the socket. This would allow it to run more tidily as non-root.


  . Think up some scheme for checking for orphan files in the spool directories.
    Perhaps -bp should always do it, but it would be nice to have it done
    automatically now and again. Maybe we just leave this for a cron job? Perhaps
    a new -bx, e.g. -bpck or something. Better, perhaps, is a separate Perl
    script. Orphan = a file that is over 24h old (or 1s when test harness) and
    either doesn't end in -D or -H, or is a -D without a matching -H (or vice
    versa).


  . Make set_process_info buffer bigger, and put the overflowed message at the
    end, thereby leaving the start.


  . Swamping with delays in checking for reserved hosts - the connections are
    counted in the total allowed. Can we improve on this somehow? Maybe shared
    memory can help here. Think about different states and different limits.


  . Lists that must use colons: can we check for other cases, and fix them up
    before passing them on? Is it worth it?


. Linux for S/390 - create configuration?

  . Process receiving error message fails - can we get more info, such as the
    stdout/stderr?


  . dbmbuild - if renaming one of .dir/.pag fails, reinstate the other. Should
    there be a lock?


  . Write a script to check for format problems in the source - formats that are
    not fixed strings and are built from outside code.


  . freeze_tell: Don't if message is a bounce message containing From: the local
    machine - even if the bounce comes from another host.


  . Add additional data into the "frozen" log message at end of delivery, e.g. if
    remote host was the local host or whatever. At least some cross referencing.


  . Someone had a requirement to install the Exim binary in a different place to
    the utilities, etc. Also, for different builds on the same host and
    architecture.


. Include (part of?) the ppid in the message id? Or a random number?

  . Re-implement the code in readconf that reads error names for retry rules.
    Make it use a table for most of the error types. Then see if we can usefully
    add any additional error types.


  . Should there be "exim -bP acls" etc? It would mean inventing some kind of
    "hide" facility within the ACL syntax.


  . VERY LONG TERM: the message ID is too small now, with the recent changes to
    cram in the sub-second time. It would be a big project to extend it; Exim
    would have to recognize both forms for a while, and become stable, before
    generating the new form. Probably a runtime switch needed. The new form needs
    at least microsecond time (or more?) and should probably cope with 64-bit
    pids, just to be safe (or leave expansion space that could be used for that).
    It should also be able to hold big enough things in base 36.


. Take a look at libexec.

. Sort out the stcncpy/strlcpy issue once and for all. Time things.

  . Error in transport filter. See test 407. All 3 processes see errors - which
    one should be noticed? Transport_filter_temp_errors may be needed.


. Think about 5xx thresholds -- too many and you're out. What about 4xx?

  . autoreply - should it call /usr/sbin/sendmail? Provide a way of not passing
    -C and -D when creating the message ('cause it won't be privileged).


. Strings containing \000 - anything we can do?

  . OpenSSL - can we pass an opened file for certificate? Repeatedly?
    Otherwise pre-initialize while root? There do seem to be functions for
    manipulating certificates, but documentation is scarce. Can we just load the
    certificate in as root in the server?


  . Consider using poll() to close unwanted fds. Is this efficient? Perhaps it
    doesn't matter for the daemon.


  . On a 64-bit system there are some cast warnings for casting addresses to
    ints. Either we must find a way of not warning, or we'll have to use unions
    to get round it.


. Run splint on the source?

  . It has been suggested that rejection because not authenticated should use
    530 and not 550, but this is hard to detect because of the way ACLs work.


  . When there is a sender verify failure, $acl_verify_message contains "sender
    verify failed", not the details of the failure. Should this change? Some of
    the waffly details are added later in smtp_in.c. In the ACL that text is in
    sender_verified_failed->user_message.


  . An empty string for a transport filter currently causes an error. Should it
    ignore? Tricky because of special expansion rules for commands.


  . GFDL for documentation (www.gnu.org/licenses/fdl.html)? The 1.2 version of
    this licence is still quite new (it is dated November 2002) so I think
    waiting for reaction/opinion is the best plan. There are Debian concerns
    about this licence. At very least, no Invariant Sections and no Cover Texts
    can be used.


  . Allow $recipients in other places. Not clear what this value should be if,
    say, the system filter has overridden them. Default would be envelope
    recipients, as now.


End

Index: RFC.conform
====================================================================
$Cambridge: exim/exim-doc/doc-misc/RFC.conform,v 1.1 2004/10/08 10:38:47 ph10 Exp $

Conformance with RFCs
---------------------

Exim is written to follow the rules laid down in the RFCs. However, there are
some circumstances where it either extends what is specified, or chooses not to
follow them strictly, for various reasons. Sometimes variations are controlled
by an option, which may default on or off. This document lists the variations
from the latest email RFCs, and discusses their background and implications.

Last Updated: 25 January 1999


1. RFC 822
----------

The original specification of the format of Internet mail messages is RFC 822,
later clarified and modified by RFC 1123. At the time of writing (January 1999)
a new RFC (currently known as draft-ietf-drums-msg-fmt-07) which updates and
consolidates all the material related to the message format is at a late stage
of drafting, and is expected to become an Internet Standard in due course.

The following is (I hope) a complete list of major variations from the draft
RFC. References in square brackets are to the -07 draft.


1.1 Line termination [2.1, 2.3]
-------------------------------

[Lines are terminated by CRLF; isolated CR and LF are not permitted.]

The CRLF requirement has to be interpreted carefully, because the RFC also says
that it does not cover the internal format "used by sites". Exim keeps messages
on its spool in Unix format, using only LF as the line terminator, and also
does local deliveries using only LF. I believe this is compliant with the RFC,
as these are both "internal formats".

Messages sent out by SMTP have CRLF line terminators. However, isolated CR
characters are treated as any other data characters, because Exim is eight-bit
clean (see 1.2 below).

See 2.1 below for a discussion of line terminators in incoming messages.


1.2 Eight-bit characters [2.1]
------------------------------

[Messages consist of 7-bit characters.]

Exim is eight-bit clean. It does not do any processing of the characters in the
body of a message.


1.3 Maximum line length [2.1, 2.3]
----------------------------------

[The maximum length of a line is 998 characters.]

Exim does not enforce any limit on line length.


1.4 The "phrase" part of an address [3.4]
-----------------------------------------

[The phrase is a sequence of "words"; a word is an "atom" or a quoted string.]

The characters that can be used in an "atom" do not include the full stop
(dot, period). Thus a header line such as

    To: John Q. Public <jqp@???>


is syntactically invalid under a strict interpretation of the RFC because the
dot in the phrase part is not quoted. However, many MTAs do not enforce this
restriction, so Exim was changed to be relaxed about it as well. In fact, the
draft RFC is moving towards allowing this. In section [4.1], which is defining
"obsolete" syntax that programs must accept (but not generate), it says this:

    The period character is added to obs-phrase.


    Note: The period character in obs-phrase is not a form that was allowed
    in earlier versions of this or any other standard. Period (nor any other
    character from specials) was not allowed in phrase because it introduced
    a parsing difficulty distinguishing between phrases and portions of an
    addr-spec (see section 4.4). It appears here because the period
    character is currently used in many messages in the display-name portion
    of addresses, especially for initials in names, and therefore must be
    interpreted properly. In the future, period may appear in the regular
    syntax of phrase.



1.5 Source routed addresses [4.4]
---------------------------------

[Source routed addresses are always enclosed in <>.]

Source routed addresses are declared obsolete in the draft RFC, but MTAs are
still required to handle them. Strictly, a source-routed address must be
enclosed in <> characters, so a header such as

    From: @a,@b:c@d


is syntactally invalid. Exim does not enforce this restriction.


1.6 Local parts [3.4.1]
-----------------------

[Dots in unquoted local parts may not be consecutive or at either end.]

Exim allows unquoted local parts to begin or end with a dot (period, full
stop), and it also permits two consecutive dots in a local part.



2. RFC 821
----------

The original specification of SMTP is RFC 821, later clarified and modified by
RFC 1123. Domain name system requirements and their implications for mail are
covered in RFCs 1035 and 974. A scheme for extending the SMTP protocol is
described in RFC 1869, and there are subsequent RFCs specifying particular
extensions.

At the time of writing (January 1999) a new RFC (currently known as
draft-ietf-drums-smtpupd-09) which updates and consolidates all the material
connected with SMTP message transmission is at a late stage of drafting, and is
expected to become an Internet Standard in due course.

The new draft is written using the terms MUST, SHOULD, and MAY, which, when
written in capital letters, have precise meanings. To quote from the draft:

    "MUST" or "MUST NOT" identify absolute requirements for conformance to
    this specification. Implementations that do not conform to them lie
    outside the scope of this specification and often will not
    interoperate properly with SMTP implementations that do conform.
    Implementations that are fully conforming also adhere to all "SHOULD"
    and "SHOULD NOT" requirements. Implementations that adhere to all
    "MUST" ("MUST NOT") but not to all of these are considered to be
    partially conforming. Such implementations may interoperate properly
    with fully conforming ones and with each other, but this will
    typically be the case only if great care is taken. Consequently, an
    implementation should violate "SHOULD" ("SHOULD NOT") requirements
    only under exceptional and well-understood circumstances.


The implementation of Exim is intended to conform to the spirit of this
paragraph. The following is (I hope) a complete list of major variations
from the draft RFC. In addition to the items listed here, there are other minor
extensions such as the tolerance of white space in places where it is not
strictly permitted by the RFC. References in square brackets are to the -09
draft sections, and brief summaries of the RFC requirement are also given in
square brackets.


2.1 Line termination [2.3.7, 4.1.1.4]
-------------------------------------

[SMTP lines are terminated by CRLF.]

Exim recognizes LF without CR as a line terminator in all forms of input. For
SMTP input, any preceding CR is discarded. An early version of Exim followed
the RFC strictly, and did not recognize LF without CR in SMTP input. However,
it seems that sites on the net send out messages with just LF terminators,
despite the warnings in the RFCs, and other MTAs handle this, so Exim was
changed. However, there is a compile time macro called STRICT_CRLF which can be
set to restore the strict behaviour, though this is undocumented.


2.2 Eight-bit characters [2.4.1]
--------------------------------

[SMTP transmits only 7-bit characters.]

Exim is eight-bit clean, and makes no attempt to modify the data in a message
in any way. In particular, for messages containing characters with the top bit
set, it neither tries to negotiate 8-bit transmission, nor converts such
characters into an encoded form. In other words, it adopts the "just send 8"
strategy. It can be configured to send out 8BITMIME in its response to EHLO
(which it does not do by default), and it recognizes the 8BITMIME keyword on
incoming messages, but neither of these affect its handling of message data.
"Just send 8" is the strategy of a number of MTAs; it is argued that it
achieves what the user wants more often than other strategies.


2.3 Use of EHLO/HELO [3.2]
--------------------------

[Client MTAs should always start with EHLO, not HELO.]

Exim sends EHLO only when it finds the string "ESMTP" in an SMTP greeting
message. If EHLO is refused with a 5xx return code, it then reverts to HELO as
required, but it does not contain logic for converting to HELO on other errors
such as loss of connection or timeout after EHLO. That is one reason why it
doesn't always send EHLO; there are reported to be ancient SMTP servers out
there which collapse on receiving EHLO. (There is also at least one server
whose banner reads "<host name> ignores ESMTP", but it is RFC 821 compliant in
that it responds with 5O0 to EHLO, so Exim successfully reverts to HELO.)


2.4 Closing the connection [4.1.1.10]
-------------------------------------

[Client must wait for response to QUIT before closing the connection.]

Exim closes the connection immediately after sending QUIT, without waiting for
the reply. There was a lot of discussion about this on one of the mailing
lists. The conclusion was that this behaviour is fine on Unix systems, which
have TCP/IP implementations that close down the underlying channel tidily even
when the associated process has terminated. Indeed, not waiting may be
beneficial, as it moves the TIME_WAIT state (waiting to ensure there's no more
data in transit) from the server to the client system. On some other operating
systems (I understand) it is a disaster to terminate the sending process
without waiting for the QUIT response, because all the data about the
connection lives in the client's process space, and is therefore thrown away
before the response arrives. The subsequent arrival of the response then causes
bad behaviour.


2.5 IPv6 address literals [4.1.2]
---------------------------------

[IPv6 address literals are introduced by "IPv6".]

Exim recognizes IPv6 literals as just the colon-separated hexadecimal form of
an IPv6 address, for example 1080:0:0:0:8:800:200C:417A, without the need for a
prefix. At present, it does not even recognize the prefix. When IPv6 becomes
more widespread, Exim will follow whatever the common usage is.


2.6 Underscores in domain names [4.1.2]
---------------------------------------

[Underscores are not legal in domain names.]

RFC 822 allows all characters except specials, space, and controls in domain
names, but the SMTP RFCs are stricter, allowing only letters, digits, and
hyphen. Exim is compliant when checking incoming addresses in SMTP commands,
but it is more relaxed by default when checking domain names that are supplied
by EHLO or HELO commands, because many client workstations get set up with
underscores in their names. There is an option that can be set to cause Exim to
refuse underscores. (There are also options to specify certain hosts from which
it will accept any old junk after EHLO or HELO. Such is the woeful state of
some SMTP clients.)


2.7 Removal of return-path headers [4.4]
----------------------------------------

[Relaying MTAs should not remove return-path.]

Exim removes Return-Path: headers from all messages, if return_path_remove is
set (the default). It does not attempt to determine if it is being a relay or
not. Indeed, for some messages it might be both a relay and a final destination
MTA for the same message.


2.8 Randomizing the order of addresses of multihomed hosts [5]
--------------------------------------------------------------

[Multihomed host addresses should not be randomized.]

Exim does randomize a list of several addresses for a single host, because
caching in resolvers will defeat the round-robinning that many namerservers
use. (Note: this is not the same as randomizing equal-valued MX records. That
is required by the RFC.)


2.9 Handling "MX points to self" [5]
------------------------------------

[MX points to self must be treated as an error.]

The RFC doesn't allow for the possibility of special-purpose routing in the
case when the lowest numbered MX record points to the local host. The default
Exim configuration is compliant, but it is possible to configure Exim to behave
differently, and there are several situations where this can be useful.


2.10 Source routing [6.1]
-------------------------

[Source routes should be stripped.]

The new RFC has moved forward in deprecating source-routed email addresses.
Exim does not strip them down by default, but can be made to do so by setting
collapse_source_routes. However, even when it is not stripping them down, it
does not add host routing to reverse-paths when processing a source-routed
forward-path.


2.11 Loop detection [6.2]
-------------------------

[Loop count for Received: headers should be at least 100.]

Exim's default setting of the received_headers_max option is 30. Most messages
these days seem to accumulate less than half a dozen Received: headers, and
even a couple of forwardings don't bring this anywhere near 30.


2.12 Addition of missing headers [6.3]
--------------------------------------

[Missing headers may be added, and domains qualified, only if client is
identified.]

Exim always adds Message-Id: and Date: headers if these are missing, whatever
the source of the message, and likewise when it expands non-fully-qualified
domains, it does so independently of the message's source.


2.13 Syntax of MAIL and RCPT commands [4.1.1.2, 4.1.1.3]
--------------------------------------------------------

Exim is more relaxed than the RFC requires:

(1) Trailing white space is ignored.

(2) It permits white space after the "FROM" and "TO" keywords.

  (3) It does not insist on the address being enclosed in <> characters. In fact,
      it recognizes addresses in RFC 822 format here, except that domain
      components are restricted to containing only letters, digits, and hyphens.


  (4) Local parts are permitted to contain null components, that is, may start or
      end with an unquoted full stop (period) or contain two consecutive
      unquoted full stops.



2.14 Non-fully-qualified domains [2.3.5]
----------------------------------------

[All domains must be fully qualified.]

A domain that is not fully qualified has some of its trailing components
missing, and is normally a local alias of some sort, for example, just a
single-component host name.

Exim can be configured to "widen" non-fully-qualified domains, either by using
the facilities of the DNS resolver, or by an explicit list of widening strings.
When this is done, it applies to addresses received by SMTP from other hosts,
as well as to locally-originated addresses. Address re-writing could also be
used for this purpose.


2.15 Unqualified addresses [4.1.2]
----------------------------------

[Addresses in SMTP commands must include domains.]

An unqualified address consists of a local part without a domain. Do not
confuse "qualified address" and "qualified domain". A qualified address may
include a non-fully-qualified domain.

There is one exception to the RFC rule: it is required that the unqualified
address "<postmaster>" always be accepted. Apart from this, Exim rejects
domainless addresses in SMTP commands by default, but it can be configured with
a list of hosts and/or networks that are permitted to send addresses without
domains in SMTP commands. Any such address that is accepted (including
<postmaster>) is qualified by adding the value of the qualify_domain option.


2.16 VRFY and EXPN [3.5.1, 3.5.2, 3.5.3, 7.3]
---------------------------------------------

[VRFY and EXPN should be supported.]

Exim does not support VRFY and EXPN by default, but a list of hosts and
networks for which they are permitted can be given.


2.17 Checking of EHLO/HELO commands [4.1.4]
-------------------------------------------

[Client must send EHLO. Server must not refuse message if EHLO/HELO check
fails.]

Exim, as a client, always sends EHLO or HELO (see 2.3 above). As a server, it
does not insist on there having been a valid EHLO or HELO command before the
start of a message transaction. Any EHLO or HELO command that is received is
rejected only if it contains a syntax error. That is, it is never rejected on
the basis of any validation checking that may be performed on the data it
contains.

However, Exim can be configured to insist that (a) there is valid EHLO/HELO
command before any message transaction and (b) the domain in that command
matches the domain obtained by looking up the IP address of the sending host.
It is possible to specify exception lists of hosts and/or networks for which
this check does not apply.


2.18 Format of delivery error messages [3.7]
--------------------------------------------

[Standard report formats should be used if possible.]

Exim's delivery failure reports do not conform to the format described in RFC
1894.


## End ##

Index: TexiNotes
====================================================================
$Cambridge: exim/exim-doc/doc-misc/TexiNotes,v 1.1 2004/10/08 10:38:47 ph10 Exp $

Notes for conversion of sgcal input into Texinfo input
------------------------------------------------------

(Dated 6 August 1996)

The escape character is @. Only @ and curly brackets are sensitive. Get them in
by @@ @{ and @} if required.

@: after a dot that is not a sentence end.

@. instead of . if sentence ends with capital letter

@copyright{} for copyright

@minus{} is a slighly longer minus sign

Input file ends with .texinfo usually.

MUST start the file with

    \input texinfo
    @c %**start of header
    @setfilename INFO-FILE-NAME
    @settitle NAME_OF_MANUAL
    $c %**end of header


Then, typically

    @ifinfo
    summary and copyright
    @end ifinfo


Followed by

    @titlepage
    title and copyright
    @end titlepage


Then the top node and master menu - for info file only

    @node       Top,       First Chapter, (dir),   (dir)
    @comment    node-name  next,          previous,   up
    @top


    @menu
    * First Chapter::       The first chapter is the
                            only chapter in the sample
    * Concept Index::       An index
    @end menu



Then the body

    @node    First Chapter,  Concept Index,  Top,    Top
    @comment  node-name       next,         previous, up
    @chapter First Chapter
    @cindex Sample index entry


    This is the contents of the first chapter
    @cindex Another sample index



Then stuff about indexes and tables of contents

    @node    Concept Index,     ,  First Chapter,   Top
    @unnumbered Concept Index


    @printindex cp


    @contents


MUST end the file with

    @bye



. NEWLINE AND NO-FILL MODE

    @page for new page
    @* forces a line break



. LINE CENTERING

    @center stuff



. ROMAN, ITALIC, BOLD ITALIC, SMALL CAPS

    @code{...} for 'code' =>  `...'  in info
    @file{...} for file names => `...' in info
    @samp{...} for sample text => `...' in info
    @var{...}  for variable => caps in info
    @dfn{...} defining a term => double quotes in info
    @emph{...} produces italic
    @strong{...} produces bold
    @sc{...} small caps  but with letters in lower case.
    @i   italic  )
    @b   bold    ) no effect on info file
    @r   roman   )



. TABBING

. CHAPTERS & SECTIONS

    @chapter <title>
    @unnumbered <title> is an unnumbered chapter
    @section




. SECTION

. FANCY VS PLAIN

    @iftex   ...  @end iftex   for printed only; likewise @ifinfo   ... @end ifinfo



. LEAVING BLANK SPACE

    @sp 10



. EM & NEM

    no can no


. DISPLAY ASIS

    @example  ...  @end example
    @display  ...  @end display     no change of font => rm



. COMMENTS

    @comment or @c introduces comment lines



. NUMBERED LISTS

    @enumerate
    @item
    first item


    @item
    second
    @end enumerate




. BULLETED LISTS

    @itemize @bullet
    ...




. CROSS REFERENCES

    @xref   start sentence
    @ref{name}
    @pxref (parenthesized)


    5 args: node name (required), cross-ref name, topic description, name of
    info file, name of printed manual.




. TABLES

    @table   for two-column tables
    @table @asis


    @item  first column
    second column


    @item ...




. INDEX

    @cindex    concept index
    @findex    function index
    @vindex    variable index
    @kindex    key index
    @pindex    program index
    @tindex    data type index


***

Index: WishList
====================================================================
$Cambridge: exim/exim-doc/doc-misc/WishList,v 1.1 2004/10/08 10:38:47 ph10 Exp $

EXIM 4 WISH LIST
----------------

Even when it was first released, Exim 4 had a Wish List because not all the
things suggested for it were implemented. The list has not stopped growing...

Another reason it is so long is that I have retained some items from the Exim 3
Wish List that never got implemented, but which seem reasonable possibilities
for later addition to Exim 4.

I have guessed at the amount of work involved, and categorized the items as
Tiny, Small, Medium, Large, or Unknown. The guesses are not based on any
detailed investigation, so must be taken as very rough.


  ------------------------------------------------------------------------------
  ------------------------------------------------------------------------------
  -----                Retained from the Exim 3 Wish List                 ------
  ------------------------------------------------------------------------------
  ------------------------------------------------------------------------------


(10) 13-Jul-98 M more flexibility for pipe returns
Ben Smithurst

The ability to specify more precisely what happens concerning the return code
from the pipe and the presence/absence of STDOUT/STDERR is requested. The
particular configuration that was requested was:

> if the command exited EX_OK, *and* produced nothing on STDOUT or
> STDERR, it succeeded...
> if the command exited EX_TEMPFAIL, defer, regardless of
> STDOUT/STDERR...
> otherwise freeze the message (this will get my attention by way of
> freeze_tell_mailmaster)...

------------------------------------------------------------------------------

(11) 17-Jul-98 G support for DSN
Andy Mell

It is unclear to me how this should work in the presence of aliases and
forwarding. Local deliveries would have to explicitly configured as deliveries
or relaying or whatever. A substantial amount of code is probably needed.

Jeffrey Goldberg
I have nothing to add except to say that for many of the reasons you've
stated, I don't think that DSN is coherent enough to be worth the effort
to implement.

Another comment:

    I thought the RFC was pretty clear on this. In a nutshell, if the
    delivery rewrites the envelope from address, it's considered a
    terminal delivery (i.e. delivery to a mailing list exploder), otherwise
    treat it as a forwarding operation (the /etc/aliases case). I would
    treat a .forward expansion as a final delivery event (it got to the
    user as far as the MTA is concerned).


    Yes, we need the DSN syntax. We also require the complete semantics of
    NOTIFY=SUCCESS,FAILURE for our application to work.


    Electronic Bill Presentment is really going to push the need for
    DSN support in MTAs. We just don't want to get stuck in a situation
    where we're faced with a non-DSN-aware MTA when we go to install
    our bill/statement engine, thus our interest in what the MTA vendors
    are planning to do about DSN.
  ------------------------------------------------------------------------------


(41) 14-Oct-98 M Find a way of modifying header lines
Oliver Smith

The problem with header_remove followed by header_add is that you can't refer
to the previous value of the header when adding a replacement. This could be
solved with a replace_header option.
------------------------------------------------------------------------------

(43) 15-Oct-98 M Sender rewrite *after* SMTP incoming checks
Andreas Edler

The anti-relaying check happens after the sender has been rewritten; there are
times when it would be helpful to do the check on the original sender, not on
the rewritten one. Quite how to configure this I'm not sure.

A related suggestion (from Steve Sargent) is to retain the original sender
address and make it accessible somehow.
------------------------------------------------------------------------------

(46) 20-Oct-98 L SMTP protocol hooks
Malcolm Ray

"But there are enough broken SMTP implementations to make me wonder whether
there isn't a case for providing hooks for tweaking the SMTP transport's
protocol exchange. Something which would allow me to say things like 'if, when
talking to lame.example.com, you get a 251 response to a MAIL command, rewrite
the response to 501 before continuing'."
------------------------------------------------------------------------------

(50) 13-Nov-98 M A "Focus" option for eximon
Frank Elsner

This is the opposite of "Hide"; it just displays a certain subset. Hmm. Could
something clever be done with regular expressions?
------------------------------------------------------------------------------

(61) 22-Dec-98 M Send failed error messages to somebody
Harald Meland

With sendmail, the failed error message is made into a error message,
with both envelope sender and recipient set to MAILER-DAEMON. The
original, bogus-envelope-sender message is then available to whoever
receives MAILER-DAEMON's mail. A more flexible approach would be to
specify a specific recipient.
------------------------------------------------------------------------------

(81) 01-Mar-99 M Addition of Content-MD5 support
Martin Hamilton

Martin supplied a suggested patch at
http://www.net.lut.ac.uk/~martin/antispam/exim-hacks/
------------------------------------------------------------------------------

(85) 15-Mar-99 M ability to rewrite addresses in non-standard headers
Dave Lewney
John Holman

Such as "return-receipt-to". See also 41.
------------------------------------------------------------------------------

(90) 21-Apr-99 M change wild prefix/suffix greediness
Ben Smithurst

Currently, when prefix or suffix containing * is set on a director, and the
fixed part occurs more than once in a local part, the length of the prefix or
suffix is maximized. For example, with suffix = -* and a local part of
foo-bar-baz the suffix is taken as bar-baz, leaving the local part as foo.
An option is proposed to invert this rule.
------------------------------------------------------------------------------

(91) 26-Apr-99 S make queue_run_in_order to newest first
"Andreas M. Kirchwitz"

The tidiest thing would be to have queue_run_order={random,oldest,newest},
and make queue_run_in_order obsolete.
------------------------------------------------------------------------------

(93) 04-May-1999 L fallback_transport

This would be a generic transport option, specifying a different transport to
be used if the first one failed. Failed hard, or failed soft? Or an option?
And if failed hard, is a bounce message sent as well, or not? There are uid
issues. Remote delivery would have to be done always in a subprocess so that
the main process could retain privilege in case the fallback transport was
local. That could be conditional. That's why this is labelled "Large". Some of
the things people want to do with this can be done by variations in the
routers, e.g. use $message_age to switch routers.
------------------------------------------------------------------------------

(94) 13-May-1999 M message to go with -Mg
Dave Holland
Alan Thew

So the admin can pass back a reason.
------------------------------------------------------------------------------

(99) 28-May-1999 M header to list failures for syntax_errors_to
mark david mcCreary

"I use the syntax_errors_to feature to email a copy of the error message.
It would be helpful to have the X-Failed-Receipients header in there,
identifying which addreses(s) are the problem, so that I don't have to
parse the body of the email message to figure out which addresses."
------------------------------------------------------------------------------

(100) 04-Jun-1999 S admin_users option, like trusted_users
Paul Mansfield
------------------------------------------------------------------------------

(102) 21-Jun-1999 M expanded basic variables
Julian King

Oh, and a wishlist entry, qualify_domain, and preferably other variables
can be set with a $lookup in the first part of the exim configuration
file, perhaps by an equivalent to backticks in shell script ("`command`")?
------------------------------------------------------------------------------

(105) 28-Jun-1999 M MIME-format bounce messages
Paul Makepeace

"Is there any work going/gone on/planned to enable exim to report delivery
status notifications using RFC1892 multipart/report MIME messages? It would be
great to have errors reported in a message/rfc822 attachment."

Jeffrey Goldberg
"I like plain bounces, so would hope that if you do this, that it be
configurable. I think that even for those who want it, it shouldn't be very
high on the wish list priority."

Other suggestions: toggle for bounces/warnings; override max_return for
certain addresses; use plain text if original not MIME. See Paul's hack
for background of what to do.

Nigel suggests using a specially named autoreply transport to generate bounces;
people could then replace this with another transport (e.g. pipe) if they want
to customize it themselves.

Eli Chen posted an unconditional patch for 3.32 that does some of this work.
That could form a basis.
------------------------------------------------------------------------------

(107) 12-Jul-1999 S defer transport at given load level
Marc Haber

------------------------------------------------------------------------------

(108) 16-Jul-1999 S remote sort by numbers of recipients
mark david mcCreary

In the absence of remote_sort, sort remote domains by the number of recipients
in each.
------------------------------------------------------------------------------

(114) 11-Nov-1999 S List of possible outgoing interfaces

Allow the smtp "interface" option to be a list: try them in turn until one
is found to work. Also allow masks to specify a range of addresses.
------------------------------------------------------------------------------

(123) 23-Dec-1999 L Use AUTH + TURN for dial-in hosts
Andrew Tverdokhleb

The way to do this would be to have Exim deliver messages into per-host
directories in, say, BSMTP format. Accept TURN if authenticated, and cause it
to run a helper program that is passed the socket in order to deliver the mail.
Provide a helper program!
------------------------------------------------------------------------------

(125) 04-Jan-2000 L Use shared memory segment for queue list
Theo Schlossnagle

The idea is that a queue-runner that finds no existing shared segment should
create one (if configured - possibly some fixed size) and all Exim processes
should maintain a list of messages in it, thereby saving on directory scans
when there are lots of messages. This needs a lot of careful thought to try to
eliminate any possibility of data loss. The interlocking could be quite tricky.
Further posters suggested using a db file to hold the list. See also 127.
------------------------------------------------------------------------------

(129) 14-Jan-2000 L Dynamically loadable lookup modules
Steve Haslam

Suggested patch provided.
------------------------------------------------------------------------------

(131) 17-Jan-2000 T Facility for assuming existence for EACCES
Peter Radcliffe

The opposite option for "+" in require_files: assume existence if cannot
peer into the directory (+ assumes non-existence).
------------------------------------------------------------------------------

(131) 29-Feb-2000 M? Control total number of outgoing SMTP calls
Brian White

This is for hosts with slow connections. Could some modification of
serialize_hosts be used for this? Or maybe use a semaphore? They seem to
be quite widely available.
------------------------------------------------------------------------------

(132) 01-Mar-2000 S Lookup host name from outgoing interface
Vadim Vygonets

Instead of primary_hostname, look up the name for the interface that is being
used for sending. Suggested patch supplied, but this should be an option of the
smtp transport.
------------------------------------------------------------------------------

(133) 06-Mar-2000 S Filter option not to log "previously sent"
Bruce Bowler

This is when using the "log" option of the autoreply driver.
------------------------------------------------------------------------------

(134) 09-Mar-2000 S Option to remove attachments when bouncing
------------------------------------------------------------------------------

(136) 13-Mar-2000 S/M Option for aliasfile to suppress "me too"

Could be tricky determining who "me" is.
------------------------------------------------------------------------------

(143) 08-May-2000 S Make quota_warn_threshold into a list
David Carter

So several warnings could be generated as the mailbox got bigger and bigger.
------------------------------------------------------------------------------

(146) 15-May-2000 M Allow SMTP error codes in retry rules

This would allow special handling of certain errors from certain hosts. In
particular, it would allow failing of certain 4xx codes.

This is now available for 4xx responses to RCPT commands. Is anything more
needed?
------------------------------------------------------------------------------

(148) 15-May-2000 S Warn recipient if message rejected for quota excession.
Heinz Ekker

Maybe not all that small, because the possibility of retrying must be taken
into account.
------------------------------------------------------------------------------

(149) 19-May-2000 L Make added headers visible in filters and other places
Hans Morten Kind

Headers added by directors/routers are not visible in subsequent processing.
This is a request to make them visible. What about removed headers? This could
be tricky to specify, hence the L.

A separate but related issue is the effect of headers added by "unseen"
directors. These are documented in chapter 19 as not being accumulated. Should
any change be made?
------------------------------------------------------------------------------

(155) 16-Jun-2000 M Special handling for certain hosts
mark david mcCreary

A means of changing the transport depending on the host name/IP of the most
preferred MX record so that all domains that route to certain hosts can be
handled specially. Maybe this could be a variable that is available in the
expansion of the "transport" option.
------------------------------------------------------------------------------

(158) 29-Jun-2000 S Configure "From" in bounces
Ben Parker

Cf Reply-To.
------------------------------------------------------------------------------

(159) 07-Jul-2000 M Keep messages for fixed time
Gary Palmer

An option to keep messages on the queue for a specified time, even if all their
destination hosts have timed out.
------------------------------------------------------------------------------

(164) 17-Aug-2000 S sender_unqualified_auth_hosts

To allow authenticated hosts to send unqualified addresses. Presumably it
needs received_... as well.
------------------------------------------------------------------------------

(167) 05-Sep-2000 L Support for ODBC

This would allow access to databases that don't have native support built into
Exim. See http://www.openlinksw.com/info/docs/rel3doc/unix/odbcsdk.htm
------------------------------------------------------------------------------

(168) 06-Sep-2000 M Deliver messages that alias to nothing to a given address
Dr ZP Han

If other people are managing alias lists, and one is empty, bounce that
delivery to a given address rather than freezing the message. Use the errors_to
address?
------------------------------------------------------------------------------

(172) 11-Sep-2000 S Allow file/directory in appendfile to override
"Michael J. Tubby"

When appendfile is called from forward or filter files, it ignores file or
directory settings. Maybe they should override. The path set by the forward or
filter is available in $address_file these days, so it could be used to create
a longer path.
------------------------------------------------------------------------------

(173) 18-Sep-2000 S A way of doing lsearches with EOL terminated keys
Jason Robertson

This is for looking up things like subject contents. Probably need an option to
exim_dbmbuild to make them into DBM files.
------------------------------------------------------------------------------

(174) 19-Sep-2000 S A way of using a different port for fallback hosts.
Dean Brooks
------------------------------------------------------------------------------

(181) 10-Nov-2000 S Compile-time options for ignoring Sendmail options

So that new ones could be accommodated easily.
------------------------------------------------------------------------------

(183) 04-Dec-2000 L dns_means_nonexist_after
Dave C.

In other words, wait a bit before giving up. This needs a mechanism for
remembering, which is not currently available. To be borne in mind for the
future.
------------------------------------------------------------------------------

(184) 04-Dec-2000 M Log more details of local caller
J. Nick Koston

"I was wondering if it was possible for exim to log the parent pid's cwd and
exe when it is called from a script/invoked by actually running /usr/sbin/exim
or /usr/sbin/sendmail." Question: is this information actually/easily
available to Exim? Needs investigation.
------------------------------------------------------------------------------

(186) 19-Dec-2000 S A simple utility to reset a retry time
Marc Haber

Basically, to do what exim_fixdb "delete" can do, but straightforwardly. There
could be an interface from eximon.
------------------------------------------------------------------------------

(187) 02-Jan-2001 M Wildcarding in headers_remove
Tamas TEVESZ

What I'd like to see is it to handle globs (or regexps, but i'm not sure this
latter would worth the hassle), in a way like:

          headers_remove = "X-*:Additional-header"
  ------------------------------------------------------------------------------


(188) 02-Jan-2001 S Make pipe timeout a temporary error
Georg v.Zezschwitz

A way to make a timeout into a temporary error.
------------------------------------------------------------------------------

(190) 03-Jan-2001 M Multiple message operations in eximon
------------------------------------------------------------------------------

(195) 19-Mar-2001 T TCP window size

TCP window size for receiving/sending, SMTP client/server.
------------------------------------------------------------------------------
------------------------------------------------------------------------------



  ------------------------------------------------------------------------------
  ------------------------------------------------------------------------------
  -----            Things that didn't make it into Exim 4                 ------
  ------------------------------------------------------------------------------
  ------------------------------------------------------------------------------


. An option to send messages to postmaster when ignore_errmsg_errors_after
times out.

. When an address is being routed, its constituents are in $local_part and
$domain, but there is currently no variable that contains the whole thing. It
could be put into $recipient, but that risks confusion with $recipients
(which is available in system filters). Maybe $address could be used?

. The ability to relay to host X without knowing all the domains that host X
might have. At ACL time, one would need to verify the recipient, and determine
that it routed to host X.

. A new lookup library that operates on a specially prepared file of IP
addresses and masks so that a single "lookup" yields a yes/no answer. This
should be a freestanding thing - needs a utility to build the file from a list.

. People want to change the wording of messages; can we find an efficient way
of allowing this? (Maybe put all messages into a separate module?) The problem
is not in the messages themselves, but in the values that get inserted into
messages. Would have to invent a new kind of function that used identified
values rather than positional ones. Use GNU gettext?

. Invent lf_hosts for those that may use LF without CR. Any other RFC
things we need to worry about?

. A user would really like to see something similar, perhaps with
"ID=$authenticated_id", similar to "helo=" and "ident=" in the default received
header. BUT there are security issues. Maybe give it as a commented out option
in the default configuration?

. Consider expanding further options that take integer values. What about
smtp_xxx options for different limits at different times of day (for example)?
What about tls_advertise_hosts (so can look at incoming IP/port)?

. How about a "hold hosts" option (cf hold_domains) to hold delivery to certain
hosts?

. Allow user filters to use "headers add", but probably not remove. Or maybe
just implement "allow" options for both of these features.

. Have the return from pipe in a variable, so that (e.g. error_message_file)
can make use of it.

. Implement randomize for ldap/sql servers.

. Add an option for ETRN that says "wait for the command to finish, and use its
stdout as the SMTP response."

. -odsomething for "ignore retry when doing immediate delivery".

. Add an option to the smtp transport to make it treat 5xx on connection as if
it were 4xx. Or possible add a sophisticated "after command X, treat xxx as
yyy".

. A way of rewriting addresses in non-standard header lines such as
Mail-Followup-To.

. Global option to enable initgroups() for exim uid. Default off.

. When verifying a sender, should it be rewritten with any T rewrites, because
it would be so rewritten if it actually was a recipient in a message?

. Sean Witham wants a way of defining macros that are not privileged, and a
sort of #ifdef structure that allows for different configurations in the same
file.

. Allow :fail: to specify that 551 be used instead of 550. Maybe allow a code
at the start, optionally? What about :defer:?

. SMTP timeout in middle of receiving message: log sender address if known, and
possibly message_id if known.

. Make -brw show rewrites for transports too.

. Have the MTA log destinations that have timed-out on a ident request and
no longer send rfc1413_queries to them. Add an option for how not to cache
these entries.

. Options and/or a utility to enable non-privileged users to view the queue
(e.g. -bpp), manipulate their own messages, etc.

. Specify a port along with a host in a route_list.

. A generalized "From" escaping scheme that also escapes >From so that the
whole thing can be reversed.

. There was a request for the \dns_again_means_nonexist\ option not to be
instantaneous, but to operate only after the DNS has been giving "try again"
for some time. Use the misc hints database.



  ------------------------------------------------------------------------------
  ------------------------------------------------------------------------------
  -----                     The Exim 4 Wish List                          ------
  ------------------------------------------------------------------------------
  ------------------------------------------------------------------------------


(1) 01-Jan-02 U Use of dynamically loaded libraries.

People want Exim to use dynamically loaded modules for a variety of reasons.
When I started to create Exim, I never expected anything other than source
distribution; the RPMs and inclusions in OS distributions caught me by
surprise. I know very little about the mechanics of dynamic loading, but I'm
aware that not all operating systems support it. I'm also aware that not all
people support it!

Furthermore, a way round this might be to supply more hooks along the lines of
local_scan(). Then people can write their own dynamic loaders if they want.
------------------------------------------------------------------------------

(3) 01-Jan-02 U Test for over-quota at SMTP time

This is a hard one, because the only way to test for over quota is to try to
deliver a message, certainly if system quotas are being used. And also, the
only available size at RCPT time is the SIZE option, though of course the test
could be run at DATA time. I think maybe we leave this one to an external
program, and require people to use ${run} to access the data. Let someone else
figure out how to extract the current mailbox size!

One suggestion is to implement

    ${file_size:/path/to/file}
    ${directory_size:/path/to/directory}


so that explicit checks can be done. It may be necessary to have four
operators, two being based on the block count, and two showing the "visible"
size. Directory scanning is expensive; is there any scope for caching? It would
seem not (you don't often get two addresses to the same user).
------------------------------------------------------------------------------

(4) 01-Jan-02 S Option to reject if no From: or Date: header line

Exim, in common with many other MTAs, inserts a From: or Date: header line if
one is missing. (It also inserts a blank Bcc:, but that is no longer needed by
RFC 2822 - it was by 822.) The suggestion is an option to give an error
instead. This could be done by making it possible to detect these insertions in
the acl_smtp_data ACL.
------------------------------------------------------------------------------

(6) 01-Jan-02 S Option to disable the use of -t
Dave C.

Would require work so that Exim itself doesn't use -t.
------------------------------------------------------------------------------

(7) 01-Jan-02 M Avoid showing LDAP passwords in log lines for LDAP errors
John W Baxter

May be tricky, because at the higher levels, the format of the query is not
understood.
------------------------------------------------------------------------------

(8) 01-Jan-02 S Expand once_repeat in autoreply
John Jetmore
------------------------------------------------------------------------------

(9) 01-Jan-02 S Headers as well as body in file for autoreply
Florian Laws
------------------------------------------------------------------------------

(10) 01-Jan-02 T Make "true" and "false" valid expansion conditions

This might help with "and" and "or" when one of the sub-conditions is, for
example, a lookup.
------------------------------------------------------------------------------

(11) 01-Jan-02 S Allow a filter to include another file.
------------------------------------------------------------------------------

(12) 01-Jan-02 M Support for different SQL servers per query

In other words, the global mysql_servers etc. is too restrictive.
------------------------------------------------------------------------------

(14) 01-Jan-02 M? Support for Sendmail milters

This could perhaps be done by extending the local_scan() idea and providing a
"standard" module which interfaced to milter.
------------------------------------------------------------------------------

(15) 01-Jan-02 M More hooks like local_scan()

One request has been for a similar hook at logging time. For other SMTP
interactions, maybe a hook into the ACL? See also 79 and 218.
------------------------------------------------------------------------------

(17) 11-Jan-02 M The construction of config.h needs refactoring

This has been hacked about substantially since the original implementation.
Given that there is a program (buildconfig), the messing around with the
environment could be abolished. Also, the distinction between "yes" and "no"
isn't always properly made (tests for #ifdef don't care about the value).
------------------------------------------------------------------------------

(18) 24-Jan-02 S Make $value retain its value after a top-level expansion

This was specifically for use in filter files. Currently it reverts to empty
as a consequence of save/restore for every lookup. It might be confusing to
do otherwise, however.
------------------------------------------------------------------------------

(19) 29-Jan-02 L Use of multiple DBM libraries

The problem is how to handle conflicting function names. Much research is
needed.
------------------------------------------------------------------------------

(20) 29-Jan-02 S Make system filter refreeze after manual thaw

Currently, a "freeze" in a system filter doesn't freeze after a manual thaw.
------------------------------------------------------------------------------

(21) 12-Feb-02 S Expand return_size_limit
Joachim Wieland

Is this really worth it? A per-transport value is also suggested - that would
mean remembering the value with each failed address and taking a minimum or
a maximimum (which?).
------------------------------------------------------------------------------

(24) 21-Feb-02 ? A way of testing TLS using -bh
------------------------------------------------------------------------------

(27) 06-Mar-02 M Distinguishing between different temporary callout errors

The request was to distinguish between a 4xx error and a failure to connect.
Problem is: how to cope when there is more than one host? Maybe only if ALL
fail to connect. An option like /callout_no_connect_ok.
------------------------------------------------------------------------------

(30) 12-Mar-02 S Add "recipients" precondition to routers.

This would avoid having to use "condition". (See also requirement for $address
mentioned above.) However, it would also require adding a caching feature, and
probably $recipient_data (cf $domain_data).
------------------------------------------------------------------------------

  (31)  21-Mar-02 S  Variables that indicate 8-bit message and 8-bit host, and
                     a way of using them to suppress a transport filter


A variable that is set if the message contains 8-bit characters, and another
that is set during the smtp transport if the host supports 8-bit. Then we also
need a condition that's expanded in the transport to control whether the filter
is run or not (e.g. transport_filter_condition).
------------------------------------------------------------------------------

(32) 22-Mar-02 M More info about callout fails for header sender verify

When there's a callout failure for an envelope address, the error message
contains details (by default) of the callout commands. This doesn't happen
for addresses in the header because there may be more than one of them, and
deciding how to give that information is tricky. Can we do better?
------------------------------------------------------------------------------

(33) 25-Mar-02 S Option to assume nomatch in dnslist lookups that time out

Currently this causes a DEFER.
------------------------------------------------------------------------------

(34) 26-Mar-02 S Access to DNS lookup functions via local_scan() API

This would make local_scan() writers lives easier for DNS usage.
------------------------------------------------------------------------------

  (36)  02-Apr-02 ?  A way of throttling, but allowing, relaying that would
                     otherwise be denied


This was suggested in connection with anonymizing messages. The "wait" command
in ACLs goes some way towards this. Is it enough?
------------------------------------------------------------------------------

(41) 17-Apr-02 T Make config.samples available as a directory for ftp

This is so that people can browse individual samples directly.
------------------------------------------------------------------------------

(42) 23-Apr-02 T An option not to flatten newlines in $message_body.

Or maybe better to provide $message_body_nl so as to have both.
------------------------------------------------------------------------------

(43) 23-Apr-02 T An option to treat 4xx as 5xx from STARTTLS

This would make Exim retry in clear unless the host is in hosts_require_tls.
------------------------------------------------------------------------------

(44) 24-Apr-02 ? Use errors_to for timeouts after redirect syntax errors

A syntax error in redirection data (with skip_syntax_errors false) causes a
defer. Eventually, the address may time out. This suggestion is that, when it
does, the bounce is sent to errors_to rather than to the sender.
------------------------------------------------------------------------------

(45) 13-May-02 T smtp_etrn_serialize_id = ....

The default behaviour would be equivalent to

    smtp_etrn_serialize_id = $smtp_command_argument
  ------------------------------------------------------------------------------


(47) 16-May-02 S Access to all addresses in batched local delivery
Miquel van Smoorenburg

In a batched local delivery with more than one recipient, there's no way to
access the list of recipients for doing custom things, such as stuffing them
all into a header. (BSMTP is the only approach; not everybody can use it.)
Suggested patch supplied.
------------------------------------------------------------------------------

(48) 21-May-02 M Support for ATRN (server and client)
Brian Candler

Server: If Exim had the ability to accept an ATRN command and then simply
invoke an external program, passing the SMTP stream on stdin and stdout and
the authenticated id as a parameter, that would do the job nicely.

Client: We need a variant of 'exim -bs' which would connect to a specified
host, send AUTH/ATRN, and then accept incoming messages as usual.
------------------------------------------------------------------------------

(50) 22-May-02 T Add comment (duplicate address) to Envelope-To:

This is just to minimize the confusion some people have.
------------------------------------------------------------------------------

(51) 07-Jun-02 S Option to use another address in callout MAIL FROM

This would be an address to try if MAIL FROM:<> failed. Is this actually going
to be helpful? See also 101.
------------------------------------------------------------------------------

(53) 11-Jun-02 S Make local_scan() dynamically loadable

David Woodhouse sent a patch. There's a more sophisticated one from Marc
Merlin. (See also Peter Benie's comments.) But should the base Exim have all
this in it?
------------------------------------------------------------------------------

(54) 11-Jun-02 S Ignore -Ac if called as mailq

I am not sure if this makes sense. This flag requests a listing of a different
mail queue, but Exim doesn't work like that. Is is not better for people to be
aware of this?
------------------------------------------------------------------------------

(55) 13-Jun-02 M Rewriting whole header lines
Dave C.

Current rewriting rules apply to individual addresses in header lines. This
feature would use a regex to match whole lines and replace them. It could be
useful for patching up syntactically invalid lines from crappy clients, before
the syntax check kicks in. (It might also be useful for hiding local host names
in Received: headers.)
------------------------------------------------------------------------------

(58) 26-Jun-02 ? Extend PAM support

Apparently PAM can do challenge-response authentication. The Exim interface
can't handle this. Investigate and think about how to do this.
------------------------------------------------------------------------------

(59) 26-Jun-02 M A "custom" authenticator

... that is simply a front end to external code. For example, there may be
an external API that hides the user password and does CRAM-MD5 when passed the
details of the challenge and response.
------------------------------------------------------------------------------

(60) 27-Jun-02 S Make trusted_users a local part list

So that it can use lsearch etc.
------------------------------------------------------------------------------

(62) 28-Jun-02 S Remove headers before DATA ACL
Patrice Fournier

"I'd like to be able to give Exim a list of headers that must be removed
from the message at arrival, before data_acl processing (and before the
rcpt_acl warn headers are added to the message)."
------------------------------------------------------------------------------

(63) 28-Jun-02 S Access to ACL-added headers in ACLs
Patrice Fournier

"I'd like also to be able to look at the already added headers by a
rcpt_acl when still checking rcpt_acl (either later in the acl for the same
RCPT TO or for another RCPT TO)."
------------------------------------------------------------------------------

(65) 28-Jun-02 M Expand fallback hosts

See also 174 of the Exim 3 list.
------------------------------------------------------------------------------

(66) 01-Jul-02 M Use Berkeley DB 4 concurrent access features

This might give better performance on very busy sites by reducing the
contention for access to hints databases. Rob Butler points out that this could
also be useful to allow updates of other DB files used by Exim to happen
concurrently. Another thing to think about with BDB is the possible use of
B-trees.
------------------------------------------------------------------------------

(68) 01-Jul-02 S Add sender host to delivery line

"Would it be possible to have a "sending_host_on_delivery" option that
logs the IP of the sending host in the => line?" Also requested was amount of
data transmitted for a non-delivery attempt.
------------------------------------------------------------------------------

(69) 03-Jul-02 T Log selector to log whoson checs
Matt Bernstein

"I'd quite like a log_selector option which could spot you'd done a whoson
lookup in your DATA ACL and maybe log it as W=user."
------------------------------------------------------------------------------

(70) 09-Jul-02 S A way of changing the RCPT address in an accept router

So as to avoid duplication problems when sending multiple addresses in multiple
copies to the same address.
------------------------------------------------------------------------------

(73) 17-Jul-02 M Match a list from within a condition

  e.g.  ${if matchdomain {$domain}{+domainlist} ...
        ${if matchhost {$sender_host_address}{1.2.3.4/10:2.3.4.5/16}...


Thought needed about how to handle host names. This may be too messy to specify
cleanly.

22-Apr-04: Implemented for domains, addresses, and local parts. Hosts are
too messy!
------------------------------------------------------------------------------

(74) 22-Jul-02 M Extend -bV to do more semantic checking

For example, diagnose "local_hosts" that should probably be "+local_hosts".
------------------------------------------------------------------------------

(75) 23-Jul-02 S Reference option on command line

The idea here is that a spam scanner that re-injects a message can supply a
reference on the command line that gets logged with R=.
------------------------------------------------------------------------------

(78) 30-Jul-02 S Expand queue_only (and/or queue_only_file)

The requirement is to make it possible to queue messages if certain conditions
are met (e.g. messages from certain local users). See also 93.

This control can now be achieved in the ACL - is this still needed?
------------------------------------------------------------------------------

(79) 31-Jul-02 S Additional info for log lines

An option to set an expanded string to be added to <= lines. And also for the
other delivery lines? See also 15.
------------------------------------------------------------------------------

(84) 09-Aug-02 S Make interfaces available in a variable

Something like $local_interfaces. Maybe limit the max length.
------------------------------------------------------------------------------

(85) 12-Aug-02 S/M Notice database connection failures

The small version of this just removes a server from the list within a single
Exim process when a connection to it fails. The bigger project would use the
retry database - but that has implications for bottlenecking and may not be
helpful. See also item 109. Another suggestion is to randomize the order in
which database servers are tried (randomize_database_servers). And another is
to measure response times and remember which server is fastest.
------------------------------------------------------------------------------

(87) 12-Aug-02 M Partial lookups for query-style lookups

The suggestion is to allow the lookup to contain a keystring (same syntax as
single-key lookups) which is then permuted and place in a suitable variable
each time - $permuted_key or something.
------------------------------------------------------------------------------

(88) 20-Aug-02 S Allow special retrying for forced defer

See also 146 in Exim 3 wish list above.
------------------------------------------------------------------------------

(89) 20-Aug-02 S Also allow retry rules on routers and transports
------------------------------------------------------------------------------

(90) 23-Aug-02 M Macros with arguments, a la C

I don't like this, because of the cost of frequent interpretation.
------------------------------------------------------------------------------

(93) 27-Aug-02 S queue_only_condition
Peter A. Savitch

queue_only_condition global option, expanded string. This contain
condition, which if evaluated to `no' or `false' or `0', behaves like
queue_only (queue_only_load ?). Don't know what to do is the string
expansion fails with DEFER (either force queueing or continue with
immediate delivery). Another option can control Exim behaviour if the
expansion fails. Don't know how the name for it ;-) See also 78.

This control can now be achieved in the ACL - is the new feature now needed?
------------------------------------------------------------------------------

(95) 27-Aug-02 S Log all parents as a router option

So that specific addresses can be logged like this. Should there be more log
selector options per router? Per transport?
------------------------------------------------------------------------------

(99) 28-Aug-02 L Test pre-conditions in order given

This would get round certain problems with require_files. However, it is
totally incompatible, and therefore an "Exim 5" wish.
------------------------------------------------------------------------------

(101) 02-Oct-02 M Callout and <> rejections

Some people don't want to fail the callout if the MAIL FROM:<> command is
rejected. Think of a way of handling this tidily. See also 51.
------------------------------------------------------------------------------

(102) 03-Oct-02 M Log option to suppress message-id logging

M because it would involve a change to eximstats.
------------------------------------------------------------------------------

(106) 09-Oct-02 S Appendfile to create directory not as user

Arrange for the setup entry to appendfile to create the directory under some
other uid (and with given owners/permissions?)
------------------------------------------------------------------------------

(109) 15-Oct-02 M Remember when LDAP (etc) servers are down

The idea would be to use some kind of retry rule, just like for hosts.
See also 85.
------------------------------------------------------------------------------

(110) 18-Oct-02 M errors_to for pipe command in filter

To work in the same was as errors_to for deliver commands.
------------------------------------------------------------------------------

(113) 15-Nov-02 M support for XMLRPC

Patch supplied for 4.10 by Joel Vandal.
------------------------------------------------------------------------------

(114) 04-Dec-02 M local_scan: return message on accept

(This actually dates from earlier.) The problem with this is that the string
currently passes into $local_scan_data. Thus, an incompatible change of some
sort would be required. Possibly a global that local_scan can set?
------------------------------------------------------------------------------

(118) 10-Dec-02 S access to Perl from local_scan
------------------------------------------------------------------------------

(119) 12-Dec-02 M ability to specify additional headers in an autoreply

This is so that vacation messages etc can have MIME headers that specify, for
example, the character set.
------------------------------------------------------------------------------

(125) 02-Jan-03 M Per-host daemon logging

"So what I would like is an option like debug_hosts, that allows to specify
an hostlist, and if the current incoming/outgoing hosts matches, creates a
logfile like $hostname_(in|out).log in my logdirectory."
------------------------------------------------------------------------------

(127) 06-Jan-03 M Different messages for different callout failures

The real requirement here is to detect when a callout "MAIL FROM:<>" failed, so
that a specific warning about that can be sent, different to the message when a
callout "RCPT TO:" fails. I think this is in fact now mostly done.
------------------------------------------------------------------------------

(129) 09-Jan-03 M Keep track of DNSBL timeouts, and refrain from calling

If so configured, keep track of DNSBL timeouts in a hints record, and don't
retry that DNSBL for a while after (a sufficient number of) timeouts. It is
effectively disabled for a while. Log enable/disable, of course. Another
thought is an option not to apply +defer_unknown unless *all* DNSBL lookups in
a list defer.
------------------------------------------------------------------------------

(130) 09-Jan-03 M A number of LDAP-related things
Peter A. Savitch

OpenLDAP 2.1 is going to be more popular (2.1.9 is available with many
bug fixes). TLS-enabled LDAP is an interesting and usefull thing.
I can try to implement some things and send the patches, like with
ldapi.

How do You see:

  1) The propagation of TLS options (key, certificate, CA certificate)
     to the OpenLDAP library.


2) (was dereferencing; done in 4.23).
------------------------------------------------------------------------------

(131) 09-Jan-03 S Additional variables
Peter A. Savitch

  $root_uid    Why?


(Some that were previously here are done)

  $smtp_accept_count    -- used for acl_smtp_connect


  $queue_runners        -- children of the listening daemon could use this
                           value for controlling the number of queue runners


I don't like either of these because they cannot be real-time values. They
would be snapshots of the values at the time the process was forked from the
daemon, and I fear they would just be confusing. For processes that were not
forked from the daemon they couldn't be set at all.
------------------------------------------------------------------------------

(131) 09-Jan-03 S Additional options
Peter A. Savitch

  exim_processes_max
  exim_file_descriptors_max
  queue_run_condition   -- to deprecate queue_run_max, better system
                           load control


Given Exim's distributed nature, I'm not at all sure how the first two of these
can usefully be implemented.
------------------------------------------------------------------------------

(132) 16-Jan-03 M Option for when a transport filter fails (crashes)

Freezing is one obvious option. Currently, Exim just retries. Another user
wanted to retry without the filter, but that is much harder.
------------------------------------------------------------------------------

(136) 24-Jan-03 M Make "personal" available as a condition for use in routers
------------------------------------------------------------------------------

(138) 28-Jan-03 M A variable containing what was matched in a host list

Or, presumably, other lists. This is so that ACL messages can say things like
"your host name matches xxxx". Note: not the same as $domain_data. Also, this
could be tricky with lookups and things that match in files.
------------------------------------------------------------------------------

(143) 06-Mar-03 L Ability to have multiple authenticators of same type

For example, to have two PLAIN authenticators; if the first fails, try the
second.
------------------------------------------------------------------------------

(144) 07-Mar-03 T ACL control = local_scan_skip to skip the local scan

A bigger project would be control = local_scan <xxx> where xxx could select
different local_scan functions (possibly by dynamic loading).

This can now be simulated using the fact that ACL variables are preserved,
so it doesn't look as it once did.
------------------------------------------------------------------------------

(145) 07-Mar-03 T Export string_cat() to local_scan()
------------------------------------------------------------------------------

(147) 17-Mar-03 T Option to treat 5xx as 4xx if received on initial connection

This issue is controversial. That may be a good reason for not changing
anything.
------------------------------------------------------------------------------

(153) 25-Apr-03 S A way of making log_as_local apply to the smtp transport

Either an option on the transport, or log_remote_as_local for the router.
Messy, either way. Maybe log_local_as_local and log_remote_as_local, and
deprecate log_as_local?
------------------------------------------------------------------------------

(154) 01-May-03 M Teergrubing at the CR/LF level

It is believed that the most effective way to teergrube is to insert a delay
between transmitting CR and LF in the SMTP response. Furthermore, this is also
the best place to test for bad synchronization (i.e. at the last possible
time).
------------------------------------------------------------------------------

(155) 01-May-03 S "control=no_pipelining" for connect and EHLO ACLs

Yet more flexibility! Maybe this should be a more general control for what is
sent in response to EHLO.
------------------------------------------------------------------------------

(156) 06-May-02 M Finer-grained synchronisation checking

On operating systems that can be asked whether any sent bytes have not yet been
ACK'd at the TCP/IP level, a finer-grained check for proper synchronisation can
be done. All bytes must have been ACK'd if the client has received the previous
response before sending the next command. See also 293.
------------------------------------------------------------------------------

(157) 07-May-03 M Newline as a list item separator

This will make life easier for lists obtained form databases where the
separator is naturally a newline.
------------------------------------------------------------------------------

(158) 13-May-03 M Ability to add to OK message for SMTP commands

For sending reasons for slow response, etc.
------------------------------------------------------------------------------

(160) 19-May-03 M Remove headers using wild cards
------------------------------------------------------------------------------

(162) 28-May-03 M/L Use of real numbers in filters, expansions, and options

The motivation for this is for handling spam scores that are real numbers. The
questions are (a) how widely should it spread and (b) whether floating point or
fixed point representations should be used. And what about the eval operator?
------------------------------------------------------------------------------

(164) 02-Jun-03 S Set variables for interface and port in smtp transport

These could be useful for varying HELO data etc. See also several other
items about interfaces above.
------------------------------------------------------------------------------

(166) 18-Jun-03 S CN verification in client TLS code

A tls_verify_cn option is suggested by Sven Geggus.
------------------------------------------------------------------------------

(168) 19-Jun-03 S Ability to add a header recording envelope rewrites

Current code adds a deleted header with only some information. Maybe what is
needed is a flag for a rewrite rule.
------------------------------------------------------------------------------

(169) 19-Jun-03 M A way of detecting timeouts in callout returns

------------------------------------------------------------------------------

(170) 23-Jun-03 S Option to accept rather than defer after local scan timeout

Suggested patch supplied.
------------------------------------------------------------------------------

(171) 23-Jun-03 S Option to make timeout a soft failure on pipe transport

------------------------------------------------------------------------------

(172) 23-Jun-03 M Option to make SQL query to specific server

------------------------------------------------------------------------------

(175) 04-Jul-03 S show_all_ancestors_in_errmsg for the redirect router

This is the opposite of hide_child_in_errmsg in effect.
------------------------------------------------------------------------------

(180) 14-Jul-03 M Extend never_users to be more flexible

e.g. never_users = ! mailnull : ! cyrus : !mailman : 0-100
------------------------------------------------------------------------------

(183) 16-Jul-03 S freeze_tell_text to add custom text to the message

------------------------------------------------------------------------------

(185) 24-Jul-03 S An expansion operator that decodes RFC 2047 strings

------------------------------------------------------------------------------

(188) 13-Aug-03 T batch_max=0 to mean unlimited

------------------------------------------------------------------------------

(189) 22-Aug-03 S Allow filter "logwrite" to write to syslog

I feel this is a dangerous facility, and also of very minority interest, at
least for user's filters. Allowing a system filter to write to mainlog or
syslog may be different. However, writing the main log would only be possible
if the filter runs as root or exim.
------------------------------------------------------------------------------

(190) 22-Aug-03 S A way of testing "forced delivery" in filter and routers
------------------------------------------------------------------------------

(191) 26-Aug-03 M Preserve $address_data for a verified recipient

The idea is to preserve it in the recipients data structure so that local_scan
can have access to it. The value could also be used as the initial value of
$address_data while routing.
------------------------------------------------------------------------------

(192) 05-Sep-03 M Better handling of TXT records for dnslists

When multiple lists are accessible via a merged lookup, handling TXT records
is difficult. An option for doing the TXT lookup in a sub-list has been
suggested, with syntax such as

          dnslists = list.example.org=127.0.0.2%dialups \
                                     ,127.0.0.3%relays \
                                     ,127.0.0.5%spews
  ------------------------------------------------------------------------------


(194) 10-Sep-03 M $addresslist_data to be like $host_data/$domain_data

------------------------------------------------------------------------------

(195) 29-Sep-03 M A variable containing the error for verify = header_syntax

Maybe there should always be a variable with the error message for all the
different kinds of verify failure.
------------------------------------------------------------------------------

(196) 30-Sep-03 S A way of detecting whether it was HELO or EHLO in the ACL

$received_protocol isn't reset until after the command is accepted (which
seems right), and $smtp_data shows only the arguments. Maybe $smtp_command?
------------------------------------------------------------------------------

(197) 30-Sep-03 S MACROS_DROP_PRIVS and ALT_CONFIG_DROP_PRIVS

Now that alternative configurations can be restricted to certain directories,
some more flexibility can be allowed. Not by default, though.
------------------------------------------------------------------------------

(198) 01-Oct-03 M Accept mail after local_scan() crash instead of defer

This may not be as easy to implement as it sounds; one is never sure of the
environment after a crash. Is is actually a good idea? The crashing local_scan
may have wrecked the memory in arbitrary ways; for example, screwing up the
recipients list...
------------------------------------------------------------------------------

(199) 01-Oct-03 M ${pipe which will pipe the message to a script ...

... and otherwise behave as ${run. Probably needs to have locking out features
so that it can be turned off for users .forwards if the sysadmin so desires.
------------------------------------------------------------------------------

(200) 07-Oct-03 L Alternative ways of storing hints

People want to store hints in databases. Some assert that SQL databases can
be made to perform satisfactorily. If a general interface could be worked on,
people could at least try different strategies. See also 66 above, which is
specifically concerned with Berkeley DB. Another possible option is a switch to
disable smtp-wait hints - to avoid contention problems.
------------------------------------------------------------------------------

(201) 07-Oct-03 M A "soft bounce" feature

This is an option that turns all hard bounces into soft bounces. The idea is
that it can be used as a safety-net while testing configurations. Instead of a
local bounce, the message stays on the queue; instead of 5xx SMTP responses,
4xx ones are given.

The ability to do the opposite - turn 4xx into 5xx under certain circumstances
might also be useful (e.g. after a certain time). This might best be done by
extending the retry logic to recognize 4xx as a special error. (This is now
done.)
------------------------------------------------------------------------------

(202) 10-Oct-03 S -bvsomething to do a callout after the verify
------------------------------------------------------------------------------

(203) 14-Oct-03 S verify=something to easily check for header presence

This is purely cosmetic; "condition" can already be used.
------------------------------------------------------------------------------

(204) 27-Oct-03 S an inverted queue_only_file

That is, queue if a file does NOT exist.
------------------------------------------------------------------------------

(205) 27-Oct-03 S expand smtp_accept_queue_per_connection

------------------------------------------------------------------------------

(206) 27-Oct-03 S appendfile: a variable containing the maildir base name

------------------------------------------------------------------------------

(207) 29-Oct-03 S ability to keep trusted users in a file - expand it.
------------------------------------------------------------------------------

(208) 31-Oct-03 M cache temporary verification errors and fail after a time

This request was for a way of turning temporary verification failures into
permanent ones after some fixed time.
------------------------------------------------------------------------------

(209) 31-Oct-03 S a way of making crashes in pipe commands temporary errors

------------------------------------------------------------------------------

(210) 31-Oct-03 S runtime option to change the daemon name used for tcprwappers

A patch for compile time was supplied, but this seems better as a runtime
option, for use with multiple Exim daemons.
------------------------------------------------------------------------------

(211) 31-Oct-03 S ability to disable debugging output from -bh & -bhc
------------------------------------------------------------------------------

(212) 31-Oct-03 M specify headers lines in HELO ACL to be added to all msgs
------------------------------------------------------------------------------

(214) 05-Nov-03 S Put the wild part of local part prefix/suffx in variables

Unfortunately, this isn't quite as trivial as it seems.
------------------------------------------------------------------------------

(215) 14-Nov-03 S A way of turning off message-submission fix-ups

Globally, and perhaps also via an ACL control so that it can be done on a
per-message basis.
------------------------------------------------------------------------------

(215) 26-Nov-03 M/L Conversion of IDNA domain names for logging

IDNA (RFCs 3490-3492) converts domains names containing non-ASCII characters
into ASCII strings of a special form. Exim will of course handle these.
However, it might be nice to convert them to a local code for logging. This
might be quite a big project: there's also output from -bp and eximon queue
display and no doubt other places as well. (Utilities that process the logs,
e.g. exigrep, eximstats, will be automatically handled if the logs are
changed.)
------------------------------------------------------------------------------

(216) 27-Nov-03 S Option to bounce if required TLS doesn't happen

This is for the smtp transport with hosts_require_tls set. Currently, it
defers. Possibly the best approach is to make the error one that can be seen by
the retry logic.
------------------------------------------------------------------------------

(217) 27-Nov-03 M A function to pass back variables from Perl

This is a function that can be called from Perl, to take a name and a value and
put that value into an Exim variable.
------------------------------------------------------------------------------

(218) 01-Dec-03 M A local_scan-like hook at system filter time

That is, make a C API available for custom filtering at this point.
------------------------------------------------------------------------------

(221) 18-Dec-03 U Merge routers and ACLs - or at least make more similar

"It will be very useful to be able to use most of the ACL conditions
(authenticated, hosts, senders, sender_domains, ... ) in routers and also the
possibility to have multiple conditions in routers. It will be great to also
be able to set variables in routers like in acl's." This is effectively a
radical suggestion for a complete re-design, and is therefore BIG.
------------------------------------------------------------------------------

(222) 19-Dec-03 S Iterative option for dnsdb

A way of getting a dnsdb lookup to chop off components until something is
found: e.g. ${lookup dndsb-i{ns=a.b.c.d}} would look for nameservers for
a.b.c.d, then b.c.d, etc.
------------------------------------------------------------------------------

(223) 22-Dec-03 S Support SOA lookup in dnsdb lookups
------------------------------------------------------------------------------

(225) 22-Dec-03 M Add acl= to routers

This would use an ACL to "control access" to a router, opening up a number
of interesting possibilities. Details of possible limitations need to be
investigated.
------------------------------------------------------------------------------

(226) 23-Dec-03 S A way of treating DEFER as fail in dnsdb lookups

(i.e. the dnsdb lookup failed, so accept the message)
------------------------------------------------------------------------------

(227) 30-Jan-04 M A configuration .if facility

"Second with the .ifdef and such, it would be nice to have a base .if,
so I could do something like
.if DEFINED_DATA == xyz
configuration here
.elseif DEFINED_DATA == abc
configuration here
.else
configuration here
.endif
also this would be nice at least in my case in the system filters, but
isn't required but you could pass the defined data to the system, in
variables."
------------------------------------------------------------------------------

(229) 30-Jan-04 M New expansion mechanism: {list ...}

  "Proposed syntax: {list {separator}{item}{item}...}
  This first expands the contents of {separator} and all of the {item}s,
  then constructs a separator-delimited list. The twist is: if an {item}
  generates the empty string, no separator will be generated for it.
  The entire construct will fail is {separator} fails, or all {item}s
  fail. If just some {item}s fail, they will be treated as if they
  generated empty strings.
  Examples:
     {list {,}{aaaaaa}{bbbbbb}{cccccc}} -> aaaaaa,bbbbbb,cccccc
     {list {,}{:fail:}{bbbbbb}{cccccc}} -> bbbbbb,cccccc
     {list {,}{aaaaaa}{:fail:}{cccccc}} -> aaaaaa,cccccc
     {list { }{aaaaaa}{bbbbbb}{}}       -> aaaaaa bbbbbb
     {list { }{:fail:}{:fail:}{:fail:}} -> :fail:
     {list {:fail:}{aaaaa}{bbbb}{cccc}} -> :fail:
  See particularly examples 2-4, which handle the case of a missing first
  and last item with ease; doing this using {if ...} would be quite difficult!"
  ------------------------------------------------------------------------------


(230) 30-Jan-04 M Find IP addresses of a domain's nameservers

This needs some way of processing a list of things in a similar way, which
should perhaps be a more general facility.
------------------------------------------------------------------------------

(231) 30-Jan-04 ? -C has a number of problems when used for real

-C was intended for testing; people are using it for "alternate"
configurations, and it doesn't work too well. Can a better way of doing this be
invented?
------------------------------------------------------------------------------

(232) 02-Feb-04 ? Make parts of the code loadable

The idea being that drivers, etc. could be compiled separately. There are, of
course, security issues. This is not something I want to go into at present.
------------------------------------------------------------------------------

(235) 02-Feb-04 T Make smtp_accept_count available as a variable

This is for use in ACLs. Of course, it is a snapshot of the count at the
start of the receiving process.
------------------------------------------------------------------------------

(236) 02-Feb-04 S String in local_scan that's added to the binary version string
------------------------------------------------------------------------------

(237) 02-Feb-04 M Add_header in ACLs because "message" is overloaded

This would be useful for verbs where "message" is an error message.
------------------------------------------------------------------------------

(238) 05-Feb-04 S ${address to handle multiple addresses

At present, ${address expects to see just one address. An extension would let
it handle header lines with multiple addresses, just retaining the actual
addresses. Or perhaps a new operator is needed?
------------------------------------------------------------------------------

(239) 23-Feb-04 ? Expansion items for encryption/decryption

Perhaps for some kind of cookie handling? This would need an external crypto
library, because there's no crypto code in Exim itself.
------------------------------------------------------------------------------

(240) 23-Feb-04 ? Some way to know if a ip is a mx for a given domain

Some kind of iterative operation for dnsdb might be a general way of providing
this.
------------------------------------------------------------------------------

(242) 01-Mar-04 ? Run a filter from an expansion condition

This would add a lot of power to ACLs, but its implementation might be tricky
because of the possibility of recursion.
------------------------------------------------------------------------------

(243) 01-Mar-04 ? Run an ACL from an expansion condition

The problem here is knowing what data is available at an arbitrary time.
------------------------------------------------------------------------------

(244) 01-Mar-04 ? Add an on-success event to transports

This could just be an expansion string, whose value is either ignored or
logged, but it could be used to run SQL updates or run programs etc.
However, what is "success" when a transport has multiple recipients?
------------------------------------------------------------------------------

(245) 01-Mar-04 M Add all the string expansion conditions to filters

Some thought would be needed on how to design the syntax for this.
------------------------------------------------------------------------------

(247) 09-Mar-04 S IP addresses that are never looked up

It would be nice if we could prevent this for certain IP addresses for
which we _know_ we'll never get a valid PTR record, like 2002::/16.
So a new option might reasonably default to:

          hosts_never_lookup = <; 2002::/16
  ------------------------------------------------------------------------------


(253) 05-Apr-04 M Use ESMTP and TLS for recipient callout verification

The best way to do this would involve quite a bit of refactoring so as to
abstract some of the code from the smtp transport into subroutines that could
also be used from the callout code. The tls parameters should probably be
taken from the transport. That might also require some substantial code
refactoring. See also 294.
------------------------------------------------------------------------------

(260) 30-Apr-04 S Respect +tls_cipher +tls_peerdn in rejectlog entries

------------------------------------------------------------------------------

(261) 05-May-04 S Add a "required_version" option

So that configurations can insist on a specific Exim version.
------------------------------------------------------------------------------

(262) 10-May-04 S Add "scratch" ACL variables

The idea is for variables that are flushed at the start of each ACL. I'm not
really convinced that these are worth implementing.
------------------------------------------------------------------------------

(263) 10-May-04 S Add variable $router_name $transport_name

These could be used in debug_print settings, which are output during -bt, and
thus don't need the privilege to run with -d.
------------------------------------------------------------------------------

(265) 25-May-04 M An init.d script for exim is needed

The old sendmail script used to "just work" because it just did -bd -q 20m or
whatever. Newer versions start more than one sendmail daemon, so do not work.
------------------------------------------------------------------------------

(267) 25-May-04 S tarpitting delay option

A modifier that sets a delay between lines for multiline responses.
------------------------------------------------------------------------------

(268) 25-May-04 S? Add a PID to every log line

Given that pids are reused non-cyclically these days, is this actually useful?
------------------------------------------------------------------------------

(269) 26-May-04 U Run both a system and a user filter in test mode

      exim -bF systemfilter -bf userfilter -f sender@dom < message


This would allow testing the way the userfilter handles the system
variables set by the systemfilter.
------------------------------------------------------------------------------

(270) 01-Jun-04 M Add headers at top and middle

Various initiatives like SPF and DomainKeys require header lines to be added
above or in the middle of existing headers. Exim always adds at the bottom.
When these requirements are more standard and clearer, some way of controlling
where header lines are added will probably become necessary. Some new syntax
will be required.

This can now be done fairly generally from local_scan(), and at the start and
after the Received: block from an ACL. Is anything more needed?
------------------------------------------------------------------------------

(271) 02-Jun-04 L Callouts at routing time

From a user's message:

> I would like to be able to:
>[...]
> 2) Forcing callouts as address verification at router level
> [ check_callout just like check_local_user ]
>
> I would like to redirect messages in some domain to "domain with callout
> verification" and to "domain without callout verification"
>
> e.g.
> userA@??? -> userX@??? (use callout to verify)
> userB@??? -> userY@??? (do not use callout verify)
>
> [both out-* domains delivered via "callout ready" transports]


  Other versions of the wish:
  * limiting callouts in acls to specific transport
        verify = recipient/callout=5s,transport:intranet_smtp
  * adding "select transport" to ACL conditions
        accept domains = +local_domains
               transport = cyrus_ltcp
               verify = recipient/callout=5s
  ------------------------------------------------------------------------------


(272) 07-Jun-04 S Expand hosts_randomize

It occurs in manualroute and in smtp.
------------------------------------------------------------------------------

(278) 21-Jun-04 M quota_warn_message_file option

Similar to the bounce and delivery warn message files.
------------------------------------------------------------------------------

(280) 23-Jun-04 M A way of adding a header line after callout defer_ok

This would record that, e.g., a sender domain verified, but the callout
could not be done.
------------------------------------------------------------------------------

(285) 16-Jul-04 M Separate and independent log_selector for rejectlog

For example: mainlog_selector and rejectlog_selector, with log_selector setting
both of them.
------------------------------------------------------------------------------

(286) 21-Jul-04 M Distinguishing a larger number of errors

For instance, detecting "connection reset by peer" (ENETRESET or ECONNRESET)
might be useful.
------------------------------------------------------------------------------

(288) 10-Aug-04 M Option for verify to require MX

e.g. verify=sender/require_mx
I'm not too keen because this is rather special purpose, and of course could
only apply if the verification happened to hit a dnslookup router.
------------------------------------------------------------------------------

(289) 10-Aug-04 L Option to treat defers in database lookups as "not found"

This is so that alternatives can be coded for when databases are down. A
suggested patch has been sent, but it just catches all instances of "defer"
from a lookup in an expansion string. These can occur for a number of different
reasons, not just connection failures. I think that we need a specific
"connection failed" indicator. Also, what about lookups in lists?
------------------------------------------------------------------------------

(291) 13-Aug-04 M An ACL or "local_scan()" to be run on size excession

The idea is to give something a chance to look at the data so far received when
more than message_size_limit (or some other limit) has arrived. I am not sure
how useful this would actually be in practice.
------------------------------------------------------------------------------

(292) 13-Aug-04 M Overall timeout for message reception

A client could in priciple keep an SMTP connection open for a very long time by
trickling in data very slowly. Also, after message_size_limit is exceeded, Exim
continues to swallow the data (though it does not write it to disk) until the
end is reached. Again, the connection could be held open for a very long time.
Some kind of overall time limit for an SMTP connection, possibly reset at the
start of each message, might be helpful in these situations.
------------------------------------------------------------------------------

(294) 23-Aug-04 L Callouts and AUTH and LMTP

People want to do callouts using LMTP as well as SMTP, and that would also
include sockets as well as TCP/IP connections. Also, people want to make use of
AUTH during the callout checking, on all types of connection. I suppose that
means making TLS available as well. This probably means a rewrite of the code
that actually does the callout. Should we use the relevant transport in a new
"callout" mode instead of keeping things separate? See also 253.
------------------------------------------------------------------------------

(296) 09-Sep-04 S Make deliver_time work for == lines as well as =>

What about ** lines?
------------------------------------------------------------------------------
--- HWM 297 ------------------------------------------------------------------
---------------------------- End of WishList ---------------------------------

Index: ABOUT
====================================================================
$Cambridge: exim/exim-doc/doc-scripts/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $

CVS directory exim/exim-doc/doc-scripts
---------------------------------------

This directory contains various scripts that are used to build the distributed
documentation from its source files.

End

Index: ABOUT
====================================================================
$Cambridge: exim/exim-doc/doc-src/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $

CVS directory exim/exim-doc/doc-src
-----------------------------------

This directory contains documentation files that are processed in some way in
order to make the documentation files that form part of Exim distributions. A
non-standard document processor is currently in use (October 2004), but in the
long term something more standard will have to take over.

End

Index: ABOUT
====================================================================
$Cambridge: exim/exim-doc/doc-txt/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $

CVS directory exim/exim-doc/doc-txt
-----------------------------------

This directory contains various documentation files that exist only as plain
text files, and are distributed in that format.

End

Index: ABOUT
====================================================================
$Cambridge: exim/exim-src/ABOUT,v 1.1 2004/10/08 10:38:48 ph10 Exp $

CVS directory exim/exim-src
---------------------------

This directory contains everything that is included in an Exim distribution
tarball, with the exception of the doc directory and an empty Local directory.
You can build Exim from the contents of this directory by adding a Local
directory that contains appropriate configuration files.

End