Re: Locking and concurrency -- system considerations

Top Page
Delete this message
Reply to this message
Author: Jon Morby
Date:  
To: Philip Hazel
CC: exim-users
Subject: Re: Locking and concurrency -- system considerations
> On Wed, 24 Sep 1997, Jon Morby wrote:
>
> > Individual logs isn't such a problem, although it would be nice to have a
> > central logging daemon which collated all the logs (a la syslog, only more
> > reliable) so you could have one large central log file.
>
> I'm afraid I wrote Exim to be simple. While being aware of the fact that
> one's programs always get used for tasks far bigger than you ever
> thought of (I've learned this the hard way over the last 30 years), I'm
> still amazed at the sort of things people fling at Exim (messages with
> 1000's of recipients and the like). I never envisaged it as an engine
> for use in extreme environments. I guess I suffer from imagination
> failure. :-)


Grasshopper, whatever you may imagine could pale in comparison :)

>
> > In the environment I envisage running Exim shared spools are essential. Th

e
> > current system we have uniques the spool file by taking the numeric ID from


> > the hostname and using that as well as the pid and current time.
> >
> > we have several machines (relay-1 -> relay-15 currently) all of which share


> > one of two spools.
> >
> > Being restricted to one spool per delivery client would be a restriction I
> > wouldn't like to have to live with.
>
> The fact that message ids are 16 characters long is probably assumed in
> too many places in Exim to make it easy to change. The current id is
> made up of 6 characters of time, in seconds, converted to base 62,
> another 6 characters of pid, also to base 62, and two digits which are
> nearly always 00, but can vary if a fast machine receives more than one
> message per second in the same process:
>
> tttttt-pppppp-nn
>
> Now, the 6-digit base-62 pid number can actually hold very large
> numbers. A 3-digit base-62 number can hold up to 238327, which is
> sufficient to hold a pid in most machines. One could therefore envisage
> using the top 3 digits to hold some kind of machine identification. Even
> if 4 digits were used for the pid (allowing up to 14776335 for the pid)
> that would still leave two digits for the machine id - 3843 possible
> values. Modifying Exim to generate message ids in this way would not be
> a big task.


Could be the answer .... It would certainly be useful to have multiple
machines writing to one area (from my viewpoint anyway)

>
> Another place where a modification would be required is when
> it is writing the header spool file. This is written with a temporary
> name and then renamed. Incorporating something machine-specific into the
> temporary name would not be too hard.
>
> > It would also be very useful to be able to share the retry/wait db's (proba

bly
> > essential).
>
> As these files are already locked when in use, you might think they
> could already be shared. However, the only locking is by fcntl() so I
> suspect things would go wrong. Code would have to be added to cause Exim
> to get a lock on a marker file *before opening a database*. However, I'm
> less convinced on the essentialness of sharing these dbs, since they are
> only hints.


True ... I was looking from the model of all the front end machines are
identical, and think they're the same machine.

>
> > > Essentially, the assumption is that each host has its own spool and log
> > > files.
> >
> > Unfortunately this assumption doesn't scale particularly well :(
>
> Well, at least it's linear. :-)


true :)

>
> > (What happens when one of your relays goes down, and you've got a couple of


> > thousand messages in it's queue?) The mail gets delayed until you can brin

g
> > another box up and mount that spool. Unacceptable in todays market with
> > consumers rather than techies coming onto the Net.
>
> How can I put this so that it doesn't sound like a whinge? You'll just
> have to believe me when I say it isn't. I'm reacting to your comment:
> If you are going to start talking market and consumers and commercial
> presssures, then I am going to respond by making remarks about paying
> commercial prices for your software. If it turns out to be possible to
> convert Exim into a "massive, time-critical, serious commercial
> environment" mailer, I'll be delighted, but that wasn't one of my
> original design goals.


No, I concur completely .... But then I haven't said that I'm not prepared to
pay something for the solution I end up using. :)

I wish that there was something out there which was able to do everything I
wanted, and scaled. As yet I haven't found anything, although exim (and
possibly qmail) are closest to the mark. (And personal preference is for exim
because of the friendly and helpful support I've received to date :)


>
> In our academic environment, if one of our relays goes down, then indeed
> the mail does get delayed until we can bring up another box (if we
> happen to have one). But then we live an an ivory tower...


Sometimes you don't know how lucky you are :)

>
> Philip
>
> -- 
> Philip Hazel                   University Computing Service,
> ph10@???             New Museums Site, Cambridge CB2 3QG,
> P.Hazel@???          England.  Phone: +44 1223 334714

>
>

-- 
Jon Morby                                  mail: jon@???
Fidonet/Internet Gateway                   http: www.fido.net




--
* This is sent by the exim-users mailing list.  To unsubscribe send a
    mail with subject "unsubscribe" to exim-users-request@???
* Exim information can be found at http://www.exim.org/