Re: [Exim] Performance bottleneck scanning large spools.

Top Page
Delete this message
Reply to this message
Author: Philip Hazel
Date:  
To: Theo Schlossnagle
CC: exim-users
Subject: Re: [Exim] Performance bottleneck scanning large spools.
On Mon, 3 Jan 2000, Theo Schlossnagle wrote:

> I don't believe it would require a complete redesign... I did not mean
> to imply that exim should now have a central control process. That is
> one of the things I really like about Exim. What I was suggesting is
> this:
>
> You said that each queue runner must scan the directories to determine
> the contents of the spool (what messages exist). If an exim process
> check for the existence of a shared memory segment with that info in it
> before it just went ahead and scanned the directory, you would save
> hundreds of thousands of system calls (and disk reads), at lest in my
> scenario.


I think I understand what you are saying, but I confess to complete
ignorance about shared memory segments. Obviously I can go away and
learn about them. Are they "standard" in all the versions of Unix that
Exim runs on?

> If the shared memory segment didn't exist, exim would create
> one and use it to hold the information read from the spool scan. Any
> process can create the shared segment and if you kill any process the
> segment will linger until all have detached.


Fine, a queue runner creates the shared segment, but who updates the
shared segment as messages arrive and depart, so that it reflects
reality when the next queue runner process finds it already exists?
No doubt there is the question of locking access to the shared segment.
Will this hold up all incoming messages while the first queue-runner is
creating it?

My big worry is that messages will get "lost" (not listed in the
segment) and therefore not delivered because nothing notices them. This
is a phenomenon that used to hit us years ago when we ran another MTA
that had a faulty "queue manager" process.

> I think that it would be some work, but I don't think it qualifies as a
> complete redesign.


It would certainly be some work; I agree it wouldn't be a *complete*
redesign but it would be quite a large redesign.

-- 
Philip Hazel            University of Cambridge Computing Service,
ph10@???      Cambridge, England. Phone: +44 1223 334714.