Re: [exim-dev] [exim] Limiting the number of pipe processes

Top Page
Delete this message
Reply to this message
Author: Todd Lyons
Date:  
To: Marko Lalic, exim-dev
Subject: Re: [exim-dev] [exim] Limiting the number of pipe processes
Moved to dev mailing list.

On Wed, Sep 4, 2013 at 12:22 PM, Phil Pennock <exim-users@???> wrote:
> On 2013-09-04 at 13:37 +0200, Marko Lalic wrote:
>> In the scenario I have, each received message needs to be processed by
>> a Python script. However, it can happen that a few hundred mails are
>> received in the same time causing a large number of processes to be
>> spawned. This in turn makes the system go OOM crashing other important
>> services. Thus, I would like to be able to specify a maximum number of
>> processes which should be allowed.
>
> Someone was working on letting Python be embedded into Exim, much as
> Perl can be. That would avoid the fork/exec overhead.


I have just finished what I think is a very thorough implementation of
embedded python into Exim. The code is based off of the master
branch, so it's a bit ahead of anything that has been released thus
far. That also means that a patch will not apply cleanly against
whatever released version you are using.

The git tree is located at http://git.exim.org/users/tlyons/exim.git.
The branch is named master_volatile_python. The instructions in
doc/doc-txt/experimental-spec.txt describe how to 1) add python to the
Exim build 2) configure Exim to load python and 3) use the functions
in the imported python script.

Marko, if you don't use git or you have never built exim from source,
ask here and we can help you. Building from the git tree is slightly
different than building from the release tarball in that paths are
slightly different, but otherwise it's the same series of steps. I
also hang out daily in #exim on the Freenode IRC network (my nick is
cannonball) if you would like more interactive feedback, but that
depends on how busy things are at work. I'm typically logged in from
05:00 - 14:00 (UTC-0700) during the week.

Performance note:
At this point, it's unclear how much startup penalty to Exim there is
due to having embedded python. A good portion of that penalty is
going to be related to how many and which python modules your script
imports. There *is* a python_at_start variable which will delay
initializing the python interpreter until the first python function is
actually encountered. In that case, if your ACL's don't actually call
any python functions (RBL blocks early, header or recipient checks
block it early, etc), then the python initialization penalty is
completely removed.

I have not written tests for it yet.

...Todd
--
The total budget at all receivers for solving senders' problems is $0.
If you want them to accept your mail and manage it the way you want,
send it the way the spec says to. --John Levine