Re: [exim] Small modification for queue runners?

On Thu, Dec 02, 2004 at 02:20:57PM +0000, Philip Hazel wrote:
> On Thu, 2 Dec 2004, Tony Finch wrote:
>
> > Postfix seems to be able to use this model successfully.
>
> The explanation is probably that Wietse is a much better programmer than
> I am. I know that I don't understand the ramifications of the
> coordination that would be necessary to write something like that.

Now that I had more time to think about it, let me summarise my partial
understanding of the problem and show a simple solution:

Currently, queue runners start one delivery each. Multiple queue runners
run uncoordinated. In order to avoid locking collisions, each queue
runner randomises the list of messages before processing it.

Until now, everybody assumed that the randomisation would treat all
mails fair and with small queues and numbers of queue runners, everything
works that way, and efficiently so.

With larger queues and larger amounts of queue runners, the picture
changes: Some mails stay on the queue pretty long. Using even more
queue runners, things don't change a lot, but CPU usage increases badly.
This gets up to the point where I can't run more queue runners to get rid
of all messages. Small bursts of messages let load rise, thus queueing
even more, making everything worse. If you remove the hints databases
at this time, the system gets unusable, which tells me they really do
help a lot and hide the problem otherwise.

So why does this happen? This is the part I don't understand entirely yet.
I suspect that most CPU time is spent trying to deliver messages that
are either currently locked by another delivery or, what happens way
more often, that were just tried by another queue runner. Single queue
runs are very long, which means new messages are not recognised until a
new queue runner is allowed to start. In the mean time, old messages are
tried by all queue runners, each finding that their retry time has not yet
come, which wastes CPU time badly. At least that's part of the picture.
I don't see a square effect here, but something like that must be going
on, because running twice the number of queue runners let hell break
lose instead of slowing things down by half.

Thanks to Philip giving me the hint to look at -Mc, the solution
is a small script which takes the output of exim -bpra, extracts all
message-IDs and feeds lines of "exim -Mc <id>" to a program which starts
a shell for each line of input, keeping up to a configurable amount of
subshells running, if there is enough work.

The effect is dramatic: A queue of 244.000 mails has been cut down to
53.000 in two hours with 300 parallel deliveries going on. The 300 Exim
queue runners before could not perform that, probably because this
queue run treats all messages fair, not retrying one before any other
has been tried.

In case anybody wants to run similar experiments, I append the small
parallel shell program and the script.

Philip: Could you shed some light on why the queue runner needs the
pipe? How does the process tree look like when delivering multiple
messages down the same channel? Does one delivery fork and exec a new
delivery, passing it the channel? If so, why can't it just exec it?

Michael
----------------------------------------------------------------------
#include <sys/types.h>
#include <sys/wait.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
char ln[4096],*end;
int p,res=0,status=0,usage=0;

  if (argc!=2) usage=1;
  else
  {
    p=strtol(argv[1],&end,10);
    if (*end || p<1) usage=1;
  }
  if (usage)
  {
    fprintf(stderr,"Usage: parsh parallelity\n");
    exit(1);
  }
  while (fgets(ln,sizeof(ln),stdin))
  {
    if (p==0) { wait(&status); ++p; }
    if (res==0 && status) res=status;
    switch (vfork())
    {
      case 0:
      {
        execl("/bin/sh","sh","-c",ln,(const char*)0);
        fprintf(stderr,"parsh: exec failed: %s\n",strerror(errno));
        exit(2);
        break;
      }
      case -1:
      {
        fprintf(stderr,"parsh: vfork failed: %s\n",strerror(errno));
        exit(2);
        break;
      }
      default: --p;
    }
  }
  while (wait(&status)!=-1) if (res==0 && status) res=status;
  return res;
}
----------------------------------------------------------------------
#!/bin/rc

CONCURRENCY=300

echo 'Exim queue runner started.'

exec >/dev/null >[2=1] </dev/null

fn sighup {}

@{
@{ while (true) { /usr/exim/bin/exim -bpra; sleep 60 } } | awk '{
id=$3
notmatched=0;
do { getline } while ($0!="")
{ print "/usr/exim/bin/exim -Mc " id; print }
}' | parsh $CONCURRENCY
} &