[Exim] lookups and string_cat()

Hi,

I sent a message a fair while back with subject 'Memory issues with pgsql
interface'. See:

http://www.exim.org/mailman/htdig/exim-users/Week-of-Mon-20011015/031291.html

I do some pretty strange things with exim so i'm not surprised that there
isn't too much of this mentioned on this mailing list else where (there is
some discussion however: http://www.exim.org/mailman/htdig/exim-users/Week-of-Mon-20011217/033371.html)

The problem is this: I see extreme memory usage when I run expand aliases
out of PostgreSQL. That is, I use exim, PostgreSQL and other software I
have written for a client to run some rather large
internal/external mailing lists. Unfortunately, it came to a head tonight
when one of my machines was unable to process a mailout for lack of
physical memory. I decided to open a nice bottle of wine and sort out the
problem. It has to do with string_cat():

/* We must *not* use store_release() here as is done in a
similar bit of code in accept.c because we cannot guarantee
that there are no other calls to the store_ functions between
calls to string_cat(). */

  if (!store_extend(string, oldsize, *size))
    {
    char *newstring = store_get(*size);
    memcpy(newstring, string, p);
    string = newstring;
    }

(This is from the 3.22 code base. I have confirmed that
string_cat() behaves more or less in the same manner in 4.01)

The problem is, of course, when exim calls this code to catenate 100000
addresses together. Since the store is not released, physical memory (1
GB) gets eaten up very very quickly. Not good.

I modified the pgsql driver specifically to pre-allocate a good deal of
memory for such queries.

if(num_tuples>500) {
        ssize = num_tuples * 30;
        result = store_get(ssize);
}

This means that subsequent calls to store_get() within the in
perform_pgsql_search() are minimised.

This is, of course, a band-aid solution. I can see only two correct
fixes: expanding string_cat() to handle store_release()'ing automagically
when possible (would require major reworking of the memory manager) or
modifying all code which uses string_cat() in the same way as the pgsql
driver.

The latter will be the best solution, since, for example, the lsearch
driver does not seem to be affected by this bug (anecdotally speaking) --
although a quick look suggests that it uses similar code. *shrug*.

Thoughts?

Gavin