Christian Balzer wrote:
> Hello,
>
> I'm trying to get a rough, rule of thumb, ballpark figure of where
> the break even point of doing lsearch local file lookups versus ones
> to a remote LDAP database might be.
You are comparing apples and oranges.
Lsearch is really slow, does not use indexing, and with more entries it
becomes slower and slower.
If you want a filebased system, use dbm, or better cdb.
> Lets assume that there are no I/O, network or CPU bottlenecks in either
> scenario, what I'm trying to compare here is the cost (time wise) in
> query setup and execution.
> A further assumption would be that for each mail/address a total of 20
> lookups will be required. I can hear the shouts of "Twenty? How did the
> loon wind up with that high number?" already, but it's a good estimate
> for what I have in mind.
No, it isn't, did you ever heard about address_data?
With address_data and clever query constructs, you can trim down the
amount of youer queries massivly (mostly).
> With LDAP one is looking at the overhead to establish a TCP connection to
> a remote system, but the database queries are of course a lot faster than
> lsearch (at least after a certain DB size).
> With lsearch the TCP overhead is gone, the files will be completely cached
> for all practical purposes and fast, up a point.
Never compare a flatfile with an indexed database.
> Right now one example would be a 140KB file with 7500 addresses in it
> (users that have a specific spam filtering activated) which seems faster
> or at least as fast as LDAP.
Maybe you should have a look at your ldap indexes?
And, if you want to have a real example, check out the spamlist from
wirehub (
http://basic.wirehub.nl/spamlist.txt)
Try it with ldap/lsearch/cdb and see who's the winner ;)
> My gut feeling is that lsearch would do fine (on this particular hardware)
> up to about 1MB file size and 50000 addresses before LDAP becomes a
> decisively better choice.
Depends on your hardware, and your ldap configuration ;)
> Any (educated) guesses and insights how realistic this is and where that
> break even point might be?
For everything you need often, and that doesn't change too much, use a
local database, otherwise use a central database.
For a general way to cache queries, check out exims ${readsocket}
operator (I will release a small sample script for query caching and an
exim-config soon).
Nico
