[ On Monday, November 15, 2004 at 19:35:52 (-0600), JupiterHost.Net wrote: ]
> Subject: [exim] exim capabilities fo 10-30 K email accounts
>
> I was wondering about deploying exim on a server that would handle 10-30
> thousand ISP customer's accounts.
>
> I realize exim can likley handle this, but I was wondering what kind of
> server you'd all recommend (1,2 or 4 processor? ram? HD space?)
Depending on how many limits you put on your users it's possible to
handle upwards of 15,000 users on a measly little PII/300Mhz with 512MB
of RAM and a decently fast disk subsystem. The particular system I have
in mind also serves all the personal web pages for those users (maybe
25% of users have homepages, some quite busy but most never get hit).
The requirements probably hinge more on the other parts of the mail
system than they do on the MTA -- e.g. I run that system above with
Cyrus IMAP -- and given that clue it should be clear that a great deal
also depends on the kind of access the users have to the network.
The system above (which runs smail, not exim, but resource wise they're
probably very similar in requirements :-), also runs Cyrus IMAP, but it
serves primarily broadband customers who are connected 24x7 with POP
(double-yuck!) or IMAP (vastly preferred for LAN/broadband use) whether
they're home using their computers or not. It handles between thirty
and sixty thousand messages per day with a 4MB size limit (and an
average size of well under 150K). It gets quite busy and sluggish at
times (as the load is not evenly distributed over the 24 hours of the
day :-), but it still works. Note it does not do virus scanning, though
we do use REGEX based content filters to reject the common MIME-encoded
virus crap at SMTP time.
Without such small message size limits you'll need something _much_ more
capable, especially to reach the upper limit of 30,000 users, and
especially if you want to do any amount of content scanning. A 64-bit
machine with oodles of RAM (i.e. >4GB which is why you need 64-bit) and
a _really_ fast storage subsystem should do the trick if you want it all
in one box.
For example I'm currently in the process of replacing that poor little
old PII with a "new" AlphaServer ES40 with four 667 MHz 21264A
processors (with 8MB cache each), 16 GB of RAM, some U160 system drives,
and an Apple Xserve RAID array with dual 2Gbit FC host adapters for mail
and web-page storage. I expect it to very easily handle well over
30,000 users, perhaps with a CPU+RAM upgrade in time (e.g. by 25,000
users), and with more storage added as needed, depending on how many
users actually chew up additional CPU cycles by using IMAP over SSL.
We could do it with multiple 32-bit systems of course, but that's a
_lot_ more work to manage (even with the newest Cyrus IMAP features) and
probably costs as much in capital in the end (assuming you would buy a
used Alpha like my client did) though of course a distributed cluster
can be scaled much more smoothly across time. I don't think though the
distributed systems architecture makes sense for e-mail services until
you expect to well exceed the 100,000 user barrier, at least not if you
truly consider the real total cost of operation. E.g. a brand new model
AlphaServer ES80 with, say, all eight 1.15GHz EV7 CPUs, 64GB RAM, and
even faster and more storage could _easily_ reach a million users (but
could be upwards of a half a million dollars in capital up-front, not
that an equivalent Sun or SGI would be any less of course :-).
We might have been able to use a dual or quad Opteron system, but I'm
much more wary of how well such new machines will fare in production,
especially when they require such fancy new compiler technology and
since I stick strictly to freeware operating systems and that means
using GCC as my compiler. (FYI I'm running all my systems with NetBSD,
which makes transition to the Alpha a no-brainer but transition to an
Opteron would be very touch and go, at least until NetBSD-2.1 hits the
streets!)
I expect a modern PC-class _server_ with, say, two or four of the faster
current Xeon P4 CPUs with decent cache, a full 4GB of RAM, and some U320
15kRPM system disks with the same Xserve RAID FC storage array would
easily reach the 30,000 user capacity but you'd spend just as much on
that level of system (if you bought it all new) as my client has spent
on the used ES40 which will still have lots of headroom to expand. An
ES47 would go much further too and they're appearing on the used market
now even though they still sell new.....
Be warned though that taking PC-class systems beyond 2GB of RAM is still
almost black magic..... at least if you want it to work reliably and
even if your OS does all the right things with it.
We considered Apple's G5 Xserve, but it's comparatively _very_ expensive
even though it's very small and very fast and 64-bit. NetBSD doesn't
run on it either and though OS X is great it's an unknown for my client.
Apple has designed their servers to go very fast doing relatively few
tasks, while a big mail server needs relatively little CPU but does lots
of tasks simultaneously (not necessarily something a "fast" CPU does
well) and of course it does _lots_ of I/O. Apple's design is much
closer to the SMP/mainframe-like architectures of the likes of the
bigger Alphas or a big Sun enterprise-class server, but it's not
scalable in its tiny 1U box and must be clustered to go far.
--
Greg A. Woods
+1 416 218-0098 VE3TCP RoboHack <woods@???>
Planix, Inc. <woods@???> Secrets of the Weird <woods@???>
