[exim] Very bizzare Exim related problem

I'm having a very unusual network problem relating to exim and smtp
traffic. I'm hoping someone might know something about it.

I run 3 spam filtering servers using Exim. Two of them are in a data
center and one is at home. The home computer is a higher MX and is
mostly an emergency backup spam filter in case the data center goes
down. At home I have a lot of bandwith. I have 8mb in and 1mb out.

The home network lives behind a LINKSYS WRT54G router that acts as a NAT
and I've set it to port forward specific ports including SMTP to the
Linux/Exim server. And everything had been working well untill about 2
weeks ago.

The internal network has a gateway (the Linksys) at 192.168.2.1. If I
ping the gateway while running Exim I get 20% packet loss. However, if I
shut down Exim I go no packet loss. Th internal net is 100mb and I know
I don't have anywhere mear that much traffic.

I did another test copying large files in and out over port 22 and I get
no packet loss even at much higher data rates than my SMTP traffic. So
it's not data rate related.

Even though it's a secondary server it is continously bombarded with
spammers attempting to spam the 500 or so domains I filter for. So the
connection rate is high.

Additionally a Windows workstation on the network also has the same
packet loss issue talking to the gateway only when the Exim server (on
the Linux box) is running. Somehow the SMPT traffic is messing up my
internal network.

Has anyone heard of this? What's happening here?