Wow, thanks for all the great help!
Like I said, this isn't my day job, and it looks like it was all my fault
after all. I have discovered "IPv4 requires a minimum MTU of 576 bytes. Below
it linux uses the minimum MTU and turns off path mtu discovery (drops DF)"[1],
which I definitely did not know. So it was my systems that were ignoring path
MTU discovery, even though I don't block ICMP and didn't (knowingly) turn off
path MTU discovery?
Sure enough, the tcpdump of a session shows that the packets are close to 576
bytes.
I'm not really sure why this only borked with gmail... a packet dump of a
SMTP session with Gmail is at
http://danielwebb.us/misc/gmail.tcpdump if
anyone knows what's going on. It looks to me like my system receives an ACK
and then ignores it and retries the packet forever.
As for the VoIP jitter: I read that Asterisk uses a hard-coded RTP packet size
of 0.02 seconds[2], so my UDP packets heading out to my provider are already
small. Here's (maybe) the problem, though: suppose a 1500 byte TCP packet
starts out on the wire, and immediately after a new VoIP packet comes in. The
VoIP packet is going to have to wait an absolute minimum of 35 ms before
starting transmission (my connection is about 340kbit outbound). That
actually doesn't sound too bad to me, but when I throttled down my MTU the
VoIP quality increased dramatically.
I do have QoS with VoIP packets given highest priority, but given my
experience here it's plenty likely I messed that up somehow too.
One thing I'm not sure about is where and how many queues are involved in the
packet traversal. For example, should I do:
ip link set dev eth0 qlen 1
or does the eth0 qdisc have access to that queue anyhow and that's a stupid
thing to do?
[1]
http://www.ussg.iu.edu/hypermail/linux/kernel/0302.0/0129.html
[2]
http://www.voip-info.org/wiki/view/Asterisk+SPA-841+Sound+Quality
