[exim] Delay on exim send increases with uptime

Hi,

I'm running exim 4.93 compiled from source on Debian via a systemd service, and I'm using it as an SMTP server with Dovecot as an IMAP server.
It works locally and remotely, in the sense that I can send emails to my Gmail address, and it doesn't have this issue on the local version, but remotely, my emails take more and more time to send as the server/service stays up for more time. I've recently hit between 80 and 300 seconds (increasing with time), but it goes down to 0 after restarting the service or rebooting the server.
Nothing shows up in the log about any failure to send or authenticate or receive the mail; it just has a very long delay.

Has anyone else seen this issue or dealt with something similar? Also, are there any debug steps I've missed (I've checked the logs at /var/spool/exim/log, and checked if my auth setup works, which it does). I'm using mutt as my MUA if that changes anything.