[exim] Why doesn't Exim authenticate against IMAP directly?

Maybe I'm missing something about why this won't work but it seems to me
like it would be a good feature.

I am currently using Cyrus SASL to do Exim SMTP athentication. It's
configured to do an IMAP call so anyone who has an IMAP account can use
their IMAP login name and password to send email through Exim.

So - why use SASL? Why not have Exim talk directly to the IMAP port and
attempt to authenticate? If is succeeds then it lets the user send
email. Seems simple enough - so why wouldn't this be as easy as I think
it is?