[exim] OAuth2 client authenticator

On Sun, Mar 12, 2023 at 03:10:54PM +0200, Victor Ustugov via Exim-users wrote:
> https://packages.debian.org/source/sid/exim4 to it. I mean 75_*.patch
>
> Script get_oauth2_access_token-v.corvax.test@??? is based on
> https://github.com/pcw11211/exim4-oauth2/blob/main/get_bearer_tocken.sh
>
> This is just a prototype. I'm going to cache responses to requests to
> get an access_token, taking into account the expires_in value, in order
> to reduce the number of HTTP requests.

For what it is worth, I am doing this already in my Oauth script for use
with MS servers. I use gawk to do that part. It would be on Gitlab
already if it were not for real life in the way of a failing car battery
getting in the way.

> Client id, client secret and refresh token will be stored in the
> database, so there will be no need to create a script to get an access
> token for each user.

But I am not doing that, so far at least.

For information, I will attach the README.md for my script as it stands today.

ael


# Introduction

This minimal project allows exim4 to send email to an SMTP server which only
allows **OAUTH2**. At least for SMTP servers in the MS orbit.

It has been tested in conjuction with [M365-IMAP](https://github.com/UvA-FNWI/M365-IMAP)
which provides a similar service for collecting email from an MS IMAP
server using [offlineimap](https://github.com/OfflineIMAP/offlineimap3).

At the time of writing exim4 does not support OAUTH2, so it needs a separate
utility to authenticate. That is the purpose of this program.

The only similar utility that the author could find was
[exim4-oauth2](https://github.com/pcw11211/exim4-oauth2), but that was aimed at a gmail
server, and could not be modified to work with an MS server. But the basic
ideas from exim4-oath2 have been used here.

## A sample OAUTH2 transmission

Below is an example of successful XAOUTH transmission of an email to an
office365 SMTP server. This example, gathered using the exim -v option,
shows the main features which may help to explain a little of the
terminology and how to configure exim.

     SMTP<< 220 LO4P123CA0173.outlook.office365.com Microsoft ESMTP MAIL Service ready at 
                          Sat, 4 Mar 2023 11 :14:06 +0000
     SMTP>> EHLO this.machine
     SMTP<< 250-LO4P123CA0173.outlook.office365.com Hello [source_ip]
            250-SIZE 157286400
            250-PIPELINING
            250-DSN
            250-ENHANCEDSTATUSCODES
            250-STARTTLS
            250-8BITMIME
            250-BINARYMIME
            250-CHUNKING
            250 SMTPUTF8

The server advertises options, and exim will choose STARTTLS below.
This is because the exim transport uses

    driver = smtp
    protocol = smtp
    hosts_require_tls = *

This TLS encryption is needed before the server will offer XOAUTH2.

    SMTP>> STARTTLS
    SMTP<< 220 2.0.0 SMTP server ready
    SMTP>> EHLO this.machine
    SMTP<< 250-LO4P123CA0173.outlook.office365.com Hello [source_ip]
         250-SIZE 157286400
         250-PIPELINING
         250-DSN
         250-ENHANCEDSTATUSCODES
         250-AUTH LOGIN XOAUTH2
         250-8BITMIME
         250-BINARYMIME
         250-CHUNKING
         250 SMTPUTF8

One of the offers above is *AUTH LOGIN XOAUTH2*, so exim will reply
*AUTH XOAUTH2* and will deliver (base84 encoded) the access token
and more which it obtains from the script *exim_ms_tokens* which is
provided by this utility.

This is because the exim configuration must have a matching authenicator of
the form:

    login_oauth2:
      driver = plaintext
      public_name = XOAUTH2
      client_send = ${run{ SOMEWHERE/exim_ms_tokens } {$value} fail } 

It is that *public_name = XOAUTH2* which allows exim to select the OAUTH2
option. *plaintext* is safe because TLS is already in operation.

In passing, note that if you need to talk to other hosts using OAUTH2, such as
gmail, you will need a conditional in *login_oauth2* to select another script
to provide the relevant token, but that is not covered here.

    SMTP>> AUTH XOAUTH2 ************
    [.. snip.. The token and more are sent]
    SMTP<< 235 2.7.0 Authentication successful
    SMTP|> MAIL FROM:<me@test> SIZE=1378 AUTH=ddd@???
    SMTP|> RCPT TO:<foo@fizz>
         will write message using CHUNKING
    SMTP+> BDAT 369 LAST
    SMTP>> QUIT
    SMTP<< 250 2.1.0 Sender OK
    SMTP<< 250 2.1.5 Recipient OK
    SMTP<< 250 2.0.0 OK <E1pYPpI-0002cc-2b@???> [Hostname=NNN265MB0919.GBRP265.PROD.OUTLOOK.COM]
    SMTP<< 221 2.0.0 Service closing transmission channel

That is a successful transmission. Note that at least some versions of exim
may report an error as below which is probably spurious: it has been suggested
that the gnutls library is complaining about a early closure which is actually
legal behaviour. So it should be safe to ignore reports like the one below.

    LOG: MAIN
     H=outlook.xx.office365.com [nn.nn.nnn.nnn] TLS error on connection (recv):
      Error in the pull function.

# Prerequisites

1. A *bash* compatible shell. *dash* is ideal.
2. [curl](https://curl.se)
3. [gawk](http://www.gnu.org/software/gawk/)
4. *coreutilties* providing basic utilities like *cut* and *date*.
5. Authority to collect tokens: see later.

# Configuring exim

A router along these lines is needed:

    exampleoauth2:
      debug_print = "R: exampleoauth2 for $local_part@$domain"
      driver = manualroute
      domains = ! +local_domains
      transport = example_t
      route_list = * "outlook.office365.com::587" byname
      host_find_failed = ignore
      same_domain_copy_routing = yes
      no_more

If you use several "smarthosts", that is a variety of remote SMTP servers,
you will probably need to include a *condition* to decide when to
select the 365 server.

The transport *example_t* could be

    example_t:
      debug_print = "T: example_t for $local_part@$domain"
      driver = smtp
      multi_domain
      hosts_require_tls = *
      hosts_try_auth = $host_address
      protocol = smtp

Last, an authenticator, as before:

    login_oauth2:
      driver = plaintext
      public_name = XOAUTH2
      client_send = ${run{ SOMEWHERE/exim_ms_tokens } {$value} fail } 

*SOMEWHERE* is the directory where you decide to place *exim_ms_tokens*, perhaps
/usr/local/etc/exim. It must be executable by the exim user: it might
be wise to restrict ordinary users from executing or reading it. Reading it
will reveal the location of the cached tokens, although if those files
are properly protected, this might not be of particular concern.

## Placing the router, transport and authenicator

You probably don't need to be told that the
router should be somewhere in the section starting *begin routers*,
and similarly *begin transports* and *begin authenicators*.

If you are using the Debian exim4 package, you will probably already have
an *exim.conf.template* that you can adapt. Set it up to use a
"smarthost" which will be outlook.office365.com::587.
Depending on whether you choose a "split" configuration or not,
edit exim.conf.template or files under /etc/exim/conf.d .

Once you have done the above, you will find that you already have a router
and transport set up, and basically just need to add the authenicator.
Just check those automatically generated router and transport:
you may not need to adjust them.

Note that when the debian package is updated, you probably want to
avoid the dpkg-reconfiguration overwriting your changes. You may need
to update the configuration manually. vimdiff with folding makes this very
easy.

# Editing *exim_ms_tokens*

This file is a shell script, and includes some documentation in the comments
especially towards the end.

It handles two files which cache the refresh and access tokens.
The first task is to decide on the location of these small files.
The access token typically has a lifetime of around an hour: this from
observation. The refresh token has a much longer lifetime which some
documentation suggests may be around 90 days.

*exim_ms_tokens* updates the cached refresh token every month or so. This
is to save write cycles especially on flash media. The access token
is likely to be written more frequently, so it may be a good idea
to place it on a *tmpfs* if SSD wear matters. This is the main reason
for keeping the tokens in different files. However, since the access
token is unlikely to be written more once an hour, this is perhaps
over cautious.

1. The refresh token is cached in REFRESH_FILE. This should only be readable
by the exim user. Obtaining the initial refresh token to be placed in
this file is covered later.

2. ACCESS_FILE contains the access token, and again should only be readable by
the exim user.

The exim user must also be able to write both of the above files.
Edit *exim_ms_tokens* to set *REFRESH_FILE* and *ACCESS_FILE*.

## Getting the initial tokens

First you need a *client_id* and a *client_secret* which are issued by MS.
Here we try to avoid disappearing into a nightmare of complexity and cloud tenancies.

If you are already a slave, eh, tenant of MS, then you may well have an *id*
and *secret*, and not need much more guidance. But otherwise, it is
much easier to use the thunderbird client values. All of this and more
is covered in the [M365-IMAP](https://github.com/UvA-FNWI/M365-IMAP)
README.md file.

Since you are using exim4 to handle your outgoing mail at least, you
may perhaps be fetching your MS-hosted email via IMAP. M365-IMAP
handles that in conjunction with offlineimap.
If that is the case, I suggest installing M365-IMAP and setting it up
which will involve fetching the refresh token which you can use here.

But even if you do not use offlineimap or any IMAP, I still suggest
installing M365-IMAP and following the instructions there to run
*get_token.py*. You will then have a file, *imap_smtp_refresh_token*,
which contains your initial refresh token.

All you need to do is to copy that *imap_smtp_refresh_token* to *REFRESH_FILE*.
That is, copy the contents to the location you selected above in the
*exim_ms_tokens* file. It may be easiest to do this as root (or using sudo)
to avoid permission problems.

But that *REFRESH_FILE* is not quite ready: it needs an expiration date
in addition to the token itself. Open the file with your favorite
text editor, and insert a leading "1234,". So the file consists of a single
line, starting with a fake expiration date way in the past of 1234, then a
"," which acts as a separator followed by the refresh token and nothing else.
This will be rewritten when *exim_ms_tokens* is called from exim.
Or directly by root when testing...

It only remains to edit *exim_ms_tokens* to insert your *client-id*
and *client_secret* values in the obvious "???...???" places between the
quotes. Set *email_addr* to your email address that you use when logging
in to the MS system: it may be different from any of the email
addresses used by exim.

Ensure that *exim_ms_tokens* permits execution by the exim
user. chmod ug+x *exim_ms_tokens* if it is owned by that user, although
it is probably harmless to allow execution by other users since it
should fail if the permissions on the token files have been set properly.

# Summary

Edit *exim_ms_tokens* to change the values of

1. *REFRESH_FILE*
2. *ACCESS_FILE*
3. *client_id*
4. *client_secret* and
5. *email_addr* .

chmod to allow the exim user to execute *exim_ms_tokens*.

Prepare the *REFRESH_FILE* to contain a leading fake expiry date, a ","
and then the refresh token from *imap_smtp_refresh_token*, or elsewhere.
"1,tttttttt...." will do where ttttt... is the refresh token and the
quotes are not part of the file.

Check that the *REFRESH_FILE* can only be read only by the exim user:
Debian-exim on Debian.

Note: if *exim_ms_token* is not run for 90 days or so and the refresh token
expires, it cannot recover. You will need to obtain a new refresh token
by running *get_token.py* or otherwise, and then set up the
*REFRESH_FILE* again.

# Testing

If you find that exim is failing to send email, and the *mainlog* does
not identify the problem, you might consider below.

If, and only if, you are happy to execute *exim_ms_tokens* as root,
then

    # ./exim_ms_tokens|hexdump -C |less
in the right directory will allow you to check that the script is
working. You can inspect the *ACCESS_FILE* and the *REFRESH_FILE*
to see if they have been modified.

But this is definitely only for experienced system admins who thoroughly
understand the risks. Otherwise, you can carry out a similar test, but will
have to arrange all the permisions and restore them afterwards which is
complex and messy.