From even my limited experience, there seem to be nontrivial numbers
of mailers which react to 5xx responses at various stages either by
proceeding as if nothing had gone wrong, or by treating the 5xx as a
temporary failure (e.g many of the hosts that we reject because they
presented illegal character(s) in the HELO string).
So far, it seems that the best chance for getting rid of bad mail
detected in earlier phases is to postpone the 5xx response until the
RCPT TO phase.