[exim] spool format error (on some list messages)

Top Page

Reply to this message
Author: exim-users
Date:  
To: exim-users
Subject: [exim] spool format error (on some list messages)
Hello,

as this gets quite long a short overiew in advance: After upgrading to
exim 4.90_1 (Ubuntu 18.04), I observe message corruption in the queue in
rare cases. All corrupted messages are from linux kernel mailing list. I
can eliminate file system as reason for corruption.


Whole story:
Since updating exim to version 4.90_1 (Ubuntu upgrade to 18.04), in rare
cases (less than 1 per 10000 messages deliveries) messages get stuck in
the queue (with message "spool format error: size=6045" when I list the
queue).

It seems, that files in /var/spool/exim4/input get corrupted (header and
body parts get mixed up). I can eliminate filesystems corruption as root
cause (no other indication for file corruption, filesystem checks w/o
errors). So far I only observe corruption on linux kernel mailing list
posts (example is attached), all are patch messages (mailers are
git-send-email in various versions and tip-git-log-daemon).

--cut exim mainlog excerpt:
2018-05-31 12:47:37.521 [4547] 1fOL7J-0001BL-DC SA: Debug: SAEximRunCond
expand returned: '1'
2018-05-31 12:47:37.522 [4547] 1fOL7J-0001BL-DC SA: Debug: check
succeeded, running spamc
2018-05-31 12:47:39.330 [4547] 1fOL7J-0001BL-DC SA: Action: scanned but
message isn't spam: score=-15.2 required=5.0 (scanned in 2/2 secs |
Message-Id: tip-j7o92acm2vnxjv70y4o3swoc@???). From
<linux-kernel-owner@???> (host=vger.kernel.org
[209.132.180.67]) for linux-kernel@XXX
2018-05-31 12:47:39.368 [4547] 1fOL7J-0001BL-DC <=
linux-kernel-owner@??? H=vger.kernel.org
[209.132.180.67]:37639 I=[1.2.3.4]:25 P=esmtp K S=7580 M8S=8
id=tip-j7o92acm2vnxjv70y4o3swoc@??? from
<linux-kernel-owner@???> for XXX
2018-05-31 12:47:39.381 [4612] 1fOL7J-0001BL-DC Format error in spool
file 1fOL7J-0001BL-DC-H: size=6763
--cut

--cut mail.log excerpt:
May 31 12:47:37 deep-thought spamd[4493]: spamd: processing message
<tip-j7o92acm2vnxjv70y4o3swoc@???> for Debian-exim:116
May 31 12:47:39 deep-thought spamd[4493]: spamd: result: . -15 -
BAYES_00,GREYLIST_ISWHITE,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI
scantime=1.8,size=5135,user=Debian-exim,uid=116,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=36468,mid=<tip-j7o92acm2vnxjv70y4o3swoc@???>,bayes=0.000000,autolearn=ham
autolearn_force=no
--cut

I run the Ubuntus exim4-daemon-heavy flavor with sa_exim for
greylisting/spamassassin integration and ClamAV for basic malware
scanning. SPF checking is enabled. All corrupted messages were accepted
without errors or warnings in exim and other (ClamAV, spamassassin, ...)
logs. No external message altering in /var/spool/exim4/input (where
corrupted messages get stuck).

--cut "exim4 --version"
Exim version 4.90_1 #2 built 14-Feb-2018 16:01:14
Copyright (c) University of Cambridge, 1995 - 2017
(c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007
- 2017
Berkeley DB: Berkeley DB 5.3.28: (September 9, 2013)
Support for: crypteq iconv() IPv6 PAM Perl Expand_dlfunc GnuTLS
move_frozen_messages Content_Scanning DKIM DNSSEC Event OCSP PRDR PROXY
SOCKS TCP_Fast_Open
Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm
dbmjz dbmnz dnsdb dsearch ldap ldapdn ldapm mysql nis nis0 passwd pgsql
sqlite
Authenticators: cram_md5 cyrus_sasl dovecot plaintext spa tls
Routers: accept dnslookup ipliteral iplookup manualroute queryprogram
redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Fixed never_users: 0
Configure owner: 0:0
Size of off_t: 8
Configuration file is /var/lib/exim4/config.autogenerated
--cut

Any hints comments highly appreciated, I can provide more examples or
details if necessary.

Best regards,
Thomas
1fOL7J-0001BL-DC-D
Commit-ID:  18a7057420f8b67f15d17087bf5c0863db752c8b
Gitweb:     https://git.kernel.org/tip/18a7057420f8b67f15d17087bf5c0863db752c8b
Author:     Arnaldo Carvalho de Melo <acme@???>
AuthorDate: Fri, 25 May 2018 16:37:36 -0300
Committer:  Arnaldo Carvalho de Melo <acme@???>
CommitDate: Wed, 30 May 2018 15:40:26 -0300


perf tools: Fix perf.data format description of NRCPUS header

In the perf.data HEADER_CPUDESC feadure header we store first the number
of available CPUs in the system, then the number of CPUs at the time of
writing the header, not the other way around.

Reported-by: Thomas-Mich Richter <tmricht@???>
Acked-by: Andi Kleen <ak@???>
Cc: Adrian Hunter <adrian.hunter@???>
Cc: David Ahern <dsahern@???>
Cc: He Kuang <hekuang@???>
Cc: Hendrik Brueckner <brueckner@???>
Cc: Jin Yao <yao.jin@???>
Cc: Jiri Olsa <jolsa@???>
Cc: Kim Phillips <kim.phillips@???>
Cc: Lakshman Annadorai <lakshmana@???>
Cc: Namhyung Kim <namhyung@???>
Cc: Simon Que <sque@???>
Cc: Stephane Eranian <eranian@???>
Cc: Wang Nan <wangnan0@???>
Link: https://lkml.kernel.org/n/tip-j7o92acm2vnxjv70y4o3swoc@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@???>
---
tools/perf/Documentation/perf.data-file-format.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt
index c57904a526ce..dfb218feaad9 100644
--- a/tools/perf/Documentation/perf.data-file-format.txt
+++ b/tools/perf/Documentation/perf.data-file-format.txt
@@ -111,8 +111,8 @@ A perf_header_string with the CPU architecture (uname -m)
A structure defining the number of CPUs.

 struct nr_cpus {
-       uint32_t nr_cpus_online;
        uint32_t nr_cpus_available; /* CPUs not yet onlined */
+       uint32_t nr_cpus_online;
 };


     HEADER_CPUDESC = 8,

1fOL7J-0001BL-DC-H
Debian-exim 116 123
<linux-kernel-owner@???>
1527763657 0
-received_time_usec .409222
-helo_name vger.kernel.org
-host_address 209.132.180.67.37639
-host_name vger.kernel.org
-interface_address 1.2.3.4.25
-active_hostname XXX
-received_protocol esmtp
-body_linecount 47
-max_received_linelength 116
-deliver_firsttime
XX
1
linux-kernel@XXX

240P Received: from vger.kernel.org ([209.132.180.67]:37639)
    by deep-thought with esmtp (Exim 4.90_1)
    (envelope-from <linux-kernel-owner@???>)
    id 1fOL7J-0001BL-DC
    for linux-kernel@XXX; Thu, 31 May 2018 12:47:39 +0200
187P Received: (majordomo@???) by vger.kernel.org via listexpand
        id S1754462AbeEaKqZ (ORCPT <rfc822;linux-kernel@XXX>);
        Thu, 31 May 2018 06:46:25 -0400
271P Received: from terminus.zytor.com ([198.137.202.136]:48299 "EHLO
        terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1754562AbeEaKqX (ORCPT
        <rfc822;linux-kernel@???>);
        Thu, 31 May 2018 06:46:23 -0400
256P Received: from terminus.zytor.com (localhost [127.0.0.1])
        by terminus.zytor.com (8.15.2/8.15.2) with ESMTPS id w4VAk69D3525797
        (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
        Thu, 31 May 2018 03:46:06 -0700
147P Received: (from tipbot@localhost)
        by terminus.zytor.com (8.15.2/8.15.2/Submit) id w4VAk6Qv3525794;
        Thu, 31 May 2018 03:46:06 -0700
040  Date:   Thu, 31 May 2018 03:46:06 -0700
093  X-Authentication-Warning: terminus.zytor.com: tipbot set sender to tipbot@??? using -f
064F From:   tip-bot for Arnaldo Carvalho de Melo <tipbot@???>
058I Message-ID: <tip-j7o92acm2vnxjv70y4o3swoc@???>
460C Cc:     kim.phillips@???, dsahern@???, wangnan0@???,
        tmricht@???, lakshmana@???, mingo@???,
        namhyung@???, eranian@???,
        linux-kernel@???, adrian.hunter@???,
        sque@???, jolsa@???, brueckner@???,
        acme@???, hekuang@???, yao.jin@???,
        ak@???, hpa@???, tglx@???
484R Reply-To: dsahern@???, kim.phillips@???, tmricht@???,
          lakshmana@???, wangnan0@???, eranian@???,
          mingo@???, namhyung@???,
          linux-kernel@???, adrian.hunter@???,
          sque@???, jolsa@???,
          brueckner@???, acme@???,
          hekuang@???, yao.jin@???, ak@???,
          hpa@???, tglx@???
042T To:     linux-tip-commits@???
090* Subject: [tip:perf/urgent] perf tools: Fix perf.data format description of
 NRCPUS header
056  Git-Commit-ID: 18a7057420f8b67f15d17087bf5c0863db752c8b
029  X-Mailer: tip-git-log-daemon
035  Robot-ID: <tip-bot.git.kernel.org>
089  Robot-Unsubscribe: Contact <mailto:hpa@kernel.org> to get blacklisted from
 these emails
018  MIME-Version: 1.0
032  Content-Transfer-Encoding: 8bit
040  Content-Type: text/plain; charset=UTF-8
028  Content-Disposition: inline
177* X-Spam-Status: No, score=1.4 required=5.0 tests=ALL_TRUSTED,BAYES_00,
        DATE_IN_FUTURE_48_96,FREEMAIL_FORGED_REPLYTO autolearn=no autolearn_force=no
        version=3.4.1
016* X-Spam-Level: *
078* X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on terminus.zytor.com
043S Sender: linux-kernel-owner@???
017  Precedence: bulk
040  List-ID: <linux-kernel.vger.kernel.org>
045  X-Mailing-List: linux-kernel@???
116  Received-SPF: none client-ip=209.132.180.67; envelope-from=linux-kernel-owner@???; helo=vger.kernel.org
037  X-SA-Exim-Connect-IP: 209.132.180.67
048  X-SA-Exim-Rcpt-To: linux-kernel@XXX
056  X-SA-Exim-Mail-From: linux-kernel-owner@???
093  X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on
    deep-thought.ursa-minor-beta.org
015  X-Spam-Level: 
175  X-Spam-Status: No, score=-15.2 required=5.0 tests=BAYES_00,GREYLIST_ISWHITE,
    HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI autolearn=ham
    autolearn_force=no version=3.4.1
031  X-Spam-Relay-Country: US US **
090  Subject: [tip:perf/urgent] perf tools: Fix perf.data format description of
 NRCPUS header
065  X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000)
066  X-SA-Exim-Scanned: Yes (on s-Mich Richter <tmricht@???>
042  Acked-by: Andi Kleen <ak@???>
044  Cc: Adrian Hunter <adrian.hunter@???>
036  Cc: David Ahern <dsahern@???>
034  Cc: He Kuang <hekuang@???>
053  Cc: Hendrik Brueckner <brueckner@???>
038  Cc: Jin Yao <yao.jin@???>
033  Cc: Jiri Olsa <jolsa@???>
040  Cc: Kim Phillips <kim.phillips@???>
046  Cc: Lakshman Annadorai <lakshmana@???>
039  Cc: Namhyung Kim <namhyung@???>
034  Cc: Simon Que <sque@???>
042  Cc: Stephane Eranian <eranian@???>
035  Cc: Wang Nan <wangnan0@???>
076  Link: https://lkml.kernel.org/n/tip-j7o92acm2vnxjv70y4o3swoc@git.kernel.org
058  Signed-off-by: Arnaldo Carvalho de Melo <acme@???>
110  ---
 tools/perf/Documentation/perf.data-file-format.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
001  
117  diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt
040  index c57904a526ce..dfb218feaad9 100644
057  --- a/tools/perf/Documentation/perf.data-file-format.txt
057  +++ b/tools/perf/Documentation/perf.data-file-format.txt
140  @@ -111,8 +111,8 @@ A perf_header_string with the CPU architecture (uname -m)
 A structure defining the number of CPUs.


 struct nr_cpus {
096  -       uint32_t nr_cpus_online;
        uint32_t nr_cpus_available; /* CPUs not yet onlined */
061  +       uint32_t nr_cpus_online;
 };


     HEADER_CPUDESC = 8,
009  _addr));
053  +    pydict_set_item_string_decref(dict_sample, "addr",
167  +            PyLong_FromUnsignedLongLong(sample->addr));
     set_sample_read_in_dict(dict_sample, sample, evsel);
     pydict_set_item_string_decref(dict, "sample", dict_sample);


066  at a single location.  Because of that we need to follow the same
067  +     * heuristic with the head, i.e we increment it before using its
068  +     * value.  Otherwise the first element of the packet queue is not
011  +     * used.
006  +     */
113  +    decoder->head = (decoder->head + 1) & (MAX_BUFFER - 1);


     *packet = decoder->packet_buffer[decoder->head];


058  -    decoder->head = (decoder->head + 1) & (MAX_BUFFER - 1);
043  -
     decoder->packet_count--;


     return 1;
010  m-Status)