Hello,
as this gets quite long a short overiew in advance: After upgrading to
exim 4.90_1 (Ubuntu 18.04), I observe message corruption in the queue in
rare cases. All corrupted messages are from linux kernel mailing list. I
can eliminate file system as reason for corruption.
Whole story:
Since updating exim to version 4.90_1 (Ubuntu upgrade to 18.04), in rare
cases (less than 1 per 10000 messages deliveries) messages get stuck in
the queue (with message "spool format error: size=6045" when I list the
queue).
It seems, that files in /var/spool/exim4/input get corrupted (header and
body parts get mixed up). I can eliminate filesystems corruption as root
cause (no other indication for file corruption, filesystem checks w/o
errors). So far I only observe corruption on linux kernel mailing list
posts (example is attached), all are patch messages (mailers are
git-send-email in various versions and tip-git-log-daemon).
--cut exim mainlog excerpt:
2018-05-31 12:47:37.521 [4547] 1fOL7J-0001BL-DC SA: Debug: SAEximRunCond
expand returned: '1'
2018-05-31 12:47:37.522 [4547] 1fOL7J-0001BL-DC SA: Debug: check
succeeded, running spamc
2018-05-31 12:47:39.330 [4547] 1fOL7J-0001BL-DC SA: Action: scanned but
message isn't spam: score=-15.2 required=5.0 (scanned in 2/2 secs |
Message-Id: tip-j7o92acm2vnxjv70y4o3swoc@???). From
<linux-kernel-owner@???> (host=vger.kernel.org
[209.132.180.67]) for linux-kernel@XXX
2018-05-31 12:47:39.368 [4547] 1fOL7J-0001BL-DC <=
linux-kernel-owner@??? H=vger.kernel.org
[209.132.180.67]:37639 I=[1.2.3.4]:25 P=esmtp K S=7580 M8S=8
id=tip-j7o92acm2vnxjv70y4o3swoc@??? from
<linux-kernel-owner@???> for XXX
2018-05-31 12:47:39.381 [4612] 1fOL7J-0001BL-DC Format error in spool
file 1fOL7J-0001BL-DC-H: size=6763
--cut
--cut mail.log excerpt:
May 31 12:47:37 deep-thought spamd[4493]: spamd: processing message
<tip-j7o92acm2vnxjv70y4o3swoc@???> for Debian-exim:116
May 31 12:47:39 deep-thought spamd[4493]: spamd: result: . -15 -
BAYES_00,GREYLIST_ISWHITE,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI
scantime=1.8,size=5135,user=Debian-exim,uid=116,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=36468,mid=<tip-j7o92acm2vnxjv70y4o3swoc@???>,bayes=0.000000,autolearn=ham
autolearn_force=no
--cut
I run the Ubuntus exim4-daemon-heavy flavor with sa_exim for
greylisting/spamassassin integration and ClamAV for basic malware
scanning. SPF checking is enabled. All corrupted messages were accepted
without errors or warnings in exim and other (ClamAV, spamassassin, ...)
logs. No external message altering in /var/spool/exim4/input (where
corrupted messages get stuck).
--cut "exim4 --version"
Exim version 4.90_1 #2 built 14-Feb-2018 16:01:14
Copyright (c) University of Cambridge, 1995 - 2017
(c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007
- 2017
Berkeley DB: Berkeley DB 5.3.28: (September 9, 2013)
Support for: crypteq iconv() IPv6 PAM Perl Expand_dlfunc GnuTLS
move_frozen_messages Content_Scanning DKIM DNSSEC Event OCSP PRDR PROXY
SOCKS TCP_Fast_Open
Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm
dbmjz dbmnz dnsdb dsearch ldap ldapdn ldapm mysql nis nis0 passwd pgsql
sqlite
Authenticators: cram_md5 cyrus_sasl dovecot plaintext spa tls
Routers: accept dnslookup ipliteral iplookup manualroute queryprogram
redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Fixed never_users: 0
Configure owner: 0:0
Size of off_t: 8
Configuration file is /var/lib/exim4/config.autogenerated
--cut
Any hints comments highly appreciated, I can provide more examples or
details if necessary.
Best regards,
Thomas
1fOL7J-0001BL-DC-D
Commit-ID: 18a7057420f8b67f15d17087bf5c0863db752c8b
Gitweb: https://git.kernel.org/tip/18a7057420f8b67f15d17087bf5c0863db752c8b
Author: Arnaldo Carvalho de Melo <acme@???>
AuthorDate: Fri, 25 May 2018 16:37:36 -0300
Committer: Arnaldo Carvalho de Melo <acme@???>
CommitDate: Wed, 30 May 2018 15:40:26 -0300
perf tools: Fix perf.data format description of NRCPUS header
In the perf.data HEADER_CPUDESC feadure header we store first the number
of available CPUs in the system, then the number of CPUs at the time of
writing the header, not the other way around.
Reported-by: Thomas-Mich Richter <tmricht@???>
Acked-by: Andi Kleen <ak@???>
Cc: Adrian Hunter <adrian.hunter@???>
Cc: David Ahern <dsahern@???>
Cc: He Kuang <hekuang@???>
Cc: Hendrik Brueckner <brueckner@???>
Cc: Jin Yao <yao.jin@???>
Cc: Jiri Olsa <jolsa@???>
Cc: Kim Phillips <kim.phillips@???>
Cc: Lakshman Annadorai <lakshmana@???>
Cc: Namhyung Kim <namhyung@???>
Cc: Simon Que <sque@???>
Cc: Stephane Eranian <eranian@???>
Cc: Wang Nan <wangnan0@???>
Link:
https://lkml.kernel.org/n/tip-j7o92acm2vnxjv70y4o3swoc@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@???>
---
tools/perf/Documentation/perf.data-file-format.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt
index c57904a526ce..dfb218feaad9 100644
--- a/tools/perf/Documentation/perf.data-file-format.txt
+++ b/tools/perf/Documentation/perf.data-file-format.txt
@@ -111,8 +111,8 @@ A perf_header_string with the CPU architecture (uname -m)
A structure defining the number of CPUs.
struct nr_cpus {
- uint32_t nr_cpus_online;
uint32_t nr_cpus_available; /* CPUs not yet onlined */
+ uint32_t nr_cpus_online;
};
HEADER_CPUDESC = 8,
1fOL7J-0001BL-DC-H
Debian-exim 116 123
<linux-kernel-owner@???>
1527763657 0
-received_time_usec .409222
-helo_name vger.kernel.org
-host_address 209.132.180.67.37639
-host_name vger.kernel.org
-interface_address 1.2.3.4.25
-active_hostname XXX
-received_protocol esmtp
-body_linecount 47
-max_received_linelength 116
-deliver_firsttime
XX
1
linux-kernel@XXX
240P Received: from vger.kernel.org ([209.132.180.67]:37639)
by deep-thought with esmtp (Exim 4.90_1)
(envelope-from <linux-kernel-owner@???>)
id 1fOL7J-0001BL-DC
for linux-kernel@XXX; Thu, 31 May 2018 12:47:39 +0200
187P Received: (majordomo@???) by vger.kernel.org via listexpand
id S1754462AbeEaKqZ (ORCPT <rfc822;linux-kernel@XXX>);
Thu, 31 May 2018 06:46:25 -0400
271P Received: from terminus.zytor.com ([198.137.202.136]:48299 "EHLO
terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
with ESMTP id S1754562AbeEaKqX (ORCPT
<rfc822;linux-kernel@???>);
Thu, 31 May 2018 06:46:23 -0400
256P Received: from terminus.zytor.com (localhost [127.0.0.1])
by terminus.zytor.com (8.15.2/8.15.2) with ESMTPS id w4VAk69D3525797
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO);
Thu, 31 May 2018 03:46:06 -0700
147P Received: (from tipbot@localhost)
by terminus.zytor.com (8.15.2/8.15.2/Submit) id w4VAk6Qv3525794;
Thu, 31 May 2018 03:46:06 -0700
040 Date: Thu, 31 May 2018 03:46:06 -0700
093 X-Authentication-Warning: terminus.zytor.com: tipbot set sender to tipbot@??? using -f
064F From: tip-bot for Arnaldo Carvalho de Melo <tipbot@???>
058I Message-ID: <tip-j7o92acm2vnxjv70y4o3swoc@???>
460C Cc: kim.phillips@???, dsahern@???, wangnan0@???,
tmricht@???, lakshmana@???, mingo@???,
namhyung@???, eranian@???,
linux-kernel@???, adrian.hunter@???,
sque@???, jolsa@???, brueckner@???,
acme@???, hekuang@???, yao.jin@???,
ak@???, hpa@???, tglx@???
484R Reply-To: dsahern@???, kim.phillips@???, tmricht@???,
lakshmana@???, wangnan0@???, eranian@???,
mingo@???, namhyung@???,
linux-kernel@???, adrian.hunter@???,
sque@???, jolsa@???,
brueckner@???, acme@???,
hekuang@???, yao.jin@???, ak@???,
hpa@???, tglx@???
042T To: linux-tip-commits@???
090* Subject: [tip:perf/urgent] perf tools: Fix perf.data format description of
NRCPUS header
056 Git-Commit-ID: 18a7057420f8b67f15d17087bf5c0863db752c8b
029 X-Mailer: tip-git-log-daemon
035 Robot-ID: <tip-bot.git.kernel.org>
089 Robot-Unsubscribe: Contact <mailto:hpa@kernel.org> to get blacklisted from
these emails
018 MIME-Version: 1.0
032 Content-Transfer-Encoding: 8bit
040 Content-Type: text/plain; charset=UTF-8
028 Content-Disposition: inline
177* X-Spam-Status: No, score=1.4 required=5.0 tests=ALL_TRUSTED,BAYES_00,
DATE_IN_FUTURE_48_96,FREEMAIL_FORGED_REPLYTO autolearn=no autolearn_force=no
version=3.4.1
016* X-Spam-Level: *
078* X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on terminus.zytor.com
043S Sender: linux-kernel-owner@???
017 Precedence: bulk
040 List-ID: <linux-kernel.vger.kernel.org>
045 X-Mailing-List: linux-kernel@???
116 Received-SPF: none client-ip=209.132.180.67; envelope-from=linux-kernel-owner@???; helo=vger.kernel.org
037 X-SA-Exim-Connect-IP: 209.132.180.67
048 X-SA-Exim-Rcpt-To: linux-kernel@XXX
056 X-SA-Exim-Mail-From: linux-kernel-owner@???
093 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on
deep-thought.ursa-minor-beta.org
015 X-Spam-Level:
175 X-Spam-Status: No, score=-15.2 required=5.0 tests=BAYES_00,GREYLIST_ISWHITE,
HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI autolearn=ham
autolearn_force=no version=3.4.1
031 X-Spam-Relay-Country: US US **
090 Subject: [tip:perf/urgent] perf tools: Fix perf.data format description of
NRCPUS header
065 X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000)
066 X-SA-Exim-Scanned: Yes (on s-Mich Richter <tmricht@???>
042 Acked-by: Andi Kleen <ak@???>
044 Cc: Adrian Hunter <adrian.hunter@???>
036 Cc: David Ahern <dsahern@???>
034 Cc: He Kuang <hekuang@???>
053 Cc: Hendrik Brueckner <brueckner@???>
038 Cc: Jin Yao <yao.jin@???>
033 Cc: Jiri Olsa <jolsa@???>
040 Cc: Kim Phillips <kim.phillips@???>
046 Cc: Lakshman Annadorai <lakshmana@???>
039 Cc: Namhyung Kim <namhyung@???>
034 Cc: Simon Que <sque@???>
042 Cc: Stephane Eranian <eranian@???>
035 Cc: Wang Nan <wangnan0@???>
076 Link: https://lkml.kernel.org/n/tip-j7o92acm2vnxjv70y4o3swoc@git.kernel.org
058 Signed-off-by: Arnaldo Carvalho de Melo <acme@???>
110 ---
tools/perf/Documentation/perf.data-file-format.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
001
117 diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt
040 index c57904a526ce..dfb218feaad9 100644
057 --- a/tools/perf/Documentation/perf.data-file-format.txt
057 +++ b/tools/perf/Documentation/perf.data-file-format.txt
140 @@ -111,8 +111,8 @@ A perf_header_string with the CPU architecture (uname -m)
A structure defining the number of CPUs.
struct nr_cpus {
096 - uint32_t nr_cpus_online;
uint32_t nr_cpus_available; /* CPUs not yet onlined */
061 + uint32_t nr_cpus_online;
};
HEADER_CPUDESC = 8,
009 _addr));
053 + pydict_set_item_string_decref(dict_sample, "addr",
167 + PyLong_FromUnsignedLongLong(sample->addr));
set_sample_read_in_dict(dict_sample, sample, evsel);
pydict_set_item_string_decref(dict, "sample", dict_sample);
066 at a single location. Because of that we need to follow the same
067 + * heuristic with the head, i.e we increment it before using its
068 + * value. Otherwise the first element of the packet queue is not
011 + * used.
006 + */
113 + decoder->head = (decoder->head + 1) & (MAX_BUFFER - 1);
*packet = decoder->packet_buffer[decoder->head];
058 - decoder->head = (decoder->head + 1) & (MAX_BUFFER - 1);
043 -
decoder->packet_count--;
return 1;
010 m-Status)
