fanf2 2005/09/13 18:51:06 BST
Modified files:
exim-doc/doc-txt ChangeLog NewStuff
exim-src/util README
Added files:
exim-src/util ratelimit.pl
Log:
Add util/ratelimit.pl to assist with choosing ratelimit settings.
Revision Changes Path
1.230 +2 -0 exim/exim-doc/doc-txt/ChangeLog
1.70 +4 -0 exim/exim-doc/doc-txt/NewStuff
1.2 +7 -0 exim/exim-src/util/README
1.1 +137 -0 exim/exim-src/util/ratelimit.pl (new)
Index: ChangeLog
===================================================================
RCS file: /home/cvs/exim/exim-doc/doc-txt/ChangeLog,v
retrieving revision 1.229
retrieving revision 1.230
diff -u -u -r1.229 -r1.230
--- ChangeLog 13 Sep 2005 15:40:07 -0000 1.229
+++ ChangeLog 13 Sep 2005 17:51:05 -0000 1.230
@@ -1,4 +1,4 @@
-$Cambridge: exim/exim-doc/doc-txt/ChangeLog,v 1.229 2005/09/13 15:40:07 ph10 Exp $
+$Cambridge: exim/exim-doc/doc-txt/ChangeLog,v 1.230 2005/09/13 17:51:05 fanf2 Exp $
Change log file for Exim from version 4.21
-------------------------------------------
@@ -187,6 +187,8 @@
avoiding the need to install special zones in a real server. This is
backwards compatible; if it can't find the fake resolver, it drops back.
Thus, both old and new test suites can be run.
+
+TF/02 Added util/ratelimit.pl
Exim version 4.52
Index: NewStuff
===================================================================
RCS file: /home/cvs/exim/exim-doc/doc-txt/NewStuff,v
retrieving revision 1.69
retrieving revision 1.70
diff -u -u -r1.69 -r1.70
--- NewStuff 13 Sep 2005 11:13:27 -0000 1.69
+++ NewStuff 13 Sep 2005 17:51:05 -0000 1.70
@@ -1,4 +1,4 @@
-$Cambridge: exim/exim-doc/doc-txt/NewStuff,v 1.69 2005/09/13 11:13:27 ph10 Exp $
+$Cambridge: exim/exim-doc/doc-txt/NewStuff,v 1.70 2005/09/13 17:51:05 fanf2 Exp $
New Features in Exim
--------------------
@@ -150,6 +150,10 @@
command (compare $smtp_command_argument - see PH/07 above). This makes it
possible to distinguish between HELO and EHLO, and also between things
like "MAIL FROM:<>" and "MAIL FROM: <>".
+
+TF/01 There's a new script in util/ratelimit.pl which extracts sending
+ rates from log files, to assist with choosing appropriate settings
+ when deploying the ratelimit ACL condition.
Exim version 4.52
Index: ratelimit.pl
====================================================================
#!/usr/bin/perl -wT
#
# $Cambridge: exim/exim-src/util/ratelimit.pl,v 1.1 2005/09/13 17:51:06 fanf2 Exp $
use strict;
sub usage () {
print <<END;
usage: ratelimit.pl <period> <regex> logfile
The aim of this script is to compute clients' peak sending rates
from an Exim log file, using the same formula as Exim's ratelimit
ACL condition. This is so that you can get an idea of a reasonable
limit setting before you deploy the restrictions.
This script isn't perfectly accurate, because the time stamps in
Exim's log files are only accurate to a seond whereas internally Exim
computes sender rates to the accuracy of your computer's clock
(typically 10ms).
The log files to be processed can be specified on the command line
after the other arguments; if no filenames are specified the script
will read from stdin.
The first command line argument is the smoothing period, as defined by
the documentation for the ratelimit ACL condition. The second argumetn
is a regular expression.
Each line is matched against the regular expression. Lines that do not
match are ignored. The regex may contain 0, 1, or 2 () capturing
sub-expressions.
If there are no () sub-expressions, then every line that matches is
used to compute a single rate. Its maximum value is reported when the
script finishes.
If there is one () sub-expression, then the text matched by the
sub-expression is used to identify a rate lookup key, similar to the
lookup key used by the ratelimit ACL condition. For example, you might
write a regex to match the client IP address, or the authenticated
username. Separate rates are computed for each different client and
the maximum rate for each client is reported when the script finishes.
If there are two () sub-expressions, then the text matched by the
first sub-expression is used to identify a rate lookup key as above,
and the second is used to match the message size recorded in the log
line, e.g. " S=(\\d+) ". In this case the byte rate is computed instead
of the message rate, similar to the per_byte option of the ratelimit
ACL condition.
END
exit 1;
}
sub iso2unix (@) {
my ($y,$m,$d,$H,$M,$S,$zs,$zh,$zm) = @_;
use integer;
$y -= $m < 3;
$m += $m < 3 ? 10 : -2;
my $z = defined $zs ? "${zs}1" * ($zh * 60 + $zm) : 0;
my $t = $y/400 - $y/100 + $y/4 + $y*365
+ $m*367/12 + $d - 719499;
return $t * 86400
+ $H * 3600
+ $M * 60
+ $S
- $z;
}
my $debug = 0;
my $progress = 0;
while (@ARGV && $ARGV[0] =~ /^-\w+$/) {
$debug = 1 if $ARGV[0] =~ s/(-\w*)d(\w*)/$1$2/;
$progress = 1 if $ARGV[0] =~ s/(-\w*)p(\w*)/$1$2/;
shift if $ARGV[0] eq "-";
}
usage if @ARGV < 2;
my $progtime = "";
my $period = shift;
my $re_txt = shift;
my $re = qr{$re_txt}o;
my %time;
my %rate;
my %max;
sub debug ($) {
my $key = shift;
printf "%s\t%12d %8s %5.2f %5.2f\n",
$_, $time{$key}, $key, $max{$key}, $rate{$key};
}
while (<>) {
next unless $_ =~ $re;
my $key = $1 || "";
my $size = $2 || 1.0;
my $time = iso2unix
($_ =~ m{^(\d{4})-(\d\d)-(\d\d)[ ]
(\d\d):(\d\d):(\d\d)[ ]
(?:([+-])(\d\d)(\d\d)[ ])?
}x);
if ($progress) {
my $prog_now = substr $_, 0, 14;
if ($progtime ne $prog_now) {
$progtime = $prog_now;
print "$progtime\n";
}
}
if (not defined $time{$key}) {
$time{$key} = $time;
$rate{$key} = 0.0;
$max{$key} = 0.0;
debug $key if $debug;
next;
}
# see acl_ratelimit() for details of the following
my $interval = $time - $time{$key};
my $i_over_p = $interval / $period;
my $a = exp(-$i_over_p);
$i_over_p = 1e-9 if $i_over_p <= 0.0;
$time{$key} = $time;
$rate{$key} = $size * (1.0 - $a) / $i_over_p + $a * $rate{$key};
$max{$key} = $rate{$key} if $rate{$key} > $max{$key};
debug $key if $debug;
}
print map {
" " x (20 - length) .
"$_ : $max{$_}\n"
} sort {
$max{$a} <=> $max{$b}
} keys %max;
# eof
Index: README
===================================================================
RCS file: /home/cvs/exim/exim-src/util/README,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -u -r1.1 -r1.2
--- README 24 May 2005 11:20:40 -0000 1.1
+++ README 13 Sep 2005 17:51:06 -0000 1.2
@@ -1,4 +1,4 @@
-$Cambridge: exim/exim-src/util/README,v 1.1 2005/05/24 11:20:40 ph10 Exp $
+$Cambridge: exim/exim-src/util/README,v 1.2 2005/09/13 17:51:06 fanf2 Exp $
The "util" directory in the Exim distribution
---------------------------------------------
@@ -24,6 +24,13 @@
A Perl script for a converting flat file into a format that is suitable for
processing by cdbmake into a cdb file. It has some advantages over the
cdbmake-12 awk script.
+
+ratelimit.pl
+------------
+
+A Perl script for computing peak sending rates from a log file. This is for
+use with the ratelimit ACL condition, so that you can get some idea of what a
+reasonable limit would be before deploying the feature.
unknownuser.sh
--------------
