Revision: 1251
http://vcs.pcre.org/viewvc?view=rev&revision=1251
Author: ph10
Date: 2013-02-20 17:42:03 +0000 (Wed, 20 Feb 2013)
Log Message:
-----------
Pass back the bumpalong value for partial matches.
Modified Paths:
--------------
code/trunk/ChangeLog
code/trunk/README
code/trunk/RunTest
code/trunk/doc/pcrepartial.3
code/trunk/pcre_dfa_exec.c
code/trunk/pcre_exec.c
code/trunk/pcretest.c
code/trunk/testdata/testinput2
code/trunk/testdata/testoutput2
code/trunk/testdata/testoutput8
Modified: code/trunk/ChangeLog
===================================================================
--- code/trunk/ChangeLog 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/ChangeLog 2013-02-20 17:42:03 UTC (rev 1251)
@@ -60,7 +60,11 @@
15. JIT compiler now supports 32 bit Macs thanks to Lawrence Velazquez.
+16. Partial matches now set offsets[2] to the "bumpalong" value, that is, the
+ offset of the starting point of the matching process, provided the offsets
+ vector is large enough.
+
Version 8.32 30-November-2012
-----------------------------
Modified: code/trunk/README
===================================================================
--- code/trunk/README 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/README 2013-02-20 17:42:03 UTC (rev 1251)
@@ -633,11 +633,15 @@
tests that are marked "never study" (see the pcretest program for how this is
done). If JIT support is available, the non-DFA tests are run a third time,
this time with a forced pcre_study() with the PCRE_STUDY_JIT_COMPILE option.
+This testing can be suppressed by putting "nojit" on the RunTest command line.
The entire set of tests is run once for each of the 8-bit, 16-bit and 32-bit
libraries that are enabled. If you want to run just one set of tests, call
RunTest with either the -8, -16 or -32 option.
+If valgrind is installed, you can run the tests under it by putting "valgrind"
+on the RunTest command line.
+
RunTest uses a file called testtry to hold the main output from pcretest.
Other files whose names begin with "test" are used as working files in some
tests. To run pcretest on just one or more specific test files, give their
@@ -932,4 +936,4 @@
Philip Hazel
Email local part: ph10
Email domain: cam.ac.uk
-Last updated: 27 October 2012
+Last updated: 20 February 2013
Modified: code/trunk/RunTest
===================================================================
--- code/trunk/RunTest 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/RunTest 2013-02-20 17:42:03 UTC (rev 1251)
@@ -14,9 +14,10 @@
# disabled with /SS.
#
# When JIT support is available, all the tests are also run with -s+ to test
-# (again, almost) everything with studying and the JIT option. There are also
-# two tests for JIT-specific features, one to be run when JIT support is
-# available, and one when it is not.
+# (again, almost) everything with studying and the JIT option, unless "nojit"
+# is given on the command line. There are also two tests for JIT-specific
+# features, one to be run when JIT support is available (unless "nojit" is
+# specified), and one when it is not.
#
# Whichever of the 8-, 16- and 32-bit libraries exist are tested. It is also
# possible to select which to test by the arguments -8, -16 or -32.
@@ -151,11 +152,12 @@
# Default values
-valgrind=
-sim=
arg8=
arg16=
arg32=
+nojit=
+sim=
+valgrind=
# This is in case the caller has set aliases (as I do - PH)
unset cp ls mv rm
@@ -221,9 +223,10 @@
-8) arg8=yes;;
-16) arg16=yes;;
-32) arg32=yes;;
+ nojit) nojit=yes;;
+ sim) shift; sim=$1;;
valgrind) valgrind="valgrind --tool=memcheck -q --smc-check=all";;
valgrind-log) valgrind="valgrind --tool=memcheck --num-callers=30 --leak-check=no --error-limit=no --smc-check=all --log-file=report.%p ";;
- sim) shift; sim=$1;;
*) echo "Unknown test number '$1'"; exit 1;;
esac
shift
@@ -309,7 +312,7 @@
jitopt=
$sim ./pcretest -C jit >/dev/null
jit=$?
-if [ $jit -ne 0 ] ; then
+if [ $jit -ne 0 -a "$nojit" != "yes" ] ; then
jitopt=-s+
fi
@@ -700,7 +703,7 @@
if [ $do12 = yes ] ; then
echo $title12
- if [ $jit -eq 0 ] ; then
+ if [ $jit -eq 0 -o "$nojit" = "yes" ] ; then
echo " Skipped because JIT is not available or not usable"
else
$sim $valgrind ./pcretest -q $bmode $testdata/testinput12 testtry
Modified: code/trunk/doc/pcrepartial.3
===================================================================
--- code/trunk/doc/pcrepartial.3 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/doc/pcrepartial.3 2013-02-20 17:42:03 UTC (rev 1251)
@@ -1,4 +1,4 @@
-.TH PCREPARTIAL 3 "24 June 2012" "PCRE 8.31"
+.TH PCREPARTIAL 3 "20 February 2013" "PCRE 8.33"
.SH NAME
PCRE - Perl-compatible regular expressions
.SH "PARTIAL MATCHING IN PCRE"
@@ -56,31 +56,34 @@
.rs
.sp
A partial match occurs during a call to \fBpcre_exec()\fP or
-\fBpcre[16|32]_exec()\fP when the end of the subject string is reached successfully,
-but matching cannot continue because more characters are needed. However, at
-least one character in the subject must have been inspected. This character
-need not form part of the final matched string; lookbehind assertions and the
-\eK escape sequence provide ways of inspecting characters before the start of a
-matched substring. The requirement for inspecting at least one character exists
-because an empty string can always be matched; without such a restriction there
-would always be a partial match of an empty string at the end of the subject.
+\fBpcre[16|32]_exec()\fP when the end of the subject string is reached
+successfully, but matching cannot continue because more characters are needed.
+However, at least one character in the subject must have been inspected. This
+character need not form part of the final matched string; lookbehind assertions
+and the \eK escape sequence provide ways of inspecting characters before the
+start of a matched substring. The requirement for inspecting at least one
+character exists because an empty string can always be matched; without such a
+restriction there would always be a partial match of an empty string at the end
+of the subject.
.P
If there are at least two slots in the offsets vector when a partial match is
returned, the first slot is set to the offset of the earliest character that
was inspected. For convenience, the second offset points to the end of the
-subject so that a substring can easily be identified.
+subject so that a substring can easily be identified. If there are at least
+three slots in the offsets vector, the third slot is set to the offset of the
+character where matching started.
.P
-For the majority of patterns, the first offset identifies the start of the
-partially matched string. However, for patterns that contain lookbehind
-assertions, or \eK, or begin with \eb or \eB, earlier characters have been
-inspected while carrying out the match. For example:
+For the majority of patterns, the contents of the first and third slots will be
+the same. However, for patterns that contain lookbehind assertions, or begin
+with \eb or \eB, characters before the one where matching started may have been
+inspected while carrying out the match. For example, consider this pattern:
.sp
/(?<=abc)123/
.sp
This pattern matches "123", but only if it is preceded by "abc". If the subject
-string is "xyzabc12", the offsets after a partial match are for the substring
-"abc12", because all these characters are needed if another match is tried
-with extra characters added to the subject.
+string is "xyzabc12", the first two offsets after a partial match are for the
+substring "abc12", because all these characters were inspected. However, the
+third offset is set to 6, because that is the offset where matching began.
.P
What happens when a partial match is identified depends on which of the two
partial matching options are set.
@@ -308,10 +311,9 @@
.P
\fBNote:\fP If the pattern contains lookbehind assertions, or \eK, or starts
with \eb or \eB, the string that is returned for a partial match includes
-characters that precede the partially matched string itself, because these must
-be retained when adding on more characters for a subsequent matching attempt.
-However, in some cases you may need to retain even earlier characters, as
-discussed in the next section.
+characters that precede the start of what would be returned for a complete
+match, because it contains all the characters that were inspected during the
+partial match.
.
.
.SH "ISSUES WITH MULTI-SEGMENT MATCHING"
@@ -330,13 +332,33 @@
offsets that are returned for a partial match. However a lookbehind assertion
later in the pattern could require even earlier characters to be inspected. You
can handle this case by using the PCRE_INFO_MAXLOOKBEHIND option of the
-\fBpcre_fullinfo()\fP or \fBpcre[16|32]_fullinfo()\fP functions to obtain the length
-of the largest lookbehind in the pattern. This length is given in characters,
-not bytes. If you always retain at least that many characters before the
-partially matched string, all should be well. (Of course, near the start of the
-subject, fewer characters may be present; in that case all characters should be
-retained.)
+\fBpcre_fullinfo()\fP or \fBpcre[16|32]_fullinfo()\fP functions to obtain the
+length of the longest lookbehind in the pattern. This length is given in
+characters, not bytes. If you always retain at least that many characters
+before the partially matched string, all should be well. (Of course, near the
+start of the subject, fewer characters may be present; in that case all
+characters should be retained.)
.P
+From release 8.33, there is a more accurate way of deciding which characters to
+retain. Instead of subtracting the length of the longest lookbehind from the
+earliest inspected character (\fIoffsets[0]\fP), the match start position
+(\fIoffsets[2]\fP) should be used, and the next match attempt started at the
+\fIoffsets[2]\fP character by setting the \fIstartoffset\fP argument of
+\fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP.
+.P
+For example, if the pattern "(?<=123)abc" is partially
+matched against the string "xx123a", the three offset values returned are 2, 6,
+and 5. This indicates that the matching process that gave a partial match
+started at offset 5, but the characters "123a" were all inspected. The maximum
+lookbehind for that pattern is 3, so taking that away from 5 shows that we need
+only keep "123a", and the next match attempt can be started at offset 3 (that
+is, at "a") when further characters have been added. When the match start is
+not the earliest inspected character, \fBpcretest\fP shows it explicitly:
+.sp
+ re> "(?<=123)abc"
+ data> xx123a\eP\eP
+ Partial match at offset 5: 123a
+.P
3. Because a partial match must always contain at least one character, what
might be considered a partial match of an empty string actually gives a "no
match" result. For example:
@@ -440,6 +462,6 @@
.rs
.sp
.nf
-Last updated: 24 June 2012
-Copyright (c) 1997-2012 University of Cambridge.
+Last updated: 20 February 2013
+Copyright (c) 1997-2013 University of Cambridge.
.fi
Modified: code/trunk/pcre_dfa_exec.c
===================================================================
--- code/trunk/pcre_dfa_exec.c 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/pcre_dfa_exec.c 2013-02-20 17:42:03 UTC (rev 1251)
@@ -7,7 +7,7 @@
below for why this module is different).
Written by Philip Hazel
- Copyright (c) 1997-2012 University of Cambridge
+ Copyright (c) 1997-2013 University of Cambridge
-----------------------------------------------------------------------------
Redistribution and use in source and binary forms, with or without
@@ -3023,15 +3023,7 @@
ptr > md->start_used_ptr) /* Inspected non-empty string */
)
)
- {
- if (offsetcount >= 2)
- {
- offsets[0] = (int)(md->start_used_ptr - start_subject);
- offsets[1] = (int)(end_subject - start_subject);
- }
match_count = PCRE_ERROR_PARTIAL;
- }
-
DPRINTF(("%.*sEnd of internal_dfa_exec %d: returning %d\n"
"%.*s---------------------\n\n", rlevel*2-2, SP, rlevel, match_count,
rlevel*2-2, SP));
@@ -3545,7 +3537,17 @@
/* Anything other than "no match" means we are done, always; otherwise, carry
on only if not anchored. */
- if (rc != PCRE_ERROR_NOMATCH || anchored) return rc;
+ if (rc != PCRE_ERROR_NOMATCH || anchored)
+ {
+ if (rc == PCRE_ERROR_PARTIAL && offsetcount >= 2)
+ {
+ offsets[0] = (int)(md->start_used_ptr - (PCRE_PUCHAR)subject);
+ offsets[1] = (int)(end_subject - (PCRE_PUCHAR)subject);
+ if (offsetcount > 2)
+ offsets[2] = (int)(current_subject - (PCRE_PUCHAR)subject);
+ }
+ return rc;
+ }
/* Advance to the next subject character unless we are at the end of a line
and firstline is set. */
Modified: code/trunk/pcre_exec.c
===================================================================
--- code/trunk/pcre_exec.c 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/pcre_exec.c 2013-02-20 17:42:03 UTC (rev 1251)
@@ -6,7 +6,7 @@
and semantics are as close as possible to those of the Perl 5 language.
Written by Philip Hazel
- Copyright (c) 1997-2012 University of Cambridge
+ Copyright (c) 1997-2013 University of Cambridge
-----------------------------------------------------------------------------
Redistribution and use in source and binary forms, with or without
@@ -6286,6 +6286,7 @@
PCRE_PUCHAR start_match = (PCRE_PUCHAR)subject + start_offset;
PCRE_PUCHAR end_subject;
PCRE_PUCHAR start_partial = NULL;
+PCRE_PUCHAR match_partial;
PCRE_PUCHAR req_char_ptr = start_match - 1;
const pcre_study_data *study;
@@ -6837,7 +6838,11 @@
md->match_function_type = 0;
md->end_offset_top = 0;
rc = match(start_match, md->start_code, start_match, 2, md, NULL, 0);
- if (md->hitend && start_partial == NULL) start_partial = md->start_used_ptr;
+ if (md->hitend && start_partial == NULL)
+ {
+ start_partial = md->start_used_ptr;
+ match_partial = start_match;
+ }
switch(rc)
{
@@ -7045,6 +7050,8 @@
{
offsets[0] = (int)(start_partial - (PCRE_PUCHAR)subject);
offsets[1] = (int)(end_subject - (PCRE_PUCHAR)subject);
+ if (offsetcount > 2)
+ offsets[2] = (int)(match_partial - (PCRE_PUCHAR)subject);
}
rc = PCRE_ERROR_PARTIAL;
}
Modified: code/trunk/pcretest.c
===================================================================
--- code/trunk/pcretest.c 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/pcretest.c 2013-02-20 17:42:03 UTC (rev 1251)
@@ -5263,14 +5263,17 @@
}
}
- /* There was a partial match */
+ /* There was a partial match. If the bumpalong point is not the same as
+ the first inspected character, show the offset explicitly. */
else if (count == PCRE_ERROR_PARTIAL)
{
- if (markptr == NULL) fprintf(outfile, "Partial match");
- else
+ fprintf(outfile, "Partial match");
+ if (use_size_offsets > 2 && use_offsets[0] != use_offsets[2])
+ fprintf(outfile, " at offset %d", use_offsets[2]);
+ if (markptr != NULL)
{
- fprintf(outfile, "Partial match, mark=");
+ fprintf(outfile, ", mark=");
PCHARSV(markptr, 0, -1, outfile);
}
if (use_size_offsets > 1)
Modified: code/trunk/testdata/testinput2
===================================================================
--- code/trunk/testdata/testinput2 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/testdata/testinput2 2013-02-20 17:42:03 UTC (rev 1251)
@@ -3836,5 +3836,13 @@
/(ab)/
ab\O3
ab\O2
+
+/(?<=123)(*MARK:xx)abc/K
+ xxxx123a\P\P
+ xxxx123a\P
+
+/123\Kabc/
+ xxxx123a\P\P
+ xxxx123a\P
/-- End of testinput2 --/
Modified: code/trunk/testdata/testoutput2
===================================================================
--- code/trunk/testdata/testoutput2 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/testdata/testoutput2 2013-02-20 17:42:03 UTC (rev 1251)
@@ -9719,17 +9719,17 @@
xyzabc123pqr
0: 123
xyzabc12\P
-Partial match: abc12
+Partial match at offset 6: abc12
xyzabc12\P\P
-Partial match: abc12
+Partial match at offset 6: abc12
/\babc\b/
+++abc+++
0: abc
+++ab\P
-Partial match: +ab
+Partial match at offset 3: +ab
+++ab\P\P
-Partial match: +ab
+Partial match at offset 3: +ab
/(?&word)(?&element)(?(DEFINE)(?<element><[^m][^>]>[^<])(?<word>\w*+))/BZ
------------------------------------------------------------------
@@ -10840,7 +10840,7 @@
/(?<=abc)def/
abc\P\P
-Partial match: abc
+Partial match at offset 3: abc
/abc$/
abc
@@ -12599,5 +12599,17 @@
ab\O2
Matched, but too many substrings
0: ab
+
+/(?<=123)(*MARK:xx)abc/K
+ xxxx123a\P\P
+Partial match at offset 7, mark=xx: 123a
+ xxxx123a\P
+Partial match at offset 7, mark=xx: 123a
+
+/123\Kabc/
+ xxxx123a\P\P
+Partial match: 123a
+ xxxx123a\P
+Partial match: 123a
/-- End of testinput2 --/
Modified: code/trunk/testdata/testoutput8
===================================================================
--- code/trunk/testdata/testoutput8 2013-02-20 09:59:29 UTC (rev 1250)
+++ code/trunk/testdata/testoutput8 2013-02-20 17:42:03 UTC (rev 1251)
@@ -985,7 +985,7 @@
xyzfo\P
No match
foob\P\>2
-Partial match: foob
+Partial match at offset 3: foob
foobar...\R\P\>4
0: ar
xyzfo\P
@@ -7466,17 +7466,17 @@
xyzabc123pqr
0: 123
xyzabc12\P
-Partial match: abc12
+Partial match at offset 6: abc12
xyzabc12\P\P
-Partial match: abc12
+Partial match at offset 6: abc12
/\babc\b/
+++abc+++
0: abc
+++ab\P
-Partial match: +ab
+Partial match at offset 3: +ab
+++ab\P\P
-Partial match: +ab
+Partial match at offset 3: +ab
/(?=C)/g+
ABCDECBA
@@ -7625,7 +7625,7 @@
/(?<=abc)def/
abc\P\P
-Partial match: abc
+Partial match at offset 3: abc
/abc$/
abc
