Revision: 1018
http://www.exim.org/viewvc/pcre2?view=rev&revision=1018
Author: ph10
Date: 2018-09-24 17:23:53 +0100 (Mon, 24 Sep 2018)
Log Message:
-----------
Implement Perl 5.28's alphabetic lookaround syntax, e.g. (*pla:...) and also
(*atomic:...).
Modified Paths:
--------------
code/trunk/ChangeLog
code/trunk/doc/html/pcre2pattern.html
code/trunk/doc/html/pcre2syntax.html
code/trunk/doc/pcre2.txt
code/trunk/doc/pcre2pattern.3
code/trunk/doc/pcre2syntax.3
code/trunk/maint/pcre2_chartables.c.non-standard
code/trunk/perltest.sh
code/trunk/src/dftables.c
code/trunk/src/pcre2.h.in
code/trunk/src/pcre2_chartables.c.dist
code/trunk/src/pcre2_compile.c
code/trunk/src/pcre2_error.c
code/trunk/src/pcre2_internal.h
code/trunk/src/pcre2_maketables.c
code/trunk/testdata/testinput1
code/trunk/testdata/testinput2
code/trunk/testdata/testoutput1
code/trunk/testdata/testoutput12-16
code/trunk/testdata/testoutput12-32
code/trunk/testdata/testoutput2
Modified: code/trunk/ChangeLog
===================================================================
--- code/trunk/ChangeLog 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/ChangeLog 2018-09-24 16:23:53 UTC (rev 1018)
@@ -5,8 +5,8 @@
Version 10.33-RC1 15-September-2018
-----------------------------------
-1. Added "allvector" to pcre2test to make it easy to check the part of the
-ovector that shouldn't be changed, in particular after substitute and failed or
+1. Added "allvector" to pcre2test to make it easy to check the part of the
+ovector that shouldn't be changed, in particular after substitute and failed or
partial matches.
2. Fix subject buffer overread in JIT when UTF is disabled and \X or \R has
@@ -15,7 +15,7 @@
3. Added support for callouts from pcre2_substitute().
4. The POSIX functions are now all called pcre2_regcomp() etc., with wrappers
-that use the standard POSIX names. This should help avoid linking with the
+that use the standard POSIX names. This should help avoid linking with the
wrong library in some environments.
5. Fix an xclass matching issue in JIT.
@@ -22,7 +22,15 @@
6. Implement PCRE2_EXTRA_ESCAPED_CR_IS_LF (see Bugzilla 2315).
+7. Implement the Perl 5.28 experimental alphabetic names for atomic groups and
+lookaround assertions, for example, (*pla:...) and (*atomic:...). These are
+characterized by a lower case letter following (* and to simplify coding for
+this, the character tables created by pcre2_maketables() were updated to add a
+new "is lower case letter" bit. At the same time, the now unused "is
+hexadecimal digit" bit was removed. The default tables in
+src/pcre2_chartables.c.dist are updated.
+
Version 10.32 10-September-2018
-------------------------------
Modified: code/trunk/doc/html/pcre2pattern.html
===================================================================
--- code/trunk/doc/html/pcre2pattern.html 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/doc/html/pcre2pattern.html 2018-09-24 16:23:53 UTC (rev 1018)
@@ -2120,6 +2120,11 @@
<pre>
(?>\d+)foo
</pre>
+Perl 5.28 introduced an experimental alphabetic form starting with (* which may
+be easier to remember:
+<pre>
+ (*atomic:\d+)foo
+</pre>
This kind of parenthesis "locks up" the part of the pattern it contains once
it has matched, and a failure further into the pattern is prevented from
backtracking into it. Backtracking past it to previous items, however, works as
@@ -2342,13 +2347,19 @@
<P>
More complicated assertions are coded as subpatterns. There are two kinds:
those that look ahead of the current position in the subject string, and those
-that look behind it, and in each case an assertion may be positive (must
-succeed for matching to continue) or negative (must not succeed for matching to
-continue). An assertion subpattern is matched in the normal way, except that,
-when matching continues after a successful assertion, the matching position in
-the subject string is as it was before the assertion was processed.
+that look behind it, and in each case an assertion may be positive (must match
+for the assertion to be true) or negative (must not match for the assertion to
+be true). An assertion subpattern is matched in the normal way, and if it is
+true, matching continues after it, but with the matching position in the
+subject string is was it was before the assertion was processed.
</P>
<P>
+A lookaround assertion may also appear as the condition in a
+<a href="#conditions">conditional subpattern</a>
+(see below). In this case, the result of matching the assertion determines
+which branch of the condition is followed.
+</P>
+<P>
Assertion subpatterns are not capturing subpatterns. If an assertion contains
capturing subpatterns within it, these are counted for the purposes of
numbering the capturing subpatterns in the whole pattern. Within each branch of
@@ -2359,7 +2370,7 @@
<P>
When a branch within an assertion fails to match, any substrings that were
captured are discarded (as happens with any pattern branch that fails to
-match). A negative assertion succeeds only when all its branches fail to match;
+match). A negative assertion is true only when all its branches fail to match;
this means that no captured substrings are ever retained after a successful
negative assertion. When an assertion contains a matching branch, what happens
depends on the type of assertion.
@@ -2368,7 +2379,7 @@
For a positive assertion, internally captured substrings in the successful
branch are retained, and matching continues with the next pattern item after
the assertion. For a negative assertion, a matching branch means that the
-assertion has failed. If the assertion is being used as a condition in a
+assertion is not true. If such an assertion is being used as a condition in a
<a href="#conditions">conditional subpattern</a>
(see below), captured substrings are retained, because matching continues with
the "no" branch of the condition. For other failing negative assertions,
@@ -2398,6 +2409,25 @@
The assertion is obeyed just once when encountered during matching.
</P>
<br><b>
+Alphabetic assertion names
+</b><br>
+<P>
+Traditionally, symbolic sequences such as (?= and (?<= have been used to specify
+lookaround assertions. Perl 5.28 introduced some experimental alphabetic
+alternatives which might be easier to remember. They all start with (* instead
+of (? and must be written using lower case letters. PCRE2 supports the
+following synonyms:
+<pre>
+ (*positive_lookahead: or (*pla: is the same as (?=
+ (*negative_lookahead: or (*nla: is the same as (?!
+ (*positive_lookbehind: or (*plb: is the same as (?<=
+ (*negative_lookbehind: or (*nlb: is the same as (?<!
+</pre>
+For example, (*pla:foo) is the same assertion as (?=foo). However, in the
+following sections, the various assertions are described using the original
+symbolic forms.
+</P>
+<br><b>
Lookahead assertions
</b><br>
<P>
@@ -3630,7 +3660,7 @@
</P>
<br><a name="SEC30" href="#TOC1">REVISION</a><br>
<P>
-Last updated: 21 September 2018
+Last updated: 24 September 2018
<br>
Copyright © 1997-2018 University of Cambridge.
<br>
Modified: code/trunk/doc/html/pcre2syntax.html
===================================================================
--- code/trunk/doc/html/pcre2syntax.html 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/doc/html/pcre2syntax.html 2018-09-24 16:23:53 UTC (rev 1018)
@@ -436,6 +436,7 @@
<P>
<pre>
(?>...) atomic, non-capturing group
+ (*atomic:...) atomic, non-capturing group
</PRE>
</P>
<br><a name="SEC15" href="#TOC1">COMMENT</a><br>
@@ -514,12 +515,23 @@
<br><a name="SEC19" href="#TOC1">LOOKAHEAD AND LOOKBEHIND ASSERTIONS</a><br>
<P>
<pre>
- (?=...) positive look ahead
- (?!...) negative look ahead
- (?<=...) positive look behind
- (?<!...) negative look behind
+ (?=...) )
+ (*pla:...) ) positive lookahead
+ (*positive_lookahead:...) )
+
+ (?!...) )
+ (*nla:...) ) negative lookahead
+ (*negative_lookahead:...) )
+
+ (?<=...) )
+ (*plb:...) ) positive lookbehind
+ (*positive_lookbehind:...) )
+
+ (?<!...) )
+ (*nlb:...) ) negative lookbehind
+ (*negative_lookbehind:...) )
</pre>
-Each top-level branch of a look behind must be of a fixed length.
+Each top-level branch of a lookbehind must be of a fixed length.
</P>
<br><a name="SEC20" href="#TOC1">BACKREFERENCES</a><br>
<P>
@@ -634,7 +646,7 @@
</P>
<br><a name="SEC27" href="#TOC1">REVISION</a><br>
<P>
-Last updated: 02 September 2018
+Last updated: 24 September 2018
<br>
Copyright © 1997-2018 University of Cambridge.
<br>
Modified: code/trunk/doc/pcre2.txt
===================================================================
--- code/trunk/doc/pcre2.txt 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/doc/pcre2.txt 2018-09-24 16:23:53 UTC (rev 1018)
@@ -7760,27 +7760,32 @@
(?>\d+)foo
- This kind of parenthesis "locks up" the part of the pattern it con-
- tains once it has matched, and a failure further into the pattern is
- prevented from backtracking into it. Backtracking past it to previous
+ Perl 5.28 introduced an experimental alphabetic form starting with (*
+ which may be easier to remember:
+
+ (*atomic:\d+)foo
+
+ This kind of parenthesis "locks up" the part of the pattern it con-
+ tains once it has matched, and a failure further into the pattern is
+ prevented from backtracking into it. Backtracking past it to previous
items, however, works as normal.
- An alternative description is that a subpattern of this type matches
- exactly the string of characters that an identical standalone pattern
+ An alternative description is that a subpattern of this type matches
+ exactly the string of characters that an identical standalone pattern
would match, if anchored at the current point in the subject string.
Atomic grouping subpatterns are not capturing subpatterns. Simple cases
such as the above example can be thought of as a maximizing repeat that
- must swallow everything it can. So, while both \d+ and \d+? are pre-
- pared to adjust the number of digits they match in order to make the
+ must swallow everything it can. So, while both \d+ and \d+? are pre-
+ pared to adjust the number of digits they match in order to make the
rest of the pattern match, (?>\d+) can only match an entire sequence of
digits.
- Atomic groups in general can of course contain arbitrarily complicated
- subpatterns, and can be nested. However, when the subpattern for an
+ Atomic groups in general can of course contain arbitrarily complicated
+ subpatterns, and can be nested. However, when the subpattern for an
atomic group is just a single repeated item, as in the example above, a
- simpler notation, called a "possessive quantifier" can be used. This
- consists of an additional + character following a quantifier. Using
+ simpler notation, called a "possessive quantifier" can be used. This
+ consists of an additional + character following a quantifier. Using
this notation, the previous example can be rewritten as
\d++foo
@@ -7790,46 +7795,46 @@
(abc|xyz){2,3}+
- Possessive quantifiers are always greedy; the setting of the
- PCRE2_UNGREEDY option is ignored. They are a convenient notation for
- the simpler forms of atomic group. However, there is no difference in
+ Possessive quantifiers are always greedy; the setting of the
+ PCRE2_UNGREEDY option is ignored. They are a convenient notation for
+ the simpler forms of atomic group. However, there is no difference in
the meaning of a possessive quantifier and the equivalent atomic group,
- though there may be a performance difference; possessive quantifiers
+ though there may be a performance difference; possessive quantifiers
should be slightly faster.
- The possessive quantifier syntax is an extension to the Perl 5.8 syn-
- tax. Jeffrey Friedl originated the idea (and the name) in the first
+ The possessive quantifier syntax is an extension to the Perl 5.8 syn-
+ tax. Jeffrey Friedl originated the idea (and the name) in the first
edition of his book. Mike McCloskey liked it, so implemented it when he
built Sun's Java package, and PCRE1 copied it from there. It ultimately
found its way into Perl at release 5.10.
- PCRE2 has an optimization that automatically "possessifies" certain
- simple pattern constructs. For example, the sequence A+B is treated as
- A++B because there is no point in backtracking into a sequence of A's
+ PCRE2 has an optimization that automatically "possessifies" certain
+ simple pattern constructs. For example, the sequence A+B is treated as
+ A++B because there is no point in backtracking into a sequence of A's
when B must follow. This feature can be disabled by the PCRE2_NO_AUTO-
POSSESS option, or starting the pattern with (*NO_AUTO_POSSESS).
- When a pattern contains an unlimited repeat inside a subpattern that
- can itself be repeated an unlimited number of times, the use of an
- atomic group is the only way to avoid some failing matches taking a
+ When a pattern contains an unlimited repeat inside a subpattern that
+ can itself be repeated an unlimited number of times, the use of an
+ atomic group is the only way to avoid some failing matches taking a
very long time indeed. The pattern
(\D+|<\d+>)*[!?]
- matches an unlimited number of substrings that either consist of non-
- digits, or digits enclosed in <>, followed by either ! or ?. When it
+ matches an unlimited number of substrings that either consist of non-
+ digits, or digits enclosed in <>, followed by either ! or ?. When it
matches, it runs quickly. However, if it is applied to
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- it takes a long time before reporting failure. This is because the
- string can be divided between the internal \D+ repeat and the external
- * repeat in a large number of ways, and all have to be tried. (The
- example uses [!?] rather than a single character at the end, because
- both PCRE2 and Perl have an optimization that allows for fast failure
- when a single character is used. They remember the last single charac-
- ter that is required for a match, and fail early if it is not present
- in the string.) If the pattern is changed so that it uses an atomic
+ it takes a long time before reporting failure. This is because the
+ string can be divided between the internal \D+ repeat and the external
+ * repeat in a large number of ways, and all have to be tried. (The
+ example uses [!?] rather than a single character at the end, because
+ both PCRE2 and Perl have an optimization that allows for fast failure
+ when a single character is used. They remember the last single charac-
+ ter that is required for a match, and fail early if it is not present
+ in the string.) If the pattern is changed so that it uses an atomic
group, like this:
((?>\D+)|<\d+>)*[!?]
@@ -7840,29 +7845,29 @@
BACKREFERENCES
Outside a character class, a backslash followed by a digit greater than
- 0 (and possibly further digits) is a backreference to a capturing sub-
- pattern earlier (that is, to its left) in the pattern, provided there
+ 0 (and possibly further digits) is a backreference to a capturing sub-
+ pattern earlier (that is, to its left) in the pattern, provided there
have been that many previous capturing left parentheses.
- However, if the decimal number following the backslash is less than 8,
- it is always taken as a backreference, and causes an error only if
- there are not that many capturing left parentheses in the entire pat-
- tern. In other words, the parentheses that are referenced need not be
- to the left of the reference for numbers less than 8. A "forward back-
- reference" of this type can make sense when a repetition is involved
- and the subpattern to the right has participated in an earlier itera-
+ However, if the decimal number following the backslash is less than 8,
+ it is always taken as a backreference, and causes an error only if
+ there are not that many capturing left parentheses in the entire pat-
+ tern. In other words, the parentheses that are referenced need not be
+ to the left of the reference for numbers less than 8. A "forward back-
+ reference" of this type can make sense when a repetition is involved
+ and the subpattern to the right has participated in an earlier itera-
tion.
- It is not possible to have a numerical "forward backreference" to a
- subpattern whose number is 8 or more using this syntax because a
- sequence such as \50 is interpreted as a character defined in octal.
+ It is not possible to have a numerical "forward backreference" to a
+ subpattern whose number is 8 or more using this syntax because a
+ sequence such as \50 is interpreted as a character defined in octal.
See the subsection entitled "Non-printing characters" above for further
- details of the handling of digits following a backslash. There is no
- such problem when named parentheses are used. A backreference to any
+ details of the handling of digits following a backslash. There is no
+ such problem when named parentheses are used. A backreference to any
subpattern is possible using named parentheses (see below).
- Another way of avoiding the ambiguity inherent in the use of digits
- following a backslash is to use the \g escape sequence. This escape
+ Another way of avoiding the ambiguity inherent in the use of digits
+ following a backslash is to use the \g escape sequence. This escape
must be followed by a signed or unsigned number, optionally enclosed in
braces. These examples are all identical:
@@ -7870,9 +7875,9 @@
(ring), \g1
(ring), \g{1}
- An unsigned number specifies an absolute reference without the ambigu-
+ An unsigned number specifies an absolute reference without the ambigu-
ity that is present in the older syntax. It is also useful when literal
- digits follow the reference. A signed number is a relative reference.
+ digits follow the reference. A signed number is a relative reference.
Consider this example:
(abc(def)ghi)\g{-1}
@@ -7879,37 +7884,37 @@
The sequence \g{-1} is a reference to the most recently started captur-
ing subpattern before \g, that is, is it equivalent to \2 in this exam-
- ple. Similarly, \g{-2} would be equivalent to \1. The use of relative
- references can be helpful in long patterns, and also in patterns that
- are created by joining together fragments that contain references
+ ple. Similarly, \g{-2} would be equivalent to \1. The use of relative
+ references can be helpful in long patterns, and also in patterns that
+ are created by joining together fragments that contain references
within themselves.
- The sequence \g{+1} is a reference to the next capturing subpattern.
- This kind of forward reference can be useful it patterns that repeat.
+ The sequence \g{+1} is a reference to the next capturing subpattern.
+ This kind of forward reference can be useful it patterns that repeat.
Perl does not support the use of + in this way.
A backreference matches whatever actually matched the capturing subpat-
- tern in the current subject string, rather than anything matching the
- subpattern itself (see "Subpatterns as subroutines" below for a way of
+ tern in the current subject string, rather than anything matching the
+ subpattern itself (see "Subpatterns as subroutines" below for a way of
doing that). So the pattern
(sens|respons)e and \1ibility
- matches "sense and sensibility" and "response and responsibility", but
- not "sense and responsibility". If caseful matching is in force at the
- time of the backreference, the case of letters is relevant. For exam-
+ matches "sense and sensibility" and "response and responsibility", but
+ not "sense and responsibility". If caseful matching is in force at the
+ time of the backreference, the case of letters is relevant. For exam-
ple,
((?i)rah)\s+\1
- matches "rah rah" and "RAH RAH", but not "RAH rah", even though the
+ matches "rah rah" and "RAH RAH", but not "RAH rah", even though the
original capturing subpattern is matched caselessly.
- There are several different ways of writing backreferences to named
- subpatterns. The .NET syntax \k{name} and the Perl syntax \k<name> or
- \k'name' are supported, as is the Python syntax (?P=name). Perl 5.10's
- unified backreference syntax, in which \g can be used for both numeric
- and named references, is also supported. We could rewrite the above
+ There are several different ways of writing backreferences to named
+ subpatterns. The .NET syntax \k{name} and the Perl syntax \k<name> or
+ \k'name' are supported, as is the Python syntax (?P=name). Perl 5.10's
+ unified backreference syntax, in which \g can be used for both numeric
+ and named references, is also supported. We could rewrite the above
example in any of the following ways:
(?<p1>(?i)rah)\s+\k<p1>
@@ -7917,32 +7922,32 @@
(?P<p1>(?i)rah)\s+(?P=p1)
(?<p1>(?i)rah)\s+\g{p1}
- A subpattern that is referenced by name may appear in the pattern
+ A subpattern that is referenced by name may appear in the pattern
before or after the reference.
- There may be more than one backreference to the same subpattern. If a
- subpattern has not actually been used in a particular match, any back-
+ There may be more than one backreference to the same subpattern. If a
+ subpattern has not actually been used in a particular match, any back-
references to it always fail by default. For example, the pattern
(a|(bc))\2
- always fails if it starts to match "a" rather than "bc". However, if
+ always fails if it starts to match "a" rather than "bc". However, if
the PCRE2_MATCH_UNSET_BACKREF option is set at compile time, a backref-
erence to an unset value matches an empty string.
- Because there may be many capturing parentheses in a pattern, all dig-
- its following a backslash are taken as part of a potential backrefer-
- ence number. If the pattern continues with a digit character, some
- delimiter must be used to terminate the backreference. If the
- PCRE2_EXTENDED or PCRE2_EXTENDED_MORE option is set, this can be white
- space. Otherwise, the \g{ syntax or an empty comment (see "Comments"
+ Because there may be many capturing parentheses in a pattern, all dig-
+ its following a backslash are taken as part of a potential backrefer-
+ ence number. If the pattern continues with a digit character, some
+ delimiter must be used to terminate the backreference. If the
+ PCRE2_EXTENDED or PCRE2_EXTENDED_MORE option is set, this can be white
+ space. Otherwise, the \g{ syntax or an empty comment (see "Comments"
below) can be used.
Recursive backreferences
- A backreference that occurs inside the parentheses to which it refers
- fails when the subpattern is first used, so, for example, (a\1) never
- matches. However, such references can be useful inside repeated sub-
+ A backreference that occurs inside the parentheses to which it refers
+ fails when the subpattern is first used, so, for example, (a\1) never
+ matches. However, such references can be useful inside repeated sub-
patterns. For example, the pattern
(a|b\1)+
@@ -7950,76 +7955,97 @@
matches any number of "a"s and also "aba", "ababbaa" etc. At each iter-
ation of the subpattern, the backreference matches the character string
corresponding to the previous iteration. In order for this to work, the
- pattern must be such that the first iteration does not need to match
- the backreference. This can be done using alternation, as in the exam-
+ pattern must be such that the first iteration does not need to match
+ the backreference. This can be done using alternation, as in the exam-
ple above, or by a quantifier with a minimum of zero.
- Backreferences of this type cause the group that they reference to be
- treated as an atomic group. Once the whole group has been matched, a
- subsequent matching failure cannot cause backtracking into the middle
+ Backreferences of this type cause the group that they reference to be
+ treated as an atomic group. Once the whole group has been matched, a
+ subsequent matching failure cannot cause backtracking into the middle
of the group.
ASSERTIONS
- An assertion is a test on the characters following or preceding the
+ An assertion is a test on the characters following or preceding the
current matching point that does not consume any characters. The simple
- assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are described
+ assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are described
above.
- More complicated assertions are coded as subpatterns. There are two
- kinds: those that look ahead of the current position in the subject
- string, and those that look behind it, and in each case an assertion
- may be positive (must succeed for matching to continue) or negative
- (must not succeed for matching to continue). An assertion subpattern is
- matched in the normal way, except that, when matching continues after a
- successful assertion, the matching position in the subject string is as
+ More complicated assertions are coded as subpatterns. There are two
+ kinds: those that look ahead of the current position in the subject
+ string, and those that look behind it, and in each case an assertion
+ may be positive (must match for the assertion to be true) or negative
+ (must not match for the assertion to be true). An assertion subpattern
+ is matched in the normal way, and if it is true, matching continues
+ after it, but with the matching position in the subject string is was
it was before the assertion was processed.
- Assertion subpatterns are not capturing subpatterns. If an assertion
- contains capturing subpatterns within it, these are counted for the
- purposes of numbering the capturing subpatterns in the whole pattern.
- Within each branch of an assertion, locally captured substrings may be
+ A lookaround assertion may also appear as the condition in a condi-
+ tional subpattern (see below). In this case, the result of matching the
+ assertion determines which branch of the condition is followed.
+
+ Assertion subpatterns are not capturing subpatterns. If an assertion
+ contains capturing subpatterns within it, these are counted for the
+ purposes of numbering the capturing subpatterns in the whole pattern.
+ Within each branch of an assertion, locally captured substrings may be
referenced in the usual way. For example, a sequence such as (.)\g{-1}
can be used to check that two adjacent characters are the same.
- When a branch within an assertion fails to match, any substrings that
- were captured are discarded (as happens with any pattern branch that
- fails to match). A negative assertion succeeds only when all its
+ When a branch within an assertion fails to match, any substrings that
+ were captured are discarded (as happens with any pattern branch that
+ fails to match). A negative assertion is true only when all its
branches fail to match; this means that no captured substrings are ever
- retained after a successful negative assertion. When an assertion con-
+ retained after a successful negative assertion. When an assertion con-
tains a matching branch, what happens depends on the type of assertion.
- For a positive assertion, internally captured substrings in the suc-
- cessful branch are retained, and matching continues with the next pat-
- tern item after the assertion. For a negative assertion, a matching
- branch means that the assertion has failed. If the assertion is being
- used as a condition in a conditional subpattern (see below), captured
- substrings are retained, because matching continues with the "no"
+ For a positive assertion, internally captured substrings in the suc-
+ cessful branch are retained, and matching continues with the next pat-
+ tern item after the assertion. For a negative assertion, a matching
+ branch means that the assertion is not true. If such an assertion is
+ being used as a condition in a conditional subpattern (see below), cap-
+ tured substrings are retained, because matching continues with the "no"
branch of the condition. For other failing negative assertions, control
passes to the previous backtracking point, thus discarding any captured
strings within the assertion.
- For compatibility with Perl, most assertion subpatterns may be
- repeated; though it makes no sense to assert the same thing several
- times, the side effect of capturing parentheses may occasionally be
- useful. However, an assertion that forms the condition for a condi-
- tional subpattern may not be quantified. In practice, for other asser-
+ For compatibility with Perl, most assertion subpatterns may be
+ repeated; though it makes no sense to assert the same thing several
+ times, the side effect of capturing parentheses may occasionally be
+ useful. However, an assertion that forms the condition for a condi-
+ tional subpattern may not be quantified. In practice, for other asser-
tions, there only three cases:
- (1) If the quantifier is {0}, the assertion is never obeyed during
- matching. However, it may contain internal capturing parenthesized
+ (1) If the quantifier is {0}, the assertion is never obeyed during
+ matching. However, it may contain internal capturing parenthesized
groups that are called from elsewhere via the subroutine mechanism.
- (2) If quantifier is {0,n} where n is greater than zero, it is treated
- as if it were {0,1}. At run time, the rest of the pattern match is
+ (2) If quantifier is {0,n} where n is greater than zero, it is treated
+ as if it were {0,1}. At run time, the rest of the pattern match is
tried with and without the assertion, the order depending on the greed-
iness of the quantifier.
- (3) If the minimum repetition is greater than zero, the quantifier is
- ignored. The assertion is obeyed just once when encountered during
+ (3) If the minimum repetition is greater than zero, the quantifier is
+ ignored. The assertion is obeyed just once when encountered during
matching.
+ Alphabetic assertion names
+
+ Traditionally, symbolic sequences such as (?= and (?<= have been used
+ to specify lookaround assertions. Perl 5.28 introduced some experimen-
+ tal alphabetic alternatives which might be easier to remember. They all
+ start with (* instead of (? and must be written using lower case let-
+ ters. PCRE2 supports the following synonyms:
+
+ (*positive_lookahead: or (*pla: is the same as (?=
+ (*negative_lookahead: or (*nla: is the same as (?!
+ (*positive_lookbehind: or (*plb: is the same as (?<=
+ (*negative_lookbehind: or (*nlb: is the same as (?<!
+
+ For example, (*pla:foo) is the same assertion as (?=foo). However, in
+ the following sections, the various assertions are described using the
+ original symbolic forms.
+
Lookahead assertions
Lookahead assertions start with (?= for positive assertions and (?! for
@@ -8027,38 +8053,38 @@
\w+(?=;)
- matches a word followed by a semicolon, but does not include the semi-
+ matches a word followed by a semicolon, but does not include the semi-
colon in the match, and
foo(?!bar)
- matches any occurrence of "foo" that is not followed by "bar". Note
+ matches any occurrence of "foo" that is not followed by "bar". Note
that the apparently similar pattern
(?!foo)bar
- does not find an occurrence of "bar" that is preceded by something
- other than "foo"; it finds any occurrence of "bar" whatsoever, because
+ does not find an occurrence of "bar" that is preceded by something
+ other than "foo"; it finds any occurrence of "bar" whatsoever, because
the assertion (?!foo) is always true when the next three characters are
"bar". A lookbehind assertion is needed to achieve the other effect.
If you want to force a matching failure at some point in a pattern, the
- most convenient way to do it is with (?!) because an empty string
- always matches, so an assertion that requires there not to be an empty
+ most convenient way to do it is with (?!) because an empty string
+ always matches, so an assertion that requires there not to be an empty
string must always fail. The backtracking control verb (*FAIL) or (*F)
is a synonym for (?!).
Lookbehind assertions
- Lookbehind assertions start with (?<= for positive assertions and (?<!
+ Lookbehind assertions start with (?<= for positive assertions and (?<!
for negative assertions. For example,
(?<!foo)bar
- does find an occurrence of "bar" that is not preceded by "foo". The
- contents of a lookbehind assertion are restricted such that all the
+ does find an occurrence of "bar" that is not preceded by "foo". The
+ contents of a lookbehind assertion are restricted such that all the
strings it matches must have a fixed length. However, if there are sev-
- eral top-level alternatives, they do not all have to have the same
+ eral top-level alternatives, they do not all have to have the same
fixed length. Thus
(?<=bullock|donkey)
@@ -8067,66 +8093,66 @@
(?<!dogs?|cats?)
- causes an error at compile time. Branches that match different length
- strings are permitted only at the top level of a lookbehind assertion.
+ causes an error at compile time. Branches that match different length
+ strings are permitted only at the top level of a lookbehind assertion.
This is an extension compared with Perl, which requires all branches to
match the same length of string. An assertion such as
(?<=ab(c|de))
- is not permitted, because its single top-level branch can match two
- different lengths, but it is acceptable to PCRE2 if rewritten to use
+ is not permitted, because its single top-level branch can match two
+ different lengths, but it is acceptable to PCRE2 if rewritten to use
two top-level branches:
(?<=abc|abde)
- In some cases, the escape sequence \K (see above) can be used instead
+ In some cases, the escape sequence \K (see above) can be used instead
of a lookbehind assertion to get round the fixed-length restriction.
- The implementation of lookbehind assertions is, for each alternative,
- to temporarily move the current position back by the fixed length and
+ The implementation of lookbehind assertions is, for each alternative,
+ to temporarily move the current position back by the fixed length and
then try to match. If there are insufficient characters before the cur-
rent position, the assertion fails.
- In UTF-8 and UTF-16 modes, PCRE2 does not allow the \C escape (which
- matches a single code unit even in a UTF mode) to appear in lookbehind
- assertions, because it makes it impossible to calculate the length of
- the lookbehind. The \X and \R escapes, which can match different num-
+ In UTF-8 and UTF-16 modes, PCRE2 does not allow the \C escape (which
+ matches a single code unit even in a UTF mode) to appear in lookbehind
+ assertions, because it makes it impossible to calculate the length of
+ the lookbehind. The \X and \R escapes, which can match different num-
bers of code units, are never permitted in lookbehinds.
- "Subroutine" calls (see below) such as (?2) or (?&X) are permitted in
- lookbehinds, as long as the subpattern matches a fixed-length string.
- However, recursion, that is, a "subroutine" call into a group that is
+ "Subroutine" calls (see below) such as (?2) or (?&X) are permitted in
+ lookbehinds, as long as the subpattern matches a fixed-length string.
+ However, recursion, that is, a "subroutine" call into a group that is
already active, is not supported.
Perl does not support backreferences in lookbehinds. PCRE2 does support
- them, but only if certain conditions are met. The
- PCRE2_MATCH_UNSET_BACKREF option must not be set, there must be no use
+ them, but only if certain conditions are met. The
+ PCRE2_MATCH_UNSET_BACKREF option must not be set, there must be no use
of (?| in the pattern (it creates duplicate subpattern numbers), and if
- the backreference is by name, the name must be unique. Of course, the
- referenced subpattern must itself be of fixed length. The following
+ the backreference is by name, the name must be unique. Of course, the
+ referenced subpattern must itself be of fixed length. The following
pattern matches words containing at least two characters that begin and
end with the same character:
\b(\w)\w++(?<=\1)
- Possessive quantifiers can be used in conjunction with lookbehind
+ Possessive quantifiers can be used in conjunction with lookbehind
assertions to specify efficient matching of fixed-length strings at the
end of subject strings. Consider a simple pattern such as
abcd$
- when applied to a long string that does not match. Because matching
- proceeds from left to right, PCRE2 will look for each "a" in the sub-
- ject and then see if what follows matches the rest of the pattern. If
+ when applied to a long string that does not match. Because matching
+ proceeds from left to right, PCRE2 will look for each "a" in the sub-
+ ject and then see if what follows matches the rest of the pattern. If
the pattern is specified as
^.*abcd$
- the initial .* matches the entire string at first, but when this fails
+ the initial .* matches the entire string at first, but when this fails
(because there is no following "a"), it backtracks to match all but the
- last character, then all but the last two characters, and so on. Once
- again the search for "a" covers the entire string, from right to left,
+ last character, then all but the last two characters, and so on. Once
+ again the search for "a" covers the entire string, from right to left,
so we are no better off. However, if the pattern is written as
^.*+(?<=abcd)
@@ -8133,8 +8159,8 @@
there can be no backtracking for the .*+ item because of the possessive
quantifier; it can match only the entire string. The subsequent lookbe-
- hind assertion does a single test on the last four characters. If it
- fails, the match fails immediately. For long strings, this approach
+ hind assertion does a single test on the last four characters. If it
+ fails, the match fails immediately. For long strings, this approach
makes a significant difference to the processing time.
Using multiple assertions
@@ -8143,18 +8169,18 @@
(?<=\d{3})(?<!999)foo
- matches "foo" preceded by three digits that are not "999". Notice that
- each of the assertions is applied independently at the same point in
- the subject string. First there is a check that the previous three
- characters are all digits, and then there is a check that the same
+ matches "foo" preceded by three digits that are not "999". Notice that
+ each of the assertions is applied independently at the same point in
+ the subject string. First there is a check that the previous three
+ characters are all digits, and then there is a check that the same
three characters are not "999". This pattern does not match "foo" pre-
- ceded by six characters, the first of which are digits and the last
- three of which are not "999". For example, it doesn't match "123abc-
+ ceded by six characters, the first of which are digits and the last
+ three of which are not "999". For example, it doesn't match "123abc-
foo". A pattern to do that is
(?<=\d{3}...)(?<!999)foo
- This time the first assertion looks at the preceding six characters,
+ This time the first assertion looks at the preceding six characters,
checking that the first three are digits, and then the second assertion
checks that the preceding three characters are not "999".
@@ -8162,29 +8188,29 @@
(?<=(?<!foo)bar)baz
- matches an occurrence of "baz" that is preceded by "bar" which in turn
+ matches an occurrence of "baz" that is preceded by "bar" which in turn
is not preceded by "foo", while
(?<=\d{3}(?!999)...)foo
- is another pattern that matches "foo" preceded by three digits and any
+ is another pattern that matches "foo" preceded by three digits and any
three characters that are not "999".
CONDITIONAL SUBPATTERNS
- It is possible to cause the matching process to obey a subpattern con-
- ditionally or to choose between two alternative subpatterns, depending
- on the result of an assertion, or whether a specific capturing subpat-
- tern has already been matched. The two possible forms of conditional
+ It is possible to cause the matching process to obey a subpattern con-
+ ditionally or to choose between two alternative subpatterns, depending
+ on the result of an assertion, or whether a specific capturing subpat-
+ tern has already been matched. The two possible forms of conditional
subpattern are:
(?(condition)yes-pattern)
(?(condition)yes-pattern|no-pattern)
- If the condition is satisfied, the yes-pattern is used; otherwise the
- no-pattern (if present) is used. An absent no-pattern is equivalent to
- an empty string (it always matches). If there are more than two alter-
+ If the condition is satisfied, the yes-pattern is used; otherwise the
+ no-pattern (if present) is used. An absent no-pattern is equivalent to
+ an empty string (it always matches). If there are more than two alter-
natives in the subpattern, a compile-time error occurs. Each of the two
alternatives may itself contain nested subpatterns of any form, includ-
ing conditional subpatterns; the restriction to two alternatives
@@ -8194,57 +8220,57 @@
(?(1) (A|B|C) | (D | (?(2)E|F) | E) )
- There are five kinds of condition: references to subpatterns, refer-
- ences to recursion, two pseudo-conditions called DEFINE and VERSION,
+ There are five kinds of condition: references to subpatterns, refer-
+ ences to recursion, two pseudo-conditions called DEFINE and VERSION,
and assertions.
Checking for a used subpattern by number
- If the text between the parentheses consists of a sequence of digits,
+ If the text between the parentheses consists of a sequence of digits,
the condition is true if a capturing subpattern of that number has pre-
- viously matched. If there is more than one capturing subpattern with
- the same number (see the earlier section about duplicate subpattern
- numbers), the condition is true if any of them have matched. An alter-
- native notation is to precede the digits with a plus or minus sign. In
- this case, the subpattern number is relative rather than absolute. The
- most recently opened parentheses can be referenced by (?(-1), the next
- most recent by (?(-2), and so on. Inside loops it can also make sense
+ viously matched. If there is more than one capturing subpattern with
+ the same number (see the earlier section about duplicate subpattern
+ numbers), the condition is true if any of them have matched. An alter-
+ native notation is to precede the digits with a plus or minus sign. In
+ this case, the subpattern number is relative rather than absolute. The
+ most recently opened parentheses can be referenced by (?(-1), the next
+ most recent by (?(-2), and so on. Inside loops it can also make sense
to refer to subsequent groups. The next parentheses to be opened can be
- referenced as (?(+1), and so on. (The value zero in any of these forms
+ referenced as (?(+1), and so on. (The value zero in any of these forms
is not used; it provokes a compile-time error.)
- Consider the following pattern, which contains non-significant white
- space to make it more readable (assume the PCRE2_EXTENDED option) and
+ Consider the following pattern, which contains non-significant white
+ space to make it more readable (assume the PCRE2_EXTENDED option) and
to divide it into three parts for ease of discussion:
( \( )? [^()]+ (?(1) \) )
- The first part matches an optional opening parenthesis, and if that
+ The first part matches an optional opening parenthesis, and if that
character is present, sets it as the first captured substring. The sec-
- ond part matches one or more characters that are not parentheses. The
- third part is a conditional subpattern that tests whether or not the
- first set of parentheses matched. If they did, that is, if subject
- started with an opening parenthesis, the condition is true, and so the
- yes-pattern is executed and a closing parenthesis is required. Other-
- wise, since no-pattern is not present, the subpattern matches nothing.
- In other words, this pattern matches a sequence of non-parentheses,
+ ond part matches one or more characters that are not parentheses. The
+ third part is a conditional subpattern that tests whether or not the
+ first set of parentheses matched. If they did, that is, if subject
+ started with an opening parenthesis, the condition is true, and so the
+ yes-pattern is executed and a closing parenthesis is required. Other-
+ wise, since no-pattern is not present, the subpattern matches nothing.
+ In other words, this pattern matches a sequence of non-parentheses,
optionally enclosed in parentheses.
- If you were embedding this pattern in a larger one, you could use a
+ If you were embedding this pattern in a larger one, you could use a
relative reference:
...other stuff... ( \( )? [^()]+ (?(-1) \) ) ...
- This makes the fragment independent of the parentheses in the larger
+ This makes the fragment independent of the parentheses in the larger
pattern.
Checking for a used subpattern by name
- Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a
- used subpattern by name. For compatibility with earlier versions of
- PCRE1, which had this facility before Perl, the syntax (?(name)...) is
- also recognized. Note, however, that undelimited names consisting of
- the letter R followed by digits are ambiguous (see the following sec-
+ Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a
+ used subpattern by name. For compatibility with earlier versions of
+ PCRE1, which had this facility before Perl, the syntax (?(name)...) is
+ also recognized. Note, however, that undelimited names consisting of
+ the letter R followed by digits are ambiguous (see the following sec-
tion).
Rewriting the above example to use a named subpattern gives this:
@@ -8251,31 +8277,31 @@
(?<OPEN> \( )? [^()]+ (?(<OPEN>) \) )
- If the name used in a condition of this kind is a duplicate, the test
- is applied to all subpatterns of the same name, and is true if any one
+ If the name used in a condition of this kind is a duplicate, the test
+ is applied to all subpatterns of the same name, and is true if any one
of them has matched.
Checking for pattern recursion
- "Recursion" in this sense refers to any subroutine-like call from one
- part of the pattern to another, whether or not it is actually recur-
- sive. See the sections entitled "Recursive patterns" and "Subpatterns
+ "Recursion" in this sense refers to any subroutine-like call from one
+ part of the pattern to another, whether or not it is actually recur-
+ sive. See the sections entitled "Recursive patterns" and "Subpatterns
as subroutines" below for details of recursion and subpattern calls.
- If a condition is the string (R), and there is no subpattern with the
- name R, the condition is true if matching is currently in a recursion
- or subroutine call to the whole pattern or any subpattern. If digits
- follow the letter R, and there is no subpattern with that name, the
+ If a condition is the string (R), and there is no subpattern with the
+ name R, the condition is true if matching is currently in a recursion
+ or subroutine call to the whole pattern or any subpattern. If digits
+ follow the letter R, and there is no subpattern with that name, the
condition is true if the most recent call is into a subpattern with the
- given number, which must exist somewhere in the overall pattern. This
+ given number, which must exist somewhere in the overall pattern. This
is a contrived example that is equivalent to a+b:
((?(R1)a+|(?1)b))
- However, in both cases, if there is a subpattern with a matching name,
- the condition tests for its being set, as described in the section
- above, instead of testing for recursion. For example, creating a group
- with the name R1 by adding (?<R1>) to the above pattern completely
+ However, in both cases, if there is a subpattern with a matching name,
+ the condition tests for its being set, as described in the section
+ above, instead of testing for recursion. For example, creating a group
+ with the name R1 by adding (?<R1>) to the above pattern completely
changes its meaning.
If a name preceded by ampersand follows the letter R, for example:
@@ -8286,7 +8312,7 @@
of that name (which must exist within the pattern).
This condition does not check the entire recursion stack. It tests only
- the current level. If the name used in a condition of this kind is a
+ the current level. If the name used in a condition of this kind is a
duplicate, the test is applied to all subpatterns of the same name, and
is true if any one of them is the most recent recursion.
@@ -8295,10 +8321,10 @@
Defining subpatterns for use by reference only
If the condition is the string (DEFINE), the condition is always false,
- even if there is a group with the name DEFINE. In this case, there may
+ even if there is a group with the name DEFINE. In this case, there may
be only one alternative in the subpattern. It is always skipped if con-
- trol reaches this point in the pattern; the idea of DEFINE is that it
- can be used to define subroutines that can be referenced from else-
+ trol reaches this point in the pattern; the idea of DEFINE is that it
+ can be used to define subroutines that can be referenced from else-
where. (The use of subroutines is described below.) For example, a pat-
tern to match an IPv4 address such as "192.168.23.245" could be written
like this (ignore white space and line breaks):
@@ -8306,53 +8332,53 @@
(?(DEFINE) (?<byte> 2[0-4]\d | 25[0-5] | 1\d\d | [1-9]?\d) )
\b (?&byte) (\.(?&byte)){3} \b
- The first part of the pattern is a DEFINE group inside which a another
- group named "byte" is defined. This matches an individual component of
- an IPv4 address (a number less than 256). When matching takes place,
- this part of the pattern is skipped because DEFINE acts like a false
- condition. The rest of the pattern uses references to the named group
- to match the four dot-separated components of an IPv4 address, insist-
+ The first part of the pattern is a DEFINE group inside which a another
+ group named "byte" is defined. This matches an individual component of
+ an IPv4 address (a number less than 256). When matching takes place,
+ this part of the pattern is skipped because DEFINE acts like a false
+ condition. The rest of the pattern uses references to the named group
+ to match the four dot-separated components of an IPv4 address, insist-
ing on a word boundary at each end.
Checking the PCRE2 version
- Programs that link with a PCRE2 library can check the version by call-
- ing pcre2_config() with appropriate arguments. Users of applications
- that do not have access to the underlying code cannot do this. A spe-
- cial "condition" called VERSION exists to allow such users to discover
+ Programs that link with a PCRE2 library can check the version by call-
+ ing pcre2_config() with appropriate arguments. Users of applications
+ that do not have access to the underlying code cannot do this. A spe-
+ cial "condition" called VERSION exists to allow such users to discover
which version of PCRE2 they are dealing with by using this condition to
- match a string such as "yesno". VERSION must be followed either by "="
+ match a string such as "yesno". VERSION must be followed either by "="
or ">=" and a version number. For example:
(?(VERSION>=10.4)yes|no)
- This pattern matches "yes" if the PCRE2 version is greater or equal to
- 10.4, or "no" otherwise. The fractional part of the version number may
+ This pattern matches "yes" if the PCRE2 version is greater or equal to
+ 10.4, or "no" otherwise. The fractional part of the version number may
not contain more than two digits.
Assertion conditions
- If the condition is not in any of the above formats, it must be an
- assertion. This may be a positive or negative lookahead or lookbehind
- assertion. Consider this pattern, again containing non-significant
+ If the condition is not in any of the above formats, it must be an
+ assertion. This may be a positive or negative lookahead or lookbehind
+ assertion. Consider this pattern, again containing non-significant
white space, and with the two alternatives on the second line:
(?(?=[^a-z]*[a-z])
\d{2}-[a-z]{3}-\d{2} | \d{2}-\d{2}-\d{2} )
- The condition is a positive lookahead assertion that matches an
- optional sequence of non-letters followed by a letter. In other words,
- it tests for the presence of at least one letter in the subject. If a
- letter is found, the subject is matched against the first alternative;
- otherwise it is matched against the second. This pattern matches
- strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are
+ The condition is a positive lookahead assertion that matches an
+ optional sequence of non-letters followed by a letter. In other words,
+ it tests for the presence of at least one letter in the subject. If a
+ letter is found, the subject is matched against the first alternative;
+ otherwise it is matched against the second. This pattern matches
+ strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are
letters and dd are digits.
- When an assertion that is a condition contains capturing subpatterns,
- any capturing that occurs in a matching branch is retained afterwards,
+ When an assertion that is a condition contains capturing subpatterns,
+ any capturing that occurs in a matching branch is retained afterwards,
for both positive and negative assertions, because matching always con-
tinues after the assertion, whether it succeeds or fails. (Compare non-
- conditional assertions, when captures are retained only for positive
+ conditional assertions, when captures are retained only for positive
assertions that succeed.)
@@ -8359,44 +8385,44 @@
COMMENTS
There are two ways of including comments in patterns that are processed
- by PCRE2. In both cases, the start of the comment must not be in a
- character class, nor in the middle of any other sequence of related
- characters such as (?: or a subpattern name or number. The characters
+ by PCRE2. In both cases, the start of the comment must not be in a
+ character class, nor in the middle of any other sequence of related
+ characters such as (?: or a subpattern name or number. The characters
that make up a comment play no part in the pattern matching.
- The sequence (?# marks the start of a comment that continues up to the
- next closing parenthesis. Nested parentheses are not permitted. If the
- PCRE2_EXTENDED or PCRE2_EXTENDED_MORE option is set, an unescaped #
- character also introduces a comment, which in this case continues to
- immediately after the next newline character or character sequence in
+ The sequence (?# marks the start of a comment that continues up to the
+ next closing parenthesis. Nested parentheses are not permitted. If the
+ PCRE2_EXTENDED or PCRE2_EXTENDED_MORE option is set, an unescaped #
+ character also introduces a comment, which in this case continues to
+ immediately after the next newline character or character sequence in
the pattern. Which characters are interpreted as newlines is controlled
- by an option passed to the compiling function or by a special sequence
+ by an option passed to the compiling function or by a special sequence
at the start of the pattern, as described in the section entitled "New-
line conventions" above. Note that the end of this type of comment is a
- literal newline sequence in the pattern; escape sequences that happen
+ literal newline sequence in the pattern; escape sequences that happen
to represent a newline do not count. For example, consider this pattern
- when PCRE2_EXTENDED is set, and the default newline convention (a sin-
+ when PCRE2_EXTENDED is set, and the default newline convention (a sin-
gle linefeed character) is in force:
abc #comment \n still comment
- On encountering the # character, pcre2_compile() skips along, looking
- for a newline in the pattern. The sequence \n is still literal at this
- stage, so it does not terminate the comment. Only an actual character
+ On encountering the # character, pcre2_compile() skips along, looking
+ for a newline in the pattern. The sequence \n is still literal at this
+ stage, so it does not terminate the comment. Only an actual character
with the code value 0x0a (the default newline) does so.
RECURSIVE PATTERNS
- Consider the problem of matching a string in parentheses, allowing for
- unlimited nested parentheses. Without the use of recursion, the best
- that can be done is to use a pattern that matches up to some fixed
- depth of nesting. It is not possible to handle an arbitrary nesting
+ Consider the problem of matching a string in parentheses, allowing for
+ unlimited nested parentheses. Without the use of recursion, the best
+ that can be done is to use a pattern that matches up to some fixed
+ depth of nesting. It is not possible to handle an arbitrary nesting
depth.
For some time, Perl has provided a facility that allows regular expres-
- sions to recurse (amongst other things). It does this by interpolating
- Perl code in the expression at run time, and the code can refer to the
+ sions to recurse (amongst other things). It does this by interpolating
+ Perl code in the expression at run time, and the code can refer to the
expression itself. A Perl pattern using code interpolation to solve the
parentheses problem can be created like this:
@@ -8406,109 +8432,109 @@
refers recursively to the pattern in which it appears.
Obviously, PCRE2 cannot support the interpolation of Perl code.
- Instead, it supports special syntax for recursion of the entire pat-
+ Instead, it supports special syntax for recursion of the entire pat-
tern, and also for individual subpattern recursion. After its introduc-
- tion in PCRE1 and Python, this kind of recursion was subsequently
+ tion in PCRE1 and Python, this kind of recursion was subsequently
introduced into Perl at release 5.10.
- A special item that consists of (? followed by a number greater than
- zero and a closing parenthesis is a recursive subroutine call of the
- subpattern of the given number, provided that it occurs inside that
- subpattern. (If not, it is a non-recursive subroutine call, which is
- described in the next section.) The special item (?R) or (?0) is a
+ A special item that consists of (? followed by a number greater than
+ zero and a closing parenthesis is a recursive subroutine call of the
+ subpattern of the given number, provided that it occurs inside that
+ subpattern. (If not, it is a non-recursive subroutine call, which is
+ described in the next section.) The special item (?R) or (?0) is a
recursive call of the entire regular expression.
- This PCRE2 pattern solves the nested parentheses problem (assume the
+ This PCRE2 pattern solves the nested parentheses problem (assume the
PCRE2_EXTENDED option is set so that white space is ignored):
\( ( [^()]++ | (?R) )* \)
- First it matches an opening parenthesis. Then it matches any number of
- substrings which can either be a sequence of non-parentheses, or a
- recursive match of the pattern itself (that is, a correctly parenthe-
+ First it matches an opening parenthesis. Then it matches any number of
+ substrings which can either be a sequence of non-parentheses, or a
+ recursive match of the pattern itself (that is, a correctly parenthe-
sized substring). Finally there is a closing parenthesis. Note the use
of a possessive quantifier to avoid backtracking into sequences of non-
parentheses.
- If this were part of a larger pattern, you would not want to recurse
+ If this were part of a larger pattern, you would not want to recurse
the entire pattern, so instead you could use this:
( \( ( [^()]++ | (?1) )* \) )
- We have put the pattern into parentheses, and caused the recursion to
+ We have put the pattern into parentheses, and caused the recursion to
refer to them instead of the whole pattern.
- In a larger pattern, keeping track of parenthesis numbers can be
- tricky. This is made easier by the use of relative references. Instead
+ In a larger pattern, keeping track of parenthesis numbers can be
+ tricky. This is made easier by the use of relative references. Instead
of (?1) in the pattern above you can write (?-2) to refer to the second
- most recently opened parentheses preceding the recursion. In other
- words, a negative number counts capturing parentheses leftwards from
+ most recently opened parentheses preceding the recursion. In other
+ words, a negative number counts capturing parentheses leftwards from
the point at which it is encountered.
Be aware however, that if duplicate subpattern numbers are in use, rel-
- ative references refer to the earliest subpattern with the appropriate
+ ative references refer to the earliest subpattern with the appropriate
number. Consider, for example:
(?|(a)|(b)) (c) (?-2)
- The first two capturing groups (a) and (b) are both numbered 1, and
- group (c) is number 2. When the reference (?-2) is encountered, the
+ The first two capturing groups (a) and (b) are both numbered 1, and
+ group (c) is number 2. When the reference (?-2) is encountered, the
second most recently opened parentheses has the number 1, but it is the
- first such group (the (a) group) to which the recursion refers. This
- would be the same if an absolute reference (?1) was used. In other
- words, relative references are just a shorthand for computing a group
+ first such group (the (a) group) to which the recursion refers. This
+ would be the same if an absolute reference (?1) was used. In other
+ words, relative references are just a shorthand for computing a group
number.
- It is also possible to refer to subsequently opened parentheses, by
- writing references such as (?+2). However, these cannot be recursive
- because the reference is not inside the parentheses that are refer-
- enced. They are always non-recursive subroutine calls, as described in
+ It is also possible to refer to subsequently opened parentheses, by
+ writing references such as (?+2). However, these cannot be recursive
+ because the reference is not inside the parentheses that are refer-
+ enced. They are always non-recursive subroutine calls, as described in
the next section.
- An alternative approach is to use named parentheses. The Perl syntax
- for this is (?&name); PCRE1's earlier syntax (?P>name) is also sup-
+ An alternative approach is to use named parentheses. The Perl syntax
+ for this is (?&name); PCRE1's earlier syntax (?P>name) is also sup-
ported. We could rewrite the above example as follows:
(?<pn> \( ( [^()]++ | (?&pn) )* \) )
- If there is more than one subpattern with the same name, the earliest
+ If there is more than one subpattern with the same name, the earliest
one is used.
The example pattern that we have been looking at contains nested unlim-
- ited repeats, and so the use of a possessive quantifier for matching
- strings of non-parentheses is important when applying the pattern to
+ ited repeats, and so the use of a possessive quantifier for matching
+ strings of non-parentheses is important when applying the pattern to
strings that do not match. For example, when this pattern is applied to
(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()
- it yields "no match" quickly. However, if a possessive quantifier is
- not used, the match runs for a very long time indeed because there are
- so many different ways the + and * repeats can carve up the subject,
+ it yields "no match" quickly. However, if a possessive quantifier is
+ not used, the match runs for a very long time indeed because there are
+ so many different ways the + and * repeats can carve up the subject,
and all have to be tested before failure can be reported.
- At the end of a match, the values of capturing parentheses are those
- from the outermost level. If you want to obtain intermediate values, a
+ At the end of a match, the values of capturing parentheses are those
+ from the outermost level. If you want to obtain intermediate values, a
callout function can be used (see below and the pcre2callout documenta-
tion). If the pattern above is matched against
(ab(cd)ef)
- the value for the inner capturing parentheses (numbered 2) is "ef",
- which is the last value taken on at the top level. If a capturing sub-
- pattern is not matched at the top level, its final captured value is
- unset, even if it was (temporarily) set at a deeper level during the
+ the value for the inner capturing parentheses (numbered 2) is "ef",
+ which is the last value taken on at the top level. If a capturing sub-
+ pattern is not matched at the top level, its final captured value is
+ unset, even if it was (temporarily) set at a deeper level during the
matching process.
- Do not confuse the (?R) item with the condition (R), which tests for
- recursion. Consider this pattern, which matches text in angle brack-
- ets, allowing for arbitrary nesting. Only digits are allowed in nested
- brackets (that is, when recursing), whereas any characters are permit-
+ Do not confuse the (?R) item with the condition (R), which tests for
+ recursion. Consider this pattern, which matches text in angle brack-
+ ets, allowing for arbitrary nesting. Only digits are allowed in nested
+ brackets (that is, when recursing), whereas any characters are permit-
ted at the outer level.
< (?: (?(R) \d++ | [^<>]*+) | (?R)) * >
- In this pattern, (?(R) is the start of a conditional subpattern, with
- two different alternatives for the recursive and non-recursive cases.
+ In this pattern, (?(R) is the start of a conditional subpattern, with
+ two different alternatives for the recursive and non-recursive cases.
The (?R) item is the actual recursive call.
Differences in recursion processing between PCRE2 and Perl
@@ -8515,66 +8541,66 @@
Some former differences between PCRE2 and Perl no longer exist.
- Before release 10.30, recursion processing in PCRE2 differed from Perl
- in that a recursive subpattern call was always treated as an atomic
- group. That is, once it had matched some of the subject string, it was
- never re-entered, even if it contained untried alternatives and there
- was a subsequent matching failure. (Historical note: PCRE implemented
+ Before release 10.30, recursion processing in PCRE2 differed from Perl
+ in that a recursive subpattern call was always treated as an atomic
+ group. That is, once it had matched some of the subject string, it was
+ never re-entered, even if it contained untried alternatives and there
+ was a subsequent matching failure. (Historical note: PCRE implemented
recursion before Perl did.)
- Starting with release 10.30, recursive subroutine calls are no longer
+ Starting with release 10.30, recursive subroutine calls are no longer
treated as atomic. That is, they can be re-entered to try unused alter-
- natives if there is a matching failure later in the pattern. This is
- now compatible with the way Perl works. If you want a subroutine call
+ natives if there is a matching failure later in the pattern. This is
+ now compatible with the way Perl works. If you want a subroutine call
to be atomic, you must explicitly enclose it in an atomic group.
- Supporting backtracking into recursions simplifies certain types of
+ Supporting backtracking into recursions simplifies certain types of
recursive pattern. For example, this pattern matches palindromic
strings:
^((.)(?1)\2|.?)$
- The second branch in the group matches a single central character in
- the palindrome when there are an odd number of characters, or nothing
- when there are an even number of characters, but in order to work it
- has to be able to try the second case when the rest of the pattern
+ The second branch in the group matches a single central character in
+ the palindrome when there are an odd number of characters, or nothing
+ when there are an even number of characters, but in order to work it
+ has to be able to try the second case when the rest of the pattern
match fails. If you want to match typical palindromic phrases, the pat-
- tern has to ignore all non-word characters, which can be done like
+ tern has to ignore all non-word characters, which can be done like
this:
^\W*+((.)\W*+(?1)\W*+\2|\W*+.?)\W*+$
- If run with the PCRE2_CASELESS option, this pattern matches phrases
- such as "A man, a plan, a canal: Panama!". Note the use of the posses-
- sive quantifier *+ to avoid backtracking into sequences of non-word
+ If run with the PCRE2_CASELESS option, this pattern matches phrases
+ such as "A man, a plan, a canal: Panama!". Note the use of the posses-
+ sive quantifier *+ to avoid backtracking into sequences of non-word
characters. Without this, PCRE2 takes a great deal longer (ten times or
- more) to match typical phrases, and Perl takes so long that you think
+ more) to match typical phrases, and Perl takes so long that you think
it has gone into a loop.
- Another way in which PCRE2 and Perl used to differ in their recursion
- processing is in the handling of captured values. Formerly in Perl,
- when a subpattern was called recursively or as a subpattern (see the
- next section), it had no access to any values that were captured out-
- side the recursion, whereas in PCRE2 these values can be referenced.
+ Another way in which PCRE2 and Perl used to differ in their recursion
+ processing is in the handling of captured values. Formerly in Perl,
+ when a subpattern was called recursively or as a subpattern (see the
+ next section), it had no access to any values that were captured out-
+ side the recursion, whereas in PCRE2 these values can be referenced.
Consider this pattern:
^(.)(\1|a(?2))
- This pattern matches "bab". The first capturing parentheses match "b",
+ This pattern matches "bab". The first capturing parentheses match "b",
then in the second group, when the backreference \1 fails to match "b",
the second alternative matches "a" and then recurses. In the recursion,
- \1 does now match "b" and so the whole match succeeds. This match used
+ \1 does now match "b" and so the whole match succeeds. This match used
to fail in Perl, but in later versions (I tried 5.024) it now works.
SUBPATTERNS AS SUBROUTINES
- If the syntax for a recursive subpattern call (either by number or by
+ If the syntax for a recursive subpattern call (either by number or by
name) is used outside the parentheses to which it refers, it operates a
bit like a subroutine in a programming language. More accurately, PCRE2
- treats the referenced subpattern as an independent subpattern which it
- tries to match at the current matching position. The called subpattern
- may be defined before or after the reference. A numbered reference can
+ treats the referenced subpattern as an independent subpattern which it
+ tries to match at the current matching position. The called subpattern
+ may be defined before or after the reference. A numbered reference can
be absolute or relative, as in these examples:
(...(absolute)...)...(?2)...
@@ -8585,30 +8611,30 @@
(sens|respons)e and \1ibility
- matches "sense and sensibility" and "response and responsibility", but
+ matches "sense and sensibility" and "response and responsibility", but
not "sense and responsibility". If instead the pattern
(sens|respons)e and (?1)ibility
- is used, it does match "sense and responsibility" as well as the other
- two strings. Another example is given in the discussion of DEFINE
+ is used, it does match "sense and responsibility" as well as the other
+ two strings. Another example is given in the discussion of DEFINE
above.
- Like recursions, subroutine calls used to be treated as atomic, but
- this changed at PCRE2 release 10.30, so backtracking into subroutine
- calls can now occur. However, any capturing parentheses that are set
+ Like recursions, subroutine calls used to be treated as atomic, but
+ this changed at PCRE2 release 10.30, so backtracking into subroutine
+ calls can now occur. However, any capturing parentheses that are set
during the subroutine call revert to their previous values afterwards.
- Processing options such as case-independence are fixed when a subpat-
- tern is defined, so if it is used as a subroutine, such options cannot
+ Processing options such as case-independence are fixed when a subpat-
+ tern is defined, so if it is used as a subroutine, such options cannot
be changed for different calls. For example, consider this pattern:
(abc)(?i:(?-1))
- It matches "abcabc". It does not match "abcABC" because the change of
+ It matches "abcabc". It does not match "abcABC" because the change of
processing option does not affect the called subpattern.
- The behaviour of backtracking control verbs in subpatterns when called
+ The behaviour of backtracking control verbs in subpatterns when called
as subroutines is described in the section entitled "Backtracking verbs
in subroutines" below.
@@ -8615,22 +8641,22 @@
ONIGURUMA SUBROUTINE SYNTAX
- For compatibility with Oniguruma, the non-Perl syntax \g followed by a
+ For compatibility with Oniguruma, the non-Perl syntax \g followed by a
name or a number enclosed either in angle brackets or single quotes, is
- an alternative syntax for referencing a subpattern as a subroutine,
- possibly recursively. Here are two of the examples used above, rewrit-
+ an alternative syntax for referencing a subpattern as a subroutine,
+ possibly recursively. Here are two of the examples used above, rewrit-
ten using this syntax:
(?<pn> \( ( (?>[^()]+) | \g<pn> )* \) )
(sens|respons)e and \g'1'ibility
- PCRE2 supports an extension to Oniguruma: if a number is preceded by a
+ PCRE2 supports an extension to Oniguruma: if a number is preceded by a
plus or a minus sign it is taken as a relative reference. For example:
(abc)(?i:\g<-1>)
- Note that \g{...} (Perl syntax) and \g<...> (Oniguruma syntax) are not
- synonymous. The former is a backreference; the latter is a subroutine
+ Note that \g{...} (Perl syntax) and \g<...> (Oniguruma syntax) are not
+ synonymous. The former is a backreference; the latter is a subroutine
call.
@@ -8637,54 +8663,54 @@
CALLOUTS
Perl has a feature whereby using the sequence (?{...}) causes arbitrary
- Perl code to be obeyed in the middle of matching a regular expression.
+ Perl code to be obeyed in the middle of matching a regular expression.
This makes it possible, amongst other things, to extract different sub-
strings that match the same pair of parentheses when there is a repeti-
tion.
- PCRE2 provides a similar feature, but of course it cannot obey arbi-
- trary Perl code. The feature is called "callout". The caller of PCRE2
- provides an external function by putting its entry point in a match
- context using the function pcre2_set_callout(), and then passing that
- context to pcre2_match() or pcre2_dfa_match(). If no match context is
+ PCRE2 provides a similar feature, but of course it cannot obey arbi-
+ trary Perl code. The feature is called "callout". The caller of PCRE2
+ provides an external function by putting its entry point in a match
+ context using the function pcre2_set_callout(), and then passing that
+ context to pcre2_match() or pcre2_dfa_match(). If no match context is
passed, or if the callout entry point is set to NULL, callouts are dis-
abled.
- Within a regular expression, (?C<arg>) indicates a point at which the
- external function is to be called. There are two kinds of callout:
- those with a numerical argument and those with a string argument. (?C)
- on its own with no argument is treated as (?C0). A numerical argument
- allows the application to distinguish between different callouts.
- String arguments were added for release 10.20 to make it possible for
- script languages that use PCRE2 to embed short scripts within patterns
+ Within a regular expression, (?C<arg>) indicates a point at which the
+ external function is to be called. There are two kinds of callout:
+ those with a numerical argument and those with a string argument. (?C)
+ on its own with no argument is treated as (?C0). A numerical argument
+ allows the application to distinguish between different callouts.
+ String arguments were added for release 10.20 to make it possible for
+ script languages that use PCRE2 to embed short scripts within patterns
in a similar way to Perl.
During matching, when PCRE2 reaches a callout point, the external func-
- tion is called. It is provided with the number or string argument of
- the callout, the position in the pattern, and one item of data that is
+ tion is called. It is provided with the number or string argument of
+ the callout, the position in the pattern, and one item of data that is
also set in the match block. The callout function may cause matching to
proceed, to backtrack, or to fail.
- By default, PCRE2 implements a number of optimizations at matching
- time, and one side-effect is that sometimes callouts are skipped. If
- you need all possible callouts to happen, you need to set options that
- disable the relevant optimizations. More details, including a complete
- description of the programming interface to the callout function, are
+ By default, PCRE2 implements a number of optimizations at matching
+ time, and one side-effect is that sometimes callouts are skipped. If
+ you need all possible callouts to happen, you need to set options that
+ disable the relevant optimizations. More details, including a complete
+ description of the programming interface to the callout function, are
given in the pcre2callout documentation.
Callouts with numerical arguments
- If you just want to have a means of identifying different callout
- points, put a number less than 256 after the letter C. For example,
+ If you just want to have a means of identifying different callout
+ points, put a number less than 256 after the letter C. For example,
this pattern has two callout points:
(?C1)abc(?C2)def
- If the PCRE2_AUTO_CALLOUT flag is passed to pcre2_compile(), numerical
- callouts are automatically installed before each item in the pattern.
- They are all numbered 255. If there is a conditional group in the pat-
+ If the PCRE2_AUTO_CALLOUT flag is passed to pcre2_compile(), numerical
+ callouts are automatically installed before each item in the pattern.
+ They are all numbered 255. If there is a conditional group in the pat-
tern whose condition is an assertion, an additional callout is inserted
- just before the condition. An explicit callout may also be set at this
+ just before the condition. An explicit callout may also be set at this
position, as in this example:
(?(?C9)(?=a)abc|def)
@@ -8694,60 +8720,60 @@
Callouts with string arguments
- A delimited string may be used instead of a number as a callout argu-
- ment. The starting delimiter must be one of ` ' " ^ % # $ { and the
+ A delimited string may be used instead of a number as a callout argu-
+ ment. The starting delimiter must be one of ` ' " ^ % # $ { and the
ending delimiter is the same as the start, except for {, where the end-
- ing delimiter is }. If the ending delimiter is needed within the
+ ing delimiter is }. If the ending delimiter is needed within the
string, it must be doubled. For example:
(?C'ab ''c'' d')xyz(?C{any text})pqr
- The doubling is removed before the string is passed to the callout
+ The doubling is removed before the string is passed to the callout
function.
BACKTRACKING CONTROL
- There are a number of special "Backtracking Control Verbs" (to use
- Perl's terminology) that modify the behaviour of backtracking during
- matching. They are generally of the form (*VERB) or (*VERB:NAME). Some
- verbs take either form, possibly behaving differently depending on
+ There are a number of special "Backtracking Control Verbs" (to use
+ Perl's terminology) that modify the behaviour of backtracking during
+ matching. They are generally of the form (*VERB) or (*VERB:NAME). Some
+ verbs take either form, possibly behaving differently depending on
whether or not a name is present.
- By default, for compatibility with Perl, a name is any sequence of
+ By default, for compatibility with Perl, a name is any sequence of
characters that does not include a closing parenthesis. The name is not
- processed in any way, and it is not possible to include a closing
- parenthesis in the name. This can be changed by setting the
- PCRE2_ALT_VERBNAMES option, but the result is no longer Perl-compati-
+ processed in any way, and it is not possible to include a closing
+ parenthesis in the name. This can be changed by setting the
+ PCRE2_ALT_VERBNAMES option, but the result is no longer Perl-compati-
ble.
- When PCRE2_ALT_VERBNAMES is set, backslash processing is applied to
- verb names and only an unescaped closing parenthesis terminates the
- name. However, the only backslash items that are permitted are \Q, \E,
- and sequences such as \x{100} that define character code points. Char-
+ When PCRE2_ALT_VERBNAMES is set, backslash processing is applied to
+ verb names and only an unescaped closing parenthesis terminates the
+ name. However, the only backslash items that are permitted are \Q, \E,
+ and sequences such as \x{100} that define character code points. Char-
acter type escapes such as \d are faulted.
A closing parenthesis can be included in a name either as \) or between
- \Q and \E. In addition to backslash processing, if the PCRE2_EXTENDED
+ \Q and \E. In addition to backslash processing, if the PCRE2_EXTENDED
or PCRE2_EXTENDED_MORE option is also set, unescaped whitespace in verb
names is skipped, and #-comments are recognized, exactly as in the rest
- of the pattern. PCRE2_EXTENDED and PCRE2_EXTENDED_MORE do not affect
+ of the pattern. PCRE2_EXTENDED and PCRE2_EXTENDED_MORE do not affect
verb names unless PCRE2_ALT_VERBNAMES is also set.
- The maximum length of a name is 255 in the 8-bit library and 65535 in
- the 16-bit and 32-bit libraries. If the name is empty, that is, if the
- closing parenthesis immediately follows the colon, the effect is as if
+ The maximum length of a name is 255 in the 8-bit library and 65535 in
+ the 16-bit and 32-bit libraries. If the name is empty, that is, if the
+ closing parenthesis immediately follows the colon, the effect is as if
the colon were not there. Any number of these verbs may occur in a pat-
tern.
- Since these verbs are specifically related to backtracking, most of
- them can be used only when the pattern is to be matched using the tra-
+ Since these verbs are specifically related to backtracking, most of
+ them can be used only when the pattern is to be matched using the tra-
ditional matching function, because that uses a backtracking algorithm.
- With the exception of (*FAIL), which behaves like a failing negative
+ With the exception of (*FAIL), which behaves like a failing negative
assertion, the backtracking control verbs cause an error if encountered
by the DFA matching function.
- The behaviour of these verbs in repeated groups, assertions, and in
+ The behaviour of these verbs in repeated groups, assertions, and in
subpatterns called as subroutines (whether or not recursively) is docu-
mented below.
@@ -8755,16 +8781,16 @@
PCRE2 contains some optimizations that are used to speed up matching by
running some checks at the start of each match attempt. For example, it
- may know the minimum length of matching subject, or that a particular
+ may know the minimum length of matching subject, or that a particular
character must be present. When one of these optimizations bypasses the
- running of a match, any included backtracking verbs will not, of
+ running of a match, any included backtracking verbs will not, of
course, be processed. You can suppress the start-of-match optimizations
- by setting the PCRE2_NO_START_OPTIMIZE option when calling pcre2_com-
- pile(), or by starting the pattern with (*NO_START_OPT). There is more
+ by setting the PCRE2_NO_START_OPTIMIZE option when calling pcre2_com-
+ pile(), or by starting the pattern with (*NO_START_OPT). There is more
discussion of this option in the section entitled "Compiling a pattern"
in the pcre2api documentation.
- Experiments with Perl suggest that it too has similar optimizations,
+ Experiments with Perl suggest that it too has similar optimizations,
and like PCRE2, turning them off can change the result of a match.
Verbs that act immediately
@@ -8773,63 +8799,63 @@
(*ACCEPT) or (*ACCEPT:NAME)
- This verb causes the match to end successfully, skipping the remainder
- of the pattern. However, when it is inside a subpattern that is called
- as a subroutine, only that subpattern is ended successfully. Matching
+ This verb causes the match to end successfully, skipping the remainder
+ of the pattern. However, when it is inside a subpattern that is called
+ as a subroutine, only that subpattern is ended successfully. Matching
then continues at the outer level. If (*ACCEPT) in triggered in a posi-
- tive assertion, the assertion succeeds; in a negative assertion, the
+ tive assertion, the assertion succeeds; in a negative assertion, the
assertion fails.
- If (*ACCEPT) is inside capturing parentheses, the data so far is cap-
+ If (*ACCEPT) is inside capturing parentheses, the data so far is cap-
tured. For example:
A((?:A|B(*ACCEPT)|C)D)
- This matches "AB", "AAD", or "ACD"; when it matches "AB", "B" is cap-
+ This matches "AB", "AAD", or "ACD"; when it matches "AB", "B" is cap-
tured by the outer parentheses.
(*FAIL) or (*FAIL:NAME)
- This verb causes a matching failure, forcing backtracking to occur. It
- may be abbreviated to (*F). It is equivalent to (?!) but easier to
+ This verb causes a matching failure, forcing backtracking to occur. It
+ may be abbreviated to (*F). It is equivalent to (?!) but easier to
read. The Perl documentation notes that it is probably useful only when
combined with (?{}) or (??{}). Those are, of course, Perl features that
- are not present in PCRE2. The nearest equivalent is the callout fea-
+ are not present in PCRE2. The nearest equivalent is the callout fea-
ture, as for example in this pattern:
a+(?C)(*FAIL)
- A match with the string "aaaa" always fails, but the callout is taken
+ A match with the string "aaaa" always fails, but the callout is taken
before each backtrack happens (in this example, 10 times).
- (*ACCEPT:NAME) and (*FAIL:NAME) behave exactly the same as
+ (*ACCEPT:NAME) and (*FAIL:NAME) behave exactly the same as
(*MARK:NAME)(*ACCEPT) and (*MARK:NAME)(*FAIL), respectively.
Recording which path was taken
- There is one verb whose main purpose is to track how a match was
- arrived at, though it also has a secondary use in conjunction with
+ There is one verb whose main purpose is to track how a match was
+ arrived at, though it also has a secondary use in conjunction with
advancing the match starting point (see (*SKIP) below).
(*MARK:NAME) or (*:NAME)
- A name is always required with this verb. There may be as many
- instances of (*MARK) as you like in a pattern, and their names do not
+ A name is always required with this verb. There may be as many
+ instances of (*MARK) as you like in a pattern, and their names do not
have to be unique.
When a match succeeds, the name of the last-encountered (*MARK:NAME) on
the matching path is passed back to the caller as described in the sec-
tion entitled "Other information about the match" in the pcre2api docu-
- mentation. This applies to all instances of (*MARK), including those
- inside assertions and atomic groups. (There are differences in those
- cases when (*MARK) is used in conjunction with (*SKIP) as described
+ mentation. This applies to all instances of (*MARK), including those
+ inside assertions and atomic groups. (There are differences in those
+ cases when (*MARK) is used in conjunction with (*SKIP) as described
below.)
- As well as (*MARK), the (*COMMIT), (*PRUNE) and (*THEN) verbs may have
- associated NAME arguments. Whichever is last on the matching path is
+ As well as (*MARK), the (*COMMIT), (*PRUNE) and (*THEN) verbs may have
+ associated NAME arguments. Whichever is last on the matching path is
passed back. See below for more details of these other verbs.
- Here is an example of pcre2test output, where the "mark" modifier
+ Here is an example of pcre2test output, where the "mark" modifier
requests the retrieval and outputting of (*MARK) data:
re> /X(*MARK:A)Y|X(*MARK:B)Z/mark
@@ -8841,16 +8867,16 @@
MK: B
The (*MARK) name is tagged with "MK:" in this output, and in this exam-
- ple it indicates which of the two alternatives matched. This is a more
- efficient way of obtaining this information than putting each alterna-
+ ple it indicates which of the two alternatives matched. This is a more
+ efficient way of obtaining this information than putting each alterna-
tive in its own capturing parentheses.
- If a verb with a name is encountered in a positive assertion that is
- true, the name is recorded and passed back if it is the last-encoun-
+ If a verb with a name is encountered in a positive assertion that is
+ true, the name is recorded and passed back if it is the last-encoun-
tered. This does not happen for negative assertions or failing positive
assertions.
- After a partial match or a failed match, the last encountered name in
+ After a partial match or a failed match, the last encountered name in
the entire match process is returned. For example:
re> /X(*MARK:A)Y|X(*MARK:B)Z/mark
@@ -8857,38 +8883,38 @@
data> XP
No match, mark = B
- Note that in this unanchored example the mark is retained from the
+ Note that in this unanchored example the mark is retained from the
match attempt that started at the letter "X" in the subject. Subsequent
match attempts starting at "P" and then with an empty string do not get
as far as the (*MARK) item, but nevertheless do not reset it.
- If you are interested in (*MARK) values after failed matches, you
- should probably set the PCRE2_NO_START_OPTIMIZE option (see above) to
+ If you are interested in (*MARK) values after failed matches, you
+ should probably set the PCRE2_NO_START_OPTIMIZE option (see above) to
ensure that the match is always attempted.
Verbs that act after backtracking
The following verbs do nothing when they are encountered. Matching con-
- tinues with what follows, but if there is a subsequent match failure,
- causing a backtrack to the verb, a failure is forced. That is, back-
- tracking cannot pass to the left of the verb. However, when one of
+ tinues with what follows, but if there is a subsequent match failure,
+ causing a backtrack to the verb, a failure is forced. That is, back-
+ tracking cannot pass to the left of the verb. However, when one of
these verbs appears inside an atomic group or in a lookaround assertion
- that is true, its effect is confined to that group, because once the
- group has been matched, there is never any backtracking into it. Back-
+ that is true, its effect is confined to that group, because once the
+ group has been matched, there is never any backtracking into it. Back-
tracking from beyond an assertion or an atomic group ignores the entire
group, and seeks a preceeding backtracking point.
- These verbs differ in exactly what kind of failure occurs when back-
- tracking reaches them. The behaviour described below is what happens
- when the verb is not in a subroutine or an assertion. Subsequent sec-
+ These verbs differ in exactly what kind of failure occurs when back-
+ tracking reaches them. The behaviour described below is what happens
+ when the verb is not in a subroutine or an assertion. Subsequent sec-
tions cover these special cases.
(*COMMIT) or (*COMMIT:NAME)
- This verb causes the whole match to fail outright if there is a later
+ This verb causes the whole match to fail outright if there is a later
matching failure that causes backtracking to reach it. Even if the pat-
- tern is unanchored, no further attempts to find a match by advancing
- the starting point take place. If (*COMMIT) is the only backtracking
+ tern is unanchored, no further attempts to find a match by advancing
+ the starting point take place. If (*COMMIT) is the only backtracking
verb that is encountered, once it has been passed pcre2_match() is com-
mitted to finding a match at the current starting point, or not at all.
For example:
@@ -8895,22 +8921,22 @@
a+(*COMMIT)b
- This matches "xxaab" but not "aacaab". It can be thought of as a kind
+ This matches "xxaab" but not "aacaab". It can be thought of as a kind
of dynamic anchor, or "I've started, so I must finish."
- The behaviour of (*COMMIT:NAME) is not the same as (*MARK:NAME)(*COM-
- MIT). It is like (*MARK:NAME) in that the name is remembered for pass-
- ing back to the caller. However, (*SKIP:NAME) searches only for names
- set with (*MARK), ignoring those set by (*COMMIT), (*PRUNE) and
+ The behaviour of (*COMMIT:NAME) is not the same as (*MARK:NAME)(*COM-
+ MIT). It is like (*MARK:NAME) in that the name is remembered for pass-
+ ing back to the caller. However, (*SKIP:NAME) searches only for names
+ set with (*MARK), ignoring those set by (*COMMIT), (*PRUNE) and
(*THEN).
- If there is more than one backtracking verb in a pattern, a different
- one that follows (*COMMIT) may be triggered first, so merely passing
+ If there is more than one backtracking verb in a pattern, a different
+ one that follows (*COMMIT) may be triggered first, so merely passing
(*COMMIT) during a match does not always guarantee that a match must be
at this starting point.
- Note that (*COMMIT) at the start of a pattern is not the same as an
- anchor, unless PCRE2's start-of-match optimizations are turned off, as
+ Note that (*COMMIT) at the start of a pattern is not the same as an
+ anchor, unless PCRE2's start-of-match optimizations are turned off, as
shown in this output from pcre2test:
re> /(*COMMIT)abc/
@@ -8921,63 +8947,63 @@
data> xyzabc
No match
- For the first pattern, PCRE2 knows that any match must start with "a",
- so the optimization skips along the subject to "a" before applying the
- pattern to the first set of data. The match attempt then succeeds. The
- second pattern disables the optimization that skips along to the first
- character. The pattern is now applied starting at "x", and so the
- (*COMMIT) causes the match to fail without trying any other starting
+ For the first pattern, PCRE2 knows that any match must start with "a",
+ so the optimization skips along the subject to "a" before applying the
+ pattern to the first set of data. The match attempt then succeeds. The
+ second pattern disables the optimization that skips along to the first
+ character. The pattern is now applied starting at "x", and so the
+ (*COMMIT) causes the match to fail without trying any other starting
points.
(*PRUNE) or (*PRUNE:NAME)
- This verb causes the match to fail at the current starting position in
+ This verb causes the match to fail at the current starting position in
the subject if there is a later matching failure that causes backtrack-
- ing to reach it. If the pattern is unanchored, the normal "bumpalong"
- advance to the next starting character then happens. Backtracking can
- occur as usual to the left of (*PRUNE), before it is reached, or when
- matching to the right of (*PRUNE), but if there is no match to the
- right, backtracking cannot cross (*PRUNE). In simple cases, the use of
- (*PRUNE) is just an alternative to an atomic group or possessive quan-
+ ing to reach it. If the pattern is unanchored, the normal "bumpalong"
+ advance to the next starting character then happens. Backtracking can
+ occur as usual to the left of (*PRUNE), before it is reached, or when
+ matching to the right of (*PRUNE), but if there is no match to the
+ right, backtracking cannot cross (*PRUNE). In simple cases, the use of
+ (*PRUNE) is just an alternative to an atomic group or possessive quan-
tifier, but there are some uses of (*PRUNE) that cannot be expressed in
- any other way. In an anchored pattern (*PRUNE) has the same effect as
+ any other way. In an anchored pattern (*PRUNE) has the same effect as
(*COMMIT).
The behaviour of (*PRUNE:NAME) is not the same as (*MARK:NAME)(*PRUNE).
It is like (*MARK:NAME) in that the name is remembered for passing back
- to the caller. However, (*SKIP:NAME) searches only for names set with
+ to the caller. However, (*SKIP:NAME) searches only for names set with
(*MARK), ignoring those set by (*COMMIT), (*PRUNE) or (*THEN).
(*SKIP)
- This verb, when given without a name, is like (*PRUNE), except that if
- the pattern is unanchored, the "bumpalong" advance is not to the next
+ This verb, when given without a name, is like (*PRUNE), except that if
+ the pattern is unanchored, the "bumpalong" advance is not to the next
character, but to the position in the subject where (*SKIP) was encoun-
- tered. (*SKIP) signifies that whatever text was matched leading up to
- it cannot be part of a successful match if there is a later mismatch.
+ tered. (*SKIP) signifies that whatever text was matched leading up to
+ it cannot be part of a successful match if there is a later mismatch.
Consider:
a+(*SKIP)b
- If the subject is "aaaac...", after the first match attempt fails
- (starting at the first character in the string), the starting point
+ If the subject is "aaaac...", after the first match attempt fails
+ (starting at the first character in the string), the starting point
skips on to start the next attempt at "c". Note that a possessive quan-
- tifer does not have the same effect as this example; although it would
- suppress backtracking during the first match attempt, the second
- attempt would start at the second character instead of skipping on to
+ tifer does not have the same effect as this example; although it would
+ suppress backtracking during the first match attempt, the second
+ attempt would start at the second character instead of skipping on to
"c".
(*SKIP:NAME)
- When (*SKIP) has an associated name, its behaviour is modified. When
- such a (*SKIP) is triggered, the previous path through the pattern is
- searched for the most recent (*MARK) that has the same name. If one is
- found, the "bumpalong" advance is to the subject position that corre-
- sponds to that (*MARK) instead of to where (*SKIP) was encountered. If
+ When (*SKIP) has an associated name, its behaviour is modified. When
+ such a (*SKIP) is triggered, the previous path through the pattern is
+ searched for the most recent (*MARK) that has the same name. If one is
+ found, the "bumpalong" advance is to the subject position that corre-
+ sponds to that (*MARK) instead of to where (*SKIP) was encountered. If
no (*MARK) with a matching name is found, the (*SKIP) is ignored.
- The search for a (*MARK) name uses the normal backtracking mechanism,
- which means that it does not see (*MARK) settings that are inside
+ The search for a (*MARK) name uses the normal backtracking mechanism,
+ which means that it does not see (*MARK) settings that are inside
atomic groups or assertions, because they are never re-entered by back-
tracking. Compare the following pcre2test examples:
@@ -8991,106 +9017,106 @@
0: b
1: b
- In the first example, the (*MARK) setting is in an atomic group, so it
+ In the first example, the (*MARK) setting is in an atomic group, so it
is not seen when (*SKIP:X) triggers, causing the (*SKIP) to be ignored.
- This allows the second branch of the pattern to be tried at the first
- character position. In the second example, the (*MARK) setting is not
- in an atomic group. This allows (*SKIP:X) to find the (*MARK) when it
+ This allows the second branch of the pattern to be tried at the first
+ character position. In the second example, the (*MARK) setting is not
+ in an atomic group. This allows (*SKIP:X) to find the (*MARK) when it
backtracks, and this causes a new matching attempt to start at the sec-
- ond character. This time, the (*MARK) is never seen because "a" does
+ ond character. This time, the (*MARK) is never seen because "a" does
not match "b", so the matcher immediately jumps to the second branch of
the pattern.
- Note that (*SKIP:NAME) searches only for names set by (*MARK:NAME). It
- ignores names that are set by (*COMMIT:NAME), (*PRUNE:NAME) or
+ Note that (*SKIP:NAME) searches only for names set by (*MARK:NAME). It
+ ignores names that are set by (*COMMIT:NAME), (*PRUNE:NAME) or
(*THEN:NAME).
(*THEN) or (*THEN:NAME)
- This verb causes a skip to the next innermost alternative when back-
- tracking reaches it. That is, it cancels any further backtracking
- within the current alternative. Its name comes from the observation
+ This verb causes a skip to the next innermost alternative when back-
+ tracking reaches it. That is, it cancels any further backtracking
+ within the current alternative. Its name comes from the observation
that it can be used for a pattern-based if-then-else block:
( COND1 (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ ) ...
- If the COND1 pattern matches, FOO is tried (and possibly further items
- after the end of the group if FOO succeeds); on failure, the matcher
- skips to the second alternative and tries COND2, without backtracking
- into COND1. If that succeeds and BAR fails, COND3 is tried. If subse-
- quently BAZ fails, there are no more alternatives, so there is a back-
- track to whatever came before the entire group. If (*THEN) is not
+ If the COND1 pattern matches, FOO is tried (and possibly further items
+ after the end of the group if FOO succeeds); on failure, the matcher
+ skips to the second alternative and tries COND2, without backtracking
+ into COND1. If that succeeds and BAR fails, COND3 is tried. If subse-
+ quently BAZ fails, there are no more alternatives, so there is a back-
+ track to whatever came before the entire group. If (*THEN) is not
inside an alternation, it acts like (*PRUNE).
- The behaviour of (*THEN:NAME) is not the same as (*MARK:NAME)(*THEN).
+ The behaviour of (*THEN:NAME) is not the same as (*MARK:NAME)(*THEN).
It is like (*MARK:NAME) in that the name is remembered for passing back
- to the caller. However, (*SKIP:NAME) searches only for names set with
+ to the caller. However, (*SKIP:NAME) searches only for names set with
(*MARK), ignoring those set by (*COMMIT), (*PRUNE) and (*THEN).
- A subpattern that does not contain a | character is just a part of the
- enclosing alternative; it is not a nested alternation with only one
- alternative. The effect of (*THEN) extends beyond such a subpattern to
- the enclosing alternative. Consider this pattern, where A, B, etc. are
- complex pattern fragments that do not contain any | characters at this
+ A subpattern that does not contain a | character is just a part of the
+ enclosing alternative; it is not a nested alternation with only one
+ alternative. The effect of (*THEN) extends beyond such a subpattern to
+ the enclosing alternative. Consider this pattern, where A, B, etc. are
+ complex pattern fragments that do not contain any | characters at this
level:
A (B(*THEN)C) | D
- If A and B are matched, but there is a failure in C, matching does not
+ If A and B are matched, but there is a failure in C, matching does not
backtrack into A; instead it moves to the next alternative, that is, D.
- However, if the subpattern containing (*THEN) is given an alternative,
+ However, if the subpattern containing (*THEN) is given an alternative,
it behaves differently:
A (B(*THEN)C | (*FAIL)) | D
- The effect of (*THEN) is now confined to the inner subpattern. After a
+ The effect of (*THEN) is now confined to the inner subpattern. After a
failure in C, matching moves to (*FAIL), which causes the whole subpat-
- tern to fail because there are no more alternatives to try. In this
+ tern to fail because there are no more alternatives to try. In this
case, matching does now backtrack into A.
- Note that a conditional subpattern is not considered as having two
- alternatives, because only one is ever used. In other words, the |
+ Note that a conditional subpattern is not considered as having two
+ alternatives, because only one is ever used. In other words, the |
character in a conditional subpattern has a different meaning. Ignoring
white space, consider:
^.*? (?(?=a) a | b(*THEN)c )
- If the subject is "ba", this pattern does not match. Because .*? is
- ungreedy, it initially matches zero characters. The condition (?=a)
- then fails, the character "b" is matched, but "c" is not. At this
- point, matching does not backtrack to .*? as might perhaps be expected
- from the presence of the | character. The conditional subpattern is
+ If the subject is "ba", this pattern does not match. Because .*? is
+ ungreedy, it initially matches zero characters. The condition (?=a)
+ then fails, the character "b" is matched, but "c" is not. At this
+ point, matching does not backtrack to .*? as might perhaps be expected
+ from the presence of the | character. The conditional subpattern is
part of the single alternative that comprises the whole pattern, and so
- the match fails. (If there was a backtrack into .*?, allowing it to
+ the match fails. (If there was a backtrack into .*?, allowing it to
match "b", the match would succeed.)
- The verbs just described provide four different "strengths" of control
+ The verbs just described provide four different "strengths" of control
when subsequent matching fails. (*THEN) is the weakest, carrying on the
- match at the next alternative. (*PRUNE) comes next, failing the match
- at the current starting position, but allowing an advance to the next
- character (for an unanchored pattern). (*SKIP) is similar, except that
+ match at the next alternative. (*PRUNE) comes next, failing the match
+ at the current starting position, but allowing an advance to the next
+ character (for an unanchored pattern). (*SKIP) is similar, except that
the advance may be more than one character. (*COMMIT) is the strongest,
causing the entire match to fail.
More than one backtracking verb
- If more than one backtracking verb is present in a pattern, the one
- that is backtracked onto first acts. For example, consider this pat-
+ If more than one backtracking verb is present in a pattern, the one
+ that is backtracked onto first acts. For example, consider this pat-
tern, where A, B, etc. are complex pattern fragments:
(A(*COMMIT)B(*THEN)C|ABD)
- If A matches but B fails, the backtrack to (*COMMIT) causes the entire
+ If A matches but B fails, the backtrack to (*COMMIT) causes the entire
match to fail. However, if A and B match, but C fails, the backtrack to
- (*THEN) causes the next alternative (ABD) to be tried. This behaviour
- is consistent, but is not always the same as Perl's. It means that if
- two or more backtracking verbs appear in succession, all the the last
+ (*THEN) causes the next alternative (ABD) to be tried. This behaviour
+ is consistent, but is not always the same as Perl's. It means that if
+ two or more backtracking verbs appear in succession, all the the last
of them has no effect. Consider this example:
...(*COMMIT)(*PRUNE)...
If there is a matching failure to the right, backtracking onto (*PRUNE)
- causes it to be triggered, and its action is taken. There can never be
+ causes it to be triggered, and its action is taken. There can never be
a backtrack onto (*COMMIT).
Backtracking verbs in repeated groups
@@ -9100,42 +9126,42 @@
/(a(*COMMIT)b)+ac/
- If the subject is "abac", Perl matches unless its optimizations are
- disabled, but PCRE2 always fails because the (*COMMIT) in the second
+ If the subject is "abac", Perl matches unless its optimizations are
+ disabled, but PCRE2 always fails because the (*COMMIT) in the second
repeat of the group acts.
Backtracking verbs in assertions
- (*FAIL) in any assertion has its normal effect: it forces an immediate
- backtrack. The behaviour of the other backtracking verbs depends on
- whether or not the assertion is standalone or acting as the condition
+ (*FAIL) in any assertion has its normal effect: it forces an immediate
+ backtrack. The behaviour of the other backtracking verbs depends on
+ whether or not the assertion is standalone or acting as the condition
in a conditional subpattern.
- (*ACCEPT) in a standalone positive assertion causes the assertion to
- succeed without any further processing; captured strings and a (*MARK)
- name (if set) are retained. In a standalone negative assertion,
- (*ACCEPT) causes the assertion to fail without any further processing;
+ (*ACCEPT) in a standalone positive assertion causes the assertion to
+ succeed without any further processing; captured strings and a (*MARK)
+ name (if set) are retained. In a standalone negative assertion,
+ (*ACCEPT) causes the assertion to fail without any further processing;
captured substrings and any (*MARK) name are discarded.
- If the assertion is a condition, (*ACCEPT) causes the condition to be
- true for a positive assertion and false for a negative one; captured
+ If the assertion is a condition, (*ACCEPT) causes the condition to be
+ true for a positive assertion and false for a negative one; captured
substrings are retained in both cases.
The remaining verbs act only when a later failure causes a backtrack to
- reach them. This means that their effect is confined to the assertion,
+ reach them. This means that their effect is confined to the assertion,
because lookaround assertions are atomic. A backtrack that occurs after
an assertion is complete does not jump back into the assertion. Note in
- particular that a (*MARK) name that is set in an assertion is not
+ particular that a (*MARK) name that is set in an assertion is not
"seen" by an instance of (*SKIP:NAME) latter in the pattern.
- The effect of (*THEN) is not allowed to escape beyond an assertion. If
- there are no more branches to try, (*THEN) causes a positive assertion
+ The effect of (*THEN) is not allowed to escape beyond an assertion. If
+ there are no more branches to try, (*THEN) causes a positive assertion
to be false, and a negative assertion to be true.
- The other backtracking verbs are not treated specially if they appear
- in a standalone positive assertion. In a conditional positive asser-
+ The other backtracking verbs are not treated specially if they appear
+ in a standalone positive assertion. In a conditional positive asser-
tion, backtracking (from within the assertion) into (*COMMIT), (*SKIP),
- or (*PRUNE) causes the condition to be false. However, for both stand-
+ or (*PRUNE) causes the condition to be false. However, for both stand-
alone and conditional negative assertions, backtracking into (*COMMIT),
(*SKIP), or (*PRUNE) causes the assertion to be true, without consider-
ing any further alternative branches.
@@ -9142,31 +9168,31 @@
Backtracking verbs in subroutines
- These behaviours occur whether or not the subpattern is called recur-
+ These behaviours occur whether or not the subpattern is called recur-
sively.
- (*ACCEPT) in a subpattern called as a subroutine causes the subroutine
- match to succeed without any further processing. Matching then contin-
- ues after the subroutine call. Perl documents this behaviour. Perl's
+ (*ACCEPT) in a subpattern called as a subroutine causes the subroutine
+ match to succeed without any further processing. Matching then contin-
+ ues after the subroutine call. Perl documents this behaviour. Perl's
treatment of the other verbs in subroutines is different in some cases.
- (*FAIL) in a subpattern called as a subroutine has its normal effect:
+ (*FAIL) in a subpattern called as a subroutine has its normal effect:
it forces an immediate backtrack.
- (*COMMIT), (*SKIP), and (*PRUNE) cause the subroutine match to fail
+ (*COMMIT), (*SKIP), and (*PRUNE) cause the subroutine match to fail
when triggered by being backtracked to in a subpattern called as a sub-
routine. There is then a backtrack at the outer level.
(*THEN), when triggered, skips to the next alternative in the innermost
enclosing group within the subpattern that has alternatives (its normal
- behaviour). However, if there is no such group within the subroutine
- subpattern, the subroutine match fails and there is a backtrack at the
+ behaviour). However, if there is no such group within the subroutine
+ subpattern, the subroutine match fails and there is a backtrack at the
outer level.
SEE ALSO
- pcre2api(3), pcre2callout(3), pcre2matching(3), pcre2syntax(3),
+ pcre2api(3), pcre2callout(3), pcre2matching(3), pcre2syntax(3),
pcre2(3).
@@ -9179,7 +9205,7 @@
REVISION
- Last updated: 21 September 2018
+ Last updated: 24 September 2018
Copyright (c) 1997-2018 University of Cambridge.
------------------------------------------------------------------------------
@@ -10291,6 +10317,7 @@
ATOMIC GROUPS
(?>...) atomic, non-capturing group
+ (*atomic:...) atomic, non-capturing group
COMMENT
@@ -10367,14 +10394,25 @@
LOOKAHEAD AND LOOKBEHIND ASSERTIONS
- (?=...) positive look ahead
- (?!...) negative look ahead
- (?<=...) positive look behind
- (?<!...) negative look behind
+ (?=...) )
+ (*pla:...) ) positive lookahead
+ (*positive_lookahead:...) )
- Each top-level branch of a look behind must be of a fixed length.
+ (?!...) )
+ (*nla:...) ) negative lookahead
+ (*negative_lookahead:...) )
+ (?<=...) )
+ (*plb:...) ) positive lookbehind
+ (*positive_lookbehind:...) )
+ (?<!...) )
+ (*nlb:...) ) negative lookbehind
+ (*negative_lookbehind:...) )
+
+ Each top-level branch of a lookbehind must be of a fixed length.
+
+
BACKREFERENCES
\n reference by number (can be ambiguous)
@@ -10487,7 +10525,7 @@
REVISION
- Last updated: 02 September 2018
+ Last updated: 24 September 2018
Copyright (c) 1997-2018 University of Cambridge.
------------------------------------------------------------------------------
Modified: code/trunk/doc/pcre2pattern.3
===================================================================
--- code/trunk/doc/pcre2pattern.3 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/doc/pcre2pattern.3 2018-09-24 16:23:53 UTC (rev 1018)
@@ -1,4 +1,4 @@
-.TH PCRE2PATTERN 3 "21 September 2018" "PCRE2 10.33"
+.TH PCRE2PATTERN 3 "24 September 2018" "PCRE2 10.33"
.SH NAME
PCRE2 - Perl-compatible regular expressions (revised API)
.SH "PCRE2 REGULAR EXPRESSION DETAILS"
@@ -2124,6 +2124,11 @@
.sp
(?>\ed+)foo
.sp
+Perl 5.28 introduced an experimental alphabetic form starting with (* which may
+be easier to remember:
+.sp
+ (*atomic:\ed+)foo
+.sp
This kind of parenthesis "locks up" the part of the pattern it contains once
it has matched, and a failure further into the pattern is prevented from
backtracking into it. Backtracking past it to previous items, however, works as
@@ -2351,12 +2356,20 @@
.P
More complicated assertions are coded as subpatterns. There are two kinds:
those that look ahead of the current position in the subject string, and those
-that look behind it, and in each case an assertion may be positive (must
-succeed for matching to continue) or negative (must not succeed for matching to
-continue). An assertion subpattern is matched in the normal way, except that,
-when matching continues after a successful assertion, the matching position in
-the subject string is as it was before the assertion was processed.
+that look behind it, and in each case an assertion may be positive (must match
+for the assertion to be true) or negative (must not match for the assertion to
+be true). An assertion subpattern is matched in the normal way, and if it is
+true, matching continues after it, but with the matching position in the
+subject string is was it was before the assertion was processed.
.P
+A lookaround assertion may also appear as the condition in a
+.\" HTML <a href="#conditions">
+.\" </a>
+conditional subpattern
+.\"
+(see below). In this case, the result of matching the assertion determines
+which branch of the condition is followed.
+.P
Assertion subpatterns are not capturing subpatterns. If an assertion contains
capturing subpatterns within it, these are counted for the purposes of
numbering the capturing subpatterns in the whole pattern. Within each branch of
@@ -2366,7 +2379,7 @@
.P
When a branch within an assertion fails to match, any substrings that were
captured are discarded (as happens with any pattern branch that fails to
-match). A negative assertion succeeds only when all its branches fail to match;
+match). A negative assertion is true only when all its branches fail to match;
this means that no captured substrings are ever retained after a successful
negative assertion. When an assertion contains a matching branch, what happens
depends on the type of assertion.
@@ -2374,7 +2387,7 @@
For a positive assertion, internally captured substrings in the successful
branch are retained, and matching continues with the next pattern item after
the assertion. For a negative assertion, a matching branch means that the
-assertion has failed. If the assertion is being used as a condition in a
+assertion is not true. If such an assertion is being used as a condition in a
.\" HTML <a href="#conditions">
.\" </a>
conditional subpattern
@@ -2406,6 +2419,25 @@
The assertion is obeyed just once when encountered during matching.
.
.
+.SS "Alphabetic assertion names"
+.rs
+.sp
+Traditionally, symbolic sequences such as (?= and (?<= have been used to specify
+lookaround assertions. Perl 5.28 introduced some experimental alphabetic
+alternatives which might be easier to remember. They all start with (* instead
+of (? and must be written using lower case letters. PCRE2 supports the
+following synonyms:
+.sp
+ (*positive_lookahead: or (*pla: is the same as (?=
+ (*negative_lookahead: or (*nla: is the same as (?!
+ (*positive_lookbehind: or (*plb: is the same as (?<=
+ (*negative_lookbehind: or (*nlb: is the same as (?<!
+.sp
+For example, (*pla:foo) is the same assertion as (?=foo). However, in the
+following sections, the various assertions are described using the original
+symbolic forms.
+.
+.
.SS "Lookahead assertions"
.rs
.sp
@@ -3660,6 +3692,6 @@
.rs
.sp
.nf
-Last updated: 21 September 2018
+Last updated: 24 September 2018
Copyright (c) 1997-2018 University of Cambridge.
.fi
Modified: code/trunk/doc/pcre2syntax.3
===================================================================
--- code/trunk/doc/pcre2syntax.3 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/doc/pcre2syntax.3 2018-09-24 16:23:53 UTC (rev 1018)
@@ -1,4 +1,4 @@
-.TH PCRE2SYNTAX 3 "02 September 2018" "PCRE2 10.32"
+.TH PCRE2SYNTAX 3 "24 September 2018" "PCRE2 10.33"
.SH NAME
PCRE2 - Perl-compatible regular expressions (revised API)
.SH "PCRE2 REGULAR EXPRESSION SYNTAX SUMMARY"
@@ -411,6 +411,7 @@
.rs
.sp
(?>...) atomic, non-capturing group
+ (*atomic:...) atomic, non-capturing group
.
.
.SH "COMMENT"
@@ -491,12 +492,23 @@
.SH "LOOKAHEAD AND LOOKBEHIND ASSERTIONS"
.rs
.sp
- (?=...) positive look ahead
- (?!...) negative look ahead
- (?<=...) positive look behind
- (?<!...) negative look behind
+ (?=...) )
+ (*pla:...) ) positive lookahead
+ (*positive_lookahead:...) )
.sp
-Each top-level branch of a look behind must be of a fixed length.
+ (?!...) )
+ (*nla:...) ) negative lookahead
+ (*negative_lookahead:...) )
+.sp
+ (?<=...) )
+ (*plb:...) ) positive lookbehind
+ (*positive_lookbehind:...) )
+.sp
+ (?<!...) )
+ (*nlb:...) ) negative lookbehind
+ (*negative_lookbehind:...) )
+.sp
+Each top-level branch of a lookbehind must be of a fixed length.
.
.
.SH "BACKREFERENCES"
@@ -621,6 +633,6 @@
.rs
.sp
.nf
-Last updated: 02 September 2018
+Last updated: 24 September 2018
Copyright (c) 1997-2018 University of Cambridge.
.fi
Modified: code/trunk/maint/pcre2_chartables.c.non-standard
===================================================================
--- code/trunk/maint/pcre2_chartables.c.non-standard 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/maint/pcre2_chartables.c.non-standard 2018-09-24 16:23:53 UTC (rev 1018)
@@ -103,19 +103,22 @@
0,0,0,0,0,0,0,128,
255,255,255,255,0,0,0,0,
0,0,0,0,0,0,0,0,
+
+/* Fiddled by hand when the table bits changed. May be broken! */
+
128,0,0,0,0,0,0,0,
-0,1,1,0,1,1,0,0,
+0,1,1,1,1,1,0,0,
0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,
1,0,0,0,128,0,0,0,
128,128,128,128,0,0,128,0,
-28,28,28,28,28,28,28,28,
-28,28,0,0,0,0,0,128,
-0,26,26,26,26,26,26,18,
+24,24,24,24,24,24,24,24,
+24,24,0,0,0,0,0,128,
+0,18,18,18,18,18,18,18,
18,18,18,18,18,18,18,18,
18,18,18,18,18,18,18,18,
18,18,18,128,128,0,128,16,
-0,26,26,26,26,26,26,18,
+0,18,18,18,18,18,18,18,
18,18,18,18,18,18,18,18,
18,18,18,18,18,18,18,18,
18,18,18,128,128,0,0,0,
@@ -125,8 +128,8 @@
0,0,0,0,0,0,0,0,
1,0,0,0,0,0,0,0,
0,0,18,0,0,0,0,0,
-0,0,20,20,0,18,0,0,
-0,20,18,0,0,0,0,0,
+0,0,24,24,0,18,0,0,
+0,24,18,0,0,0,0,0,
18,18,18,18,18,18,18,18,
18,18,18,18,18,18,18,18,
18,18,18,18,18,18,18,0,
Modified: code/trunk/perltest.sh
===================================================================
--- code/trunk/perltest.sh 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/perltest.sh 2018-09-24 16:23:53 UTC (rev 1018)
@@ -75,6 +75,10 @@
(echo "$prefix" ; cat <<'PERLEND'
+# The alpha assertions currently give warnings even when -w is not specified.
+
+no warnings "experimental::alpha_assertions";
+
# Function for turning a string into a string of printing chars.
sub pchars {
@@ -129,6 +133,9 @@
printf($outfile "Perl $] Regular Expressions\n\n");
+$extra_modifiers = "";
+$default_show_mark = 0;
+
# Main loop
NEXT_RE:
@@ -370,8 +377,11 @@
}
}
-# printf $outfile "\n";
+# By closing OUTFILE explicitly, we avoid a Perl warning in -w mode
+# "main::OUTFILE" used only once".
+close(OUTFILE) if $outfile eq "OUTFILE";
+
PERLEND
) | $perl $perlarg - $@
Modified: code/trunk/src/dftables.c
===================================================================
--- code/trunk/src/dftables.c 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/src/dftables.c 2018-09-24 16:23:53 UTC (rev 1018)
@@ -183,10 +183,10 @@
"/* This table identifies various classes of character by individual bits:\n"
" 0x%02x white space character\n"
" 0x%02x letter\n"
+ " 0x%02x lower case letter\n"
" 0x%02x decimal digit\n"
- " 0x%02x hexadecimal digit\n"
" 0x%02x alphanumeric or '_'\n*/\n\n",
- ctype_space, ctype_letter, ctype_digit, ctype_xdigit, ctype_word);
+ ctype_space, ctype_letter, ctype_lcletter, ctype_digit, ctype_word);
fprintf(f, " ");
for (i = 0; i < 256; i++)
Modified: code/trunk/src/pcre2.h.in
===================================================================
--- code/trunk/src/pcre2.h.in 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/src/pcre2.h.in 2018-09-24 16:23:53 UTC (rev 1018)
@@ -320,6 +320,7 @@
#define PCRE2_ERROR_BAD_LITERAL_OPTIONS 192
#define PCRE2_ERROR_SUPPORTED_ONLY_IN_UNICODE 193
#define PCRE2_ERROR_INVALID_HYPHEN_IN_OPTIONS 194
+#define PCRE2_ERROR_ALPHA_ASSERTION_UNKNOWN 195
/* "Expected" matching error codes: no match and partial match. */
Modified: code/trunk/src/pcre2_chartables.c.dist
===================================================================
--- code/trunk/src/pcre2_chartables.c.dist 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/src/pcre2_chartables.c.dist 2018-09-24 16:23:53 UTC (rev 1018)
@@ -157,8 +157,8 @@
/* This table identifies various classes of character by individual bits:
0x01 white space character
0x02 letter
- 0x04 decimal digit
- 0x08 hexadecimal digit
+ 0x04 lower case letter
+ 0x08 decimal digit
0x10 alphanumeric or '_'
*/
@@ -168,16 +168,16 @@
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00, /* 24- 31 */
0x01,0x00,0x00,0x00,0x00,0x00,0x00,0x00, /* - ' */
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00, /* ( - / */
- 0x1c,0x1c,0x1c,0x1c,0x1c,0x1c,0x1c,0x1c, /* 0 - 7 */
- 0x1c,0x1c,0x00,0x00,0x00,0x00,0x00,0x00, /* 8 - ? */
- 0x00,0x1a,0x1a,0x1a,0x1a,0x1a,0x1a,0x12, /* @ - G */
+ 0x18,0x18,0x18,0x18,0x18,0x18,0x18,0x18, /* 0 - 7 */
+ 0x18,0x18,0x00,0x00,0x00,0x00,0x00,0x00, /* 8 - ? */
+ 0x00,0x12,0x12,0x12,0x12,0x12,0x12,0x12, /* @ - G */
0x12,0x12,0x12,0x12,0x12,0x12,0x12,0x12, /* H - O */
0x12,0x12,0x12,0x12,0x12,0x12,0x12,0x12, /* P - W */
0x12,0x12,0x12,0x00,0x00,0x00,0x00,0x10, /* X - _ */
- 0x00,0x1a,0x1a,0x1a,0x1a,0x1a,0x1a,0x12, /* ` - g */
- 0x12,0x12,0x12,0x12,0x12,0x12,0x12,0x12, /* h - o */
- 0x12,0x12,0x12,0x12,0x12,0x12,0x12,0x12, /* p - w */
- 0x12,0x12,0x12,0x00,0x00,0x00,0x00,0x00, /* x -127 */
+ 0x00,0x16,0x16,0x16,0x16,0x16,0x16,0x16, /* ` - g */
+ 0x16,0x16,0x16,0x16,0x16,0x16,0x16,0x16, /* h - o */
+ 0x16,0x16,0x16,0x16,0x16,0x16,0x16,0x16, /* p - w */
+ 0x16,0x16,0x16,0x00,0x00,0x00,0x00,0x00, /* x -127 */
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00, /* 128-135 */
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00, /* 136-143 */
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00, /* 144-151 */
Modified: code/trunk/src/pcre2_compile.c
===================================================================
--- code/trunk/src/pcre2_compile.c 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/src/pcre2_compile.c 2018-09-24 16:23:53 UTC (rev 1018)
@@ -615,6 +615,46 @@
OP_MARK, OP_ACCEPT, OP_FAIL, OP_COMMIT, OP_COMMIT_ARG, OP_PRUNE,
OP_PRUNE_ARG, OP_SKIP, OP_SKIP_ARG, OP_THEN, OP_THEN_ARG };
+/* Table of "alpha assertions" like (*pla:...), similar to the (*VERB) table. */
+
+typedef struct alasitem {
+ unsigned int len; /* Length of name */
+ uint32_t meta; /* Base META_ code */
+} alasitem;
+
+static const char alasnames[] =
+ STRING_pla0
+ STRING_plb0
+ STRING_nla0
+ STRING_nlb0
+ STRING_positive_lookahead0
+ STRING_positive_lookbehind0
+ STRING_negative_lookahead0
+ STRING_negative_lookbehind0
+ STRING_atomic0
+ STRING_sr0
+ STRING_asr0
+ STRING_script_run0
+ STRING_atomic_script_run;
+
+static const alasitem alasmeta[] = {
+ { 3, META_LOOKAHEAD },
+ { 3, META_LOOKBEHIND },
+ { 3, META_LOOKAHEADNOT },
+ { 3, META_LOOKBEHINDNOT },
+ { 18, META_LOOKAHEAD },
+ { 19, META_LOOKBEHIND },
+ { 18, META_LOOKAHEADNOT },
+ { 19, META_LOOKBEHINDNOT },
+ { 6, META_ATOMIC },
+ { 2, 0 }, /* sr = script run */
+ { 3, 0 }, /* asr = atomic script run */
+ { 10, 0 }, /* script run */
+ { 17, 0 } /* atomic script run */
+};
+
+static const int alascount = sizeof(alasmeta)/sizeof(alasitem);
+
/* Offsets from OP_STAR for case-independent and negative repeat opcodes. */
static uint32_t chartypeoffset[] = {
@@ -732,7 +772,7 @@
ERR61, ERR62, ERR63, ERR64, ERR65, ERR66, ERR67, ERR68, ERR69, ERR70,
ERR71, ERR72, ERR73, ERR74, ERR75, ERR76, ERR77, ERR78, ERR79, ERR80,
ERR81, ERR82, ERR83, ERR84, ERR85, ERR86, ERR87, ERR88, ERR89, ERR90,
- ERR91, ERR92, ERR93, ERR94 };
+ ERR91, ERR92, ERR93, ERR94, ERR95 };
/* This is a table of start-of-pattern options such as (*UTF) and settings such
as (*LIMIT_MATCH=nnnn) and (*CRLF). For completeness and backward
@@ -1447,9 +1487,9 @@
c = (uint32_t)i;
if (cb != NULL && c == CHAR_CR &&
(cb->cx->extra_options & PCRE2_EXTRA_ESCAPED_CR_IS_LF) != 0)
- c = CHAR_LF;
+ c = CHAR_LF;
}
- else /* Negative table entry */
+ else /* Negative table entry */
{
escape = -i; /* Else return a special escape */
if (cb != NULL && (escape == ESC_P || escape == ESC_p || escape == ESC_X))
@@ -1499,7 +1539,7 @@
}
}
-/* Escapes that need further processing, including those that are unknown, have
+/* Escapes that need further processing, including those that are unknown, have
a zero entry in the lookup table. When called from pcre2_substitute(), only \c,
\o, and \x are recognized (and \u when BSUX is set). */
@@ -2133,9 +2173,10 @@
*************************************************/
/* This function is called from parse_regex() below whenever it needs to read
-the name of a subpattern or a (*VERB). The initial pointer must be to the
-character before the name. If that character is '*' we are reading a verb name.
-The pointer is updated to point after the name, for a VERB, or after tha name's
+the name of a subpattern or a (*VERB) or an (*alpha_assertion). The initial
+pointer must be to the character before the name. If that character is '*' we
+are reading a verb or alpha assertion name. The pointer is updated to point
+after the name, for a VERB or alpha assertion name, or after tha name's
terminator for a subpattern name. Returning both the offset and the name
pointer is redundant information, but some callers use one and some the other,
so it is simplest just to return both.
@@ -2160,28 +2201,30 @@
int *errorcodeptr, compile_block *cb)
{
PCRE2_SPTR ptr = *ptrptr;
-BOOL is_verb = (*ptr == CHAR_ASTERISK);
+BOOL is_group = (*ptr != CHAR_ASTERISK);
uint32_t namelen = 0;
-uint32_t ctype = is_verb? ctype_letter : ctype_word;
-if (++ptr >= ptrend)
+if (++ptr >= ptrend) /* No characters in name */
{
- *errorcodeptr = is_verb? ERR60: /* Verb not recognized or malformed */
- ERR62; /* Subpattern name expected */
+ *errorcodeptr = is_group? ERR62: /* Subpattern name expected */
+ ERR60; /* Verb not recognized or malformed */
goto FAILED;
}
+
+/* A group name must not start with a digit. If either of the others start with
+a digit it just won't be recognized. */
+
+if (is_group && IS_DIGIT(*ptr))
+ {
+ *errorcodeptr = ERR44;
+ goto FAILED;
+ }
*nameptr = ptr;
*offsetptr = (PCRE2_SIZE)(ptr - cb->start_pattern);
-if (IS_DIGIT(*ptr))
+while (ptr < ptrend && MAX_255(*ptr) && (cb->ctypes[*ptr] & ctype_word) != 0)
{
- *errorcodeptr = ERR44; /* Group name must not start with digit */
- goto FAILED;
- }
-
-while (ptr < ptrend && MAX_255(*ptr) && (cb->ctypes[*ptr] & ctype) != 0)
- {
ptr++;
namelen++;
if (namelen > MAX_NAME_SIZE)
@@ -2192,9 +2235,9 @@
}
/* Subpattern names must not be empty, and their terminator is checked here.
-(What follows a verb name is checked separately.) */
+(What follows a verb or alpha assertion name is checked separately.) */
-if (!is_verb)
+if (is_group)
{
if (namelen == 0)
{
@@ -2652,25 +2695,32 @@
if (expect_cond_assert > 0)
{
BOOL ok = c == CHAR_LEFT_PARENTHESIS && ptrend - ptr >= 3 &&
- ptr[0] == CHAR_QUESTION_MARK;
- if (ok) switch(ptr[1])
+ (ptr[0] == CHAR_QUESTION_MARK || ptr[0] == CHAR_ASTERISK);
+ if (ok)
{
- case CHAR_C:
- ok = expect_cond_assert == 2;
- break;
+ if (ptr[0] == CHAR_ASTERISK) /* New alpha assertion format, possibly */
+ {
+ ok = MAX_255(ptr[1]) && (cb->ctypes[ptr[1]] & ctype_lcletter) != 0;
+ }
+ else switch(ptr[1]) /* Traditional symbolic format */
+ {
+ case CHAR_C:
+ ok = expect_cond_assert == 2;
+ break;
+
+ case CHAR_EQUALS_SIGN:
+ case CHAR_EXCLAMATION_MARK:
+ break;
+
+ case CHAR_LESS_THAN_SIGN:
+ ok = ptr[2] == CHAR_EQUALS_SIGN || ptr[2] == CHAR_EXCLAMATION_MARK;
+ break;
+
+ default:
+ ok = FALSE;
+ }
+ }
- case CHAR_EQUALS_SIGN:
- case CHAR_EXCLAMATION_MARK:
- break;
-
- case CHAR_LESS_THAN_SIGN:
- ok = ptr[2] == CHAR_EQUALS_SIGN || ptr[2] == CHAR_EXCLAMATION_MARK;
- break;
-
- default:
- ok = FALSE;
- }
-
if (!ok)
{
ptr--; /* Adjust error offset */
@@ -3453,7 +3503,8 @@
case CHAR_LEFT_PARENTHESIS:
if (ptr >= ptrend) goto UNCLOSED_PARENTHESIS;
- /* If ( is not followed by ? it is either a capture or a special verb. */
+ /* If ( is not followed by ? it is either a capture or a special verb or an
+ alpha assertion. */
if (*ptr != CHAR_QUESTION_MARK)
{
@@ -3473,14 +3524,89 @@
else *parsed_pattern++ = META_NOCAPTURE;
}
+ /* Do nothing for (* followed by end of pattern or ) so it gives a "bad
+ quantifier" error rather than "(*MARK) must have an argument". */
- /* ---- Handle (*VERB) and (*VERB:NAME) ---- */
+ else if (ptrend - ptr <= 1 || (c = ptr[1]) == CHAR_RIGHT_PARENTHESIS)
+ break;
- /* Do nothing for (*) so it gives a "bad quantifier" error rather than
- "(*MARK) must have an argument". */
+ /* Handle "alpha assertions" such as (*pla:...). Most of these are
+ synonyms for the historical symbolic assertions, but the script run ones
+ are new. They are distinguished by starting with a lower case letter.
+ Checking both ends of the alphabet makes this work in all character
+ codes. */
- else if (ptrend - ptr > 1 && ptr[1] != CHAR_RIGHT_PARENTHESIS)
+ else if (CHMAX_255(c) && (cb->ctypes[c] & ctype_lcletter) != 0)
{
+ uint32_t meta;
+
+ vn = alasnames;
+ if (!read_name(&ptr, ptrend, 0, &offset, &name, &namelen, &errorcode,
+ cb)) goto FAILED;
+ if (ptr >= ptrend || *ptr != CHAR_COLON)
+ {
+ errorcode = ERR95; /* Malformed */
+ goto FAILED;
+ }
+
+ /* Scan the table of alpha assertion names */
+
+ for (i = 0; i < alascount; i++)
+ {
+ if (namelen == alasmeta[i].len &&
+ PRIV(strncmp_c8)(name, vn, namelen) == 0)
+ break;
+ vn += alasmeta[i].len + 1;
+ }
+
+ if (i >= alascount)
+ {
+ errorcode = ERR95; /* Alpha assertion not recognized */
+ goto FAILED;
+ }
+
+ /* Check for expecting an assertion condition. If so, only lookaround
+ assertions are valid. */
+
+ meta = alasmeta[i].meta;
+ if (prev_expect_cond_assert > 0 &&
+ (meta < META_LOOKAHEAD || meta > META_LOOKBEHINDNOT))
+ {
+ errorcode = ERR28; /* Assertion expected */
+ goto FAILED;
+ }
+
+ switch(meta)
+ {
+ case META_ATOMIC:
+ goto ATOMIC_GROUP;
+
+ case META_LOOKAHEAD:
+ goto POSITIVE_LOOK_AHEAD;
+
+ case META_LOOKAHEADNOT:
+ goto NEGATIVE_LOOK_AHEAD;
+
+ case META_LOOKBEHIND:
+ case META_LOOKBEHINDNOT:
+ *parsed_pattern++ = meta;
+ ptr--;
+ goto LOOKBEHIND;
+
+ /* FIXME: Script Run stuff ... */
+
+
+
+
+
+ }
+ }
+
+
+ /* ---- Handle (*VERB) and (*VERB:NAME) ---- */
+
+ else
+ {
vn = verbnames;
if (!read_name(&ptr, ptrend, 0, &offset, &name, &namelen, &errorcode,
cb)) goto FAILED;
@@ -3946,14 +4072,15 @@
if (++ptr >= ptrend) goto UNCLOSED_PARENTHESIS;
nest_depth++;
- /* If the next character is ? there must be an assertion next (optionally
- preceded by a callout). We do not check this here, but instead we set
- expect_cond_assert to 2. If this is still greater than zero (callouts
- decrement it) when the next assertion is read, it will be marked as a
- condition that must not be repeated. A value greater than zero also
- causes checking that an assertion (possibly with callout) follows. */
+ /* If the next character is ? or * there must be an assertion next
+ (optionally preceded by a callout). We do not check this here, but
+ instead we set expect_cond_assert to 2. If this is still greater than
+ zero (callouts decrement it) when the next assertion is read, it will be
+ marked as a condition that must not be repeated. A value greater than
+ zero also causes checking that an assertion (possibly with callout)
+ follows. */
- if (*ptr == CHAR_QUESTION_MARK)
+ if (*ptr == CHAR_QUESTION_MARK || *ptr == CHAR_ASTERISK)
{
*parsed_pattern++ = META_COND_ASSERT;
ptr--; /* Pull pointer back to the opening parenthesis. */
@@ -4099,6 +4226,7 @@
/* ---- Atomic group ---- */
case CHAR_GREATER_THAN_SIGN:
+ ATOMIC_GROUP: /* Come from (*atomic: */
*parsed_pattern++ = META_ATOMIC;
nest_depth++;
ptr++;
@@ -4108,11 +4236,13 @@
/* ---- Lookahead assertions ---- */
case CHAR_EQUALS_SIGN:
+ POSITIVE_LOOK_AHEAD: /* Come from (*pla: */
*parsed_pattern++ = META_LOOKAHEAD;
ptr++;
goto POST_ASSERTION;
case CHAR_EXCLAMATION_MARK:
+ NEGATIVE_LOOK_AHEAD: /* Come from (*nla: */
*parsed_pattern++ = META_LOOKAHEADNOT;
ptr++;
goto POST_ASSERTION;
@@ -4132,6 +4262,8 @@
}
*parsed_pattern++ = (ptr[1] == CHAR_EQUALS_SIGN)?
META_LOOKBEHIND : META_LOOKBEHINDNOT;
+
+ LOOKBEHIND: /* Come from (*plb: and (*nlb: */
*has_lookbehind = TRUE;
offset = (PCRE2_SIZE)(ptr - cb->start_pattern - 2);
PUTOFFSET(offset, parsed_pattern);
Modified: code/trunk/src/pcre2_error.c
===================================================================
--- code/trunk/src/pcre2_error.c 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/src/pcre2_error.c 2018-09-24 16:23:53 UTC (rev 1018)
@@ -181,6 +181,8 @@
"invalid option bits with PCRE2_LITERAL\0"
"\\N{U+dddd} is supported only in Unicode (UTF) mode\0"
"invalid hyphen in option setting\0"
+ /* 95 */
+ "(*alpha_assertion) not recognized\0"
;
/* Match-time and UTF error texts are in the same format. */
Modified: code/trunk/src/pcre2_internal.h
===================================================================
--- code/trunk/src/pcre2_internal.h 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/src/pcre2_internal.h 2018-09-24 16:23:53 UTC (rev 1018)
@@ -569,11 +569,11 @@
without checking pcre2_jit_compile.c, which has an assertion to ensure that
ctype_word has the value 16. */
-#define ctype_space 0x01
-#define ctype_letter 0x02
-#define ctype_digit 0x04
-#define ctype_xdigit 0x08 /* not actually used any more */
-#define ctype_word 0x10 /* alphanumeric or '_' */
+#define ctype_space 0x01
+#define ctype_letter 0x02
+#define ctype_lcletter 0x04
+#define ctype_digit 0x08
+#define ctype_word 0x10 /* alphanumeric or '_' */
/* Offsets of the various tables from the base tables pointer, and
total length of the tables. */
@@ -874,35 +874,49 @@
#define STR_RIGHT_CURLY_BRACKET "}"
#define STR_TILDE "~"
-#define STRING_ACCEPT0 "ACCEPT\0"
-#define STRING_COMMIT0 "COMMIT\0"
-#define STRING_F0 "F\0"
-#define STRING_FAIL0 "FAIL\0"
-#define STRING_MARK0 "MARK\0"
-#define STRING_PRUNE0 "PRUNE\0"
-#define STRING_SKIP0 "SKIP\0"
-#define STRING_THEN "THEN"
+#define STRING_ACCEPT0 "ACCEPT\0"
+#define STRING_COMMIT0 "COMMIT\0"
+#define STRING_F0 "F\0"
+#define STRING_FAIL0 "FAIL\0"
+#define STRING_MARK0 "MARK\0"
+#define STRING_PRUNE0 "PRUNE\0"
+#define STRING_SKIP0 "SKIP\0"
+#define STRING_THEN "THEN"
-#define STRING_alpha0 "alpha\0"
-#define STRING_lower0 "lower\0"
-#define STRING_upper0 "upper\0"
-#define STRING_alnum0 "alnum\0"
-#define STRING_ascii0 "ascii\0"
-#define STRING_blank0 "blank\0"
-#define STRING_cntrl0 "cntrl\0"
-#define STRING_digit0 "digit\0"
-#define STRING_graph0 "graph\0"
-#define STRING_print0 "print\0"
-#define STRING_punct0 "punct\0"
-#define STRING_space0 "space\0"
-#define STRING_word0 "word\0"
-#define STRING_xdigit "xdigit"
+#define STRING_atomic0 "atomic\0"
+#define STRING_pla0 "pla\0"
+#define STRING_plb0 "plb\0"
+#define STRING_nla0 "nla\0"
+#define STRING_nlb0 "nlb\0"
+#define STRING_sr0 "sr\0"
+#define STRING_asr0 "asr\0"
+#define STRING_positive_lookahead0 "positive_lookahead\0"
+#define STRING_positive_lookbehind0 "positive_lookbehind\0"
+#define STRING_negative_lookahead0 "negative_lookahead\0"
+#define STRING_negative_lookbehind0 "negative_lookbehind\0"
+#define STRING_script_run0 "script_run\0"
+#define STRING_atomic_script_run "atomic_script_run"
-#define STRING_DEFINE "DEFINE"
-#define STRING_VERSION "VERSION"
-#define STRING_WEIRD_STARTWORD "[:<:]]"
-#define STRING_WEIRD_ENDWORD "[:>:]]"
+#define STRING_alpha0 "alpha\0"
+#define STRING_lower0 "lower\0"
+#define STRING_upper0 "upper\0"
+#define STRING_alnum0 "alnum\0"
+#define STRING_ascii0 "ascii\0"
+#define STRING_blank0 "blank\0"
+#define STRING_cntrl0 "cntrl\0"
+#define STRING_digit0 "digit\0"
+#define STRING_graph0 "graph\0"
+#define STRING_print0 "print\0"
+#define STRING_punct0 "punct\0"
+#define STRING_space0 "space\0"
+#define STRING_word0 "word\0"
+#define STRING_xdigit "xdigit"
+#define STRING_DEFINE "DEFINE"
+#define STRING_VERSION "VERSION"
+#define STRING_WEIRD_STARTWORD "[:<:]]"
+#define STRING_WEIRD_ENDWORD "[:>:]]"
+
#define STRING_CR_RIGHTPAR "CR)"
#define STRING_LF_RIGHTPAR "LF)"
#define STRING_CRLF_RIGHTPAR "CRLF)"
@@ -1150,35 +1164,49 @@
#define STR_RIGHT_CURLY_BRACKET "\175"
#define STR_TILDE "\176"
-#define STRING_ACCEPT0 STR_A STR_C STR_C STR_E STR_P STR_T "\0"
-#define STRING_COMMIT0 STR_C STR_O STR_M STR_M STR_I STR_T "\0"
-#define STRING_F0 STR_F "\0"
-#define STRING_FAIL0 STR_F STR_A STR_I STR_L "\0"
-#define STRING_MARK0 STR_M STR_A STR_R STR_K "\0"
-#define STRING_PRUNE0 STR_P STR_R STR_U STR_N STR_E "\0"
-#define STRING_SKIP0 STR_S STR_K STR_I STR_P "\0"
-#define STRING_THEN STR_T STR_H STR_E STR_N
+#define STRING_ACCEPT0 STR_A STR_C STR_C STR_E STR_P STR_T "\0"
+#define STRING_COMMIT0 STR_C STR_O STR_M STR_M STR_I STR_T "\0"
+#define STRING_F0 STR_F "\0"
+#define STRING_FAIL0 STR_F STR_A STR_I STR_L "\0"
+#define STRING_MARK0 STR_M STR_A STR_R STR_K "\0"
+#define STRING_PRUNE0 STR_P STR_R STR_U STR_N STR_E "\0"
+#define STRING_SKIP0 STR_S STR_K STR_I STR_P "\0"
+#define STRING_THEN STR_T STR_H STR_E STR_N
-#define STRING_alpha0 STR_a STR_l STR_p STR_h STR_a "\0"
-#define STRING_lower0 STR_l STR_o STR_w STR_e STR_r "\0"
-#define STRING_upper0 STR_u STR_p STR_p STR_e STR_r "\0"
-#define STRING_alnum0 STR_a STR_l STR_n STR_u STR_m "\0"
-#define STRING_ascii0 STR_a STR_s STR_c STR_i STR_i "\0"
-#define STRING_blank0 STR_b STR_l STR_a STR_n STR_k "\0"
-#define STRING_cntrl0 STR_c STR_n STR_t STR_r STR_l "\0"
-#define STRING_digit0 STR_d STR_i STR_g STR_i STR_t "\0"
-#define STRING_graph0 STR_g STR_r STR_a STR_p STR_h "\0"
-#define STRING_print0 STR_p STR_r STR_i STR_n STR_t "\0"
-#define STRING_punct0 STR_p STR_u STR_n STR_c STR_t "\0"
-#define STRING_space0 STR_s STR_p STR_a STR_c STR_e "\0"
-#define STRING_word0 STR_w STR_o STR_r STR_d "\0"
-#define STRING_xdigit STR_x STR_d STR_i STR_g STR_i STR_t
+#define STRING_atomic0 STR_a STR_t STR_o STR_m STR_i STR_c "\0"
+#define STRING_pla0 STR_p STR_l STR_a "\0"
+#define STRING_plb0 STR_p STR_l STR_b "\0"
+#define STRING_nla0 STR_n STR_l STR_a "\0"
+#define STRING_nlb0 STR_n STR_l STR_b "\0"
+#define STRING_sr0 STR_s STR_r "\0"
+#define STRING_asr0 STR_a STR_s STR_r "\0"
+#define STRING_positive_lookahead0 STR_p STR_o STR_s STR_i STR_t STR_i STR_v STR_e STR_UNDERSCORE STR_l STR_o STR_o STR_k STR_a STR_h STR_e STR_a STR_d "\0"
+#define STRING_positive_lookbehind0 STR_p STR_o STR_s STR_i STR_t STR_i STR_v STR_e STR_UNDERSCORE STR_l STR_o STR_o STR_k STR_b STR_e STR_h STR_i STR_n STR_d "\0"
+#define STRING_negative_lookahead0 STR_n STR_e STR_g STR_a STR_t STR_i STR_v STR_e STR_UNDERSCORE STR_l STR_o STR_o STR_k STR_a STR_h STR_e STR_a STR_d "\0"
+#define STRING_negative_lookbehind0 STR_n STR_e STR_g STR_a STR_t STR_i STR_v STR_e STR_UNDERSCORE STR_l STR_o STR_o STR_k STR_b STR_e STR_h STR_i STR_n STR_d "\0"
+#define STRING_script_run0 STR_s STR_c STR_r STR_i STR_p STR_t STR_UNDERSCORE STR_r STR_u STR_n "\0"
+#define STRING_atomic_script_run STR_a STR_t STR_o STR_m STR_i STR_c STR_UNDERSCORE STR_s STR_c STR_r STR_i STR_p STR_t STR_UNDERSCORE STR_r STR_u STR_n
-#define STRING_DEFINE STR_D STR_E STR_F STR_I STR_N STR_E
-#define STRING_VERSION STR_V STR_E STR_R STR_S STR_I STR_O STR_N
-#define STRING_WEIRD_STARTWORD STR_LEFT_SQUARE_BRACKET STR_COLON STR_LESS_THAN_SIGN STR_COLON STR_RIGHT_SQUARE_BRACKET STR_RIGHT_SQUARE_BRACKET
-#define STRING_WEIRD_ENDWORD STR_LEFT_SQUARE_BRACKET STR_COLON STR_GREATER_THAN_SIGN STR_COLON STR_RIGHT_SQUARE_BRACKET STR_RIGHT_SQUARE_BRACKET
+#define STRING_alpha0 STR_a STR_l STR_p STR_h STR_a "\0"
+#define STRING_lower0 STR_l STR_o STR_w STR_e STR_r "\0"
+#define STRING_upper0 STR_u STR_p STR_p STR_e STR_r "\0"
+#define STRING_alnum0 STR_a STR_l STR_n STR_u STR_m "\0"
+#define STRING_ascii0 STR_a STR_s STR_c STR_i STR_i "\0"
+#define STRING_blank0 STR_b STR_l STR_a STR_n STR_k "\0"
+#define STRING_cntrl0 STR_c STR_n STR_t STR_r STR_l "\0"
+#define STRING_digit0 STR_d STR_i STR_g STR_i STR_t "\0"
+#define STRING_graph0 STR_g STR_r STR_a STR_p STR_h "\0"
+#define STRING_print0 STR_p STR_r STR_i STR_n STR_t "\0"
+#define STRING_punct0 STR_p STR_u STR_n STR_c STR_t "\0"
+#define STRING_space0 STR_s STR_p STR_a STR_c STR_e "\0"
+#define STRING_word0 STR_w STR_o STR_r STR_d "\0"
+#define STRING_xdigit STR_x STR_d STR_i STR_g STR_i STR_t
+#define STRING_DEFINE STR_D STR_E STR_F STR_I STR_N STR_E
+#define STRING_VERSION STR_V STR_E STR_R STR_S STR_I STR_O STR_N
+#define STRING_WEIRD_STARTWORD STR_LEFT_SQUARE_BRACKET STR_COLON STR_LESS_THAN_SIGN STR_COLON STR_RIGHT_SQUARE_BRACKET STR_RIGHT_SQUARE_BRACKET
+#define STRING_WEIRD_ENDWORD STR_LEFT_SQUARE_BRACKET STR_COLON STR_GREATER_THAN_SIGN STR_COLON STR_RIGHT_SQUARE_BRACKET STR_RIGHT_SQUARE_BRACKET
+
#define STRING_CR_RIGHTPAR STR_C STR_R STR_RIGHT_PARENTHESIS
#define STRING_LF_RIGHTPAR STR_L STR_F STR_RIGHT_PARENTHESIS
#define STRING_CRLF_RIGHTPAR STR_C STR_R STR_L STR_F STR_RIGHT_PARENTHESIS
Modified: code/trunk/src/pcre2_maketables.c
===================================================================
--- code/trunk/src/pcre2_maketables.c 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/src/pcre2_maketables.c 2018-09-24 16:23:53 UTC (rev 1018)
@@ -138,8 +138,8 @@
int x = 0;
if (isspace(i)) x += ctype_space;
if (isalpha(i)) x += ctype_letter;
+ if (islower(i)) x += ctype_lcletter;
if (isdigit(i)) x += ctype_digit;
- if (isxdigit(i)) x += ctype_xdigit;
if (isalnum(i) || i == '_') x += ctype_word;
*p++ = x;
}
Modified: code/trunk/testdata/testinput1
===================================================================
--- code/trunk/testdata/testinput1 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/testdata/testinput1 2018-09-24 16:23:53 UTC (rev 1018)
@@ -6263,4 +6263,69 @@
aBCDEF
AbCDe f
+/(*pla:foo).{6}/
+ abcfoobarxyz
+\= Expect no match
+ abcfooba
+
+/(*positive_lookahead:foo).{6}/
+ abcfoobarxyz
+
+/(?(*pla:foo).{6}|a..)/
+ foobarbaz
+ abcfoobar
+
+/(?(*positive_lookahead:foo).{6}|a..)/
+ foobarbaz
+ abcfoobar
+
+/(*plb:foo)bar/
+ abcfoobar
+\= Expect no match
+ abcbarfoo
+
+/(*positive_lookbehind:foo)bar/
+ abcfoobar
+\= Expect no match
+ abcbarfoo
+
+/(?(*plb:foo)bar|baz)/
+ abcfoobar
+ bazfoobar
+ abcbazfoobar
+ foobazfoobar
+
+/(?(*positive_lookbehind:foo)bar|baz)/
+ abcfoobar
+ bazfoobar
+ abcbazfoobar
+ foobazfoobar
+
+/(*nlb:foo)bar/
+ abcbarfoo
+\= Expect no match
+ abcfoobar
+
+/(*negative_lookbehind:foo)bar/
+ abcbarfoo
+\= Expect no match
+ abcfoobar
+
+/(?(*nlb:foo)bar|baz)/
+ abcfoobaz
+ abcbarbaz
+\= Expect no match
+ abcfoobar
+
+/(?(*negative_lookbehind:foo)bar|baz)/
+ abcfoobaz
+ abcbarbaz
+\= Expect no match
+ abcfoobar
+
+/(*atomic:a+)\w/
+ aaab
+\= Expect no match
+ aaaa
+
# End of testinput1
Modified: code/trunk/testdata/testinput2
===================================================================
--- code/trunk/testdata/testinput2 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/testdata/testinput2 2018-09-24 16:23:53 UTC (rev 1018)
@@ -5525,4 +5525,10 @@
\= Expect no match
abc\ndef\nxyz
+/(?(*ACCEPT)xxx)/
+
+/(?(*atomic:xx)xxx)/
+
+/(?(*script_run:xxx)zzz)/
+
# End of testinput2
Modified: code/trunk/testdata/testoutput1
===================================================================
--- code/trunk/testdata/testoutput1 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/testdata/testoutput1 2018-09-24 16:23:53 UTC (rev 1018)
@@ -9929,4 +9929,100 @@
AbCDe f
No match
+/(*pla:foo).{6}/
+ abcfoobarxyz
+ 0: foobar
+\= Expect no match
+ abcfooba
+No match
+
+/(*positive_lookahead:foo).{6}/
+ abcfoobarxyz
+ 0: foobar
+
+/(?(*pla:foo).{6}|a..)/
+ foobarbaz
+ 0: foobar
+ abcfoobar
+ 0: abc
+
+/(?(*positive_lookahead:foo).{6}|a..)/
+ foobarbaz
+ 0: foobar
+ abcfoobar
+ 0: abc
+
+/(*plb:foo)bar/
+ abcfoobar
+ 0: bar
+\= Expect no match
+ abcbarfoo
+No match
+
+/(*positive_lookbehind:foo)bar/
+ abcfoobar
+ 0: bar
+\= Expect no match
+ abcbarfoo
+No match
+
+/(?(*plb:foo)bar|baz)/
+ abcfoobar
+ 0: bar
+ bazfoobar
+ 0: baz
+ abcbazfoobar
+ 0: baz
+ foobazfoobar
+ 0: bar
+
+/(?(*positive_lookbehind:foo)bar|baz)/
+ abcfoobar
+ 0: bar
+ bazfoobar
+ 0: baz
+ abcbazfoobar
+ 0: baz
+ foobazfoobar
+ 0: bar
+
+/(*nlb:foo)bar/
+ abcbarfoo
+ 0: bar
+\= Expect no match
+ abcfoobar
+No match
+
+/(*negative_lookbehind:foo)bar/
+ abcbarfoo
+ 0: bar
+\= Expect no match
+ abcfoobar
+No match
+
+/(?(*nlb:foo)bar|baz)/
+ abcfoobaz
+ 0: baz
+ abcbarbaz
+ 0: bar
+\= Expect no match
+ abcfoobar
+No match
+
+/(?(*negative_lookbehind:foo)bar|baz)/
+ abcfoobaz
+ 0: baz
+ abcbarbaz
+ 0: bar
+\= Expect no match
+ abcfoobar
+No match
+
+/(*atomic:a+)\w/
+ aaab
+ 0: aaab
+\= Expect no match
+ aaaa
+No match
+
# End of testinput1
Modified: code/trunk/testdata/testoutput12-16
===================================================================
--- code/trunk/testdata/testoutput12-16 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/testdata/testoutput12-16 2018-09-24 16:23:53 UTC (rev 1018)
@@ -575,7 +575,7 @@
Subject length lower bound = 3
/(*CRLF)(*UTF32)(*BSR_UNICODE)a\Rb/I
-Failed: error 160 at offset 12: (*VERB) not recognized or malformed
+Failed: error 160 at offset 14: (*VERB) not recognized or malformed
/\h/I,utf
Capturing subpattern count = 0
Modified: code/trunk/testdata/testoutput12-32
===================================================================
--- code/trunk/testdata/testoutput12-32 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/testdata/testoutput12-32 2018-09-24 16:23:53 UTC (rev 1018)
@@ -538,7 +538,7 @@
Failed: error -27: UTF-32 error: code points 0xd800-0xdfff are not defined at offset 2
/(*UTF16)\x{11234}/
-Failed: error 160 at offset 5: (*VERB) not recognized or malformed
+Failed: error 160 at offset 7: (*VERB) not recognized or malformed
abcd\x{11234}pqr
/(*UTF)\x{11234}/I
@@ -559,7 +559,7 @@
abcd\x{11234}pqr
/(*CRLF)(*UTF16)(*BSR_UNICODE)a\Rb/I
-Failed: error 160 at offset 12: (*VERB) not recognized or malformed
+Failed: error 160 at offset 14: (*VERB) not recognized or malformed
/(*CRLF)(*UTF32)(*BSR_UNICODE)a\Rb/I
Capturing subpattern count = 0
Modified: code/trunk/testdata/testoutput2
===================================================================
--- code/trunk/testdata/testoutput2 2018-09-21 16:59:48 UTC (rev 1017)
+++ code/trunk/testdata/testoutput2 2018-09-24 16:23:53 UTC (rev 1018)
@@ -16812,6 +16812,15 @@
abc\ndef\nxyz
No match
+/(?(*ACCEPT)xxx)/
+Failed: error 128 at offset 2: assertion expected after (?( or (?(?C)
+
+/(?(*atomic:xx)xxx)/
+Failed: error 128 at offset 10: assertion expected after (?( or (?(?C)
+
+/(?(*script_run:xxx)zzz)/
+Failed: error 128 at offset 14: assertion expected after (?( or (?(?C)
+
# End of testinput2
Error -70: PCRE2_ERROR_BADDATA (unknown error number)
Error -62: bad serialized data
