Revision: 859
http://vcs.pcre.org/viewvc?view=rev&revision=859
Author: ph10
Date: 2012-01-09 17:43:54 +0000 (Mon, 09 Jan 2012)
Log Message:
-----------
Documentation.
Modified Paths:
--------------
code/trunk/doc/pcre16.3
code/trunk/doc/pcreapi.3
code/trunk/doc/pcrecallout.3
code/trunk/doc/pcrejit.3
code/trunk/doc/pcrepattern.3
code/trunk/doc/pcreperform.3
code/trunk/doc/pcreposix.3
Modified: code/trunk/doc/pcre16.3
===================================================================
--- code/trunk/doc/pcre16.3 2012-01-08 17:55:38 UTC (rev 858)
+++ code/trunk/doc/pcre16.3 2012-01-09 17:43:54 UTC (rev 859)
@@ -53,11 +53,11 @@
.ti +5n
.B int \fIstringcount\fP, PCRE_SPTR16 \fIstringname\fP,
.ti +5n
-.B PCRE_SCHAR16 *\fIbuffer\fP, int \fIbuffersize\fP);
+.B PCRE_UCHAR16 *\fIbuffer\fP, int \fIbuffersize\fP);
.PP
.B int pcre16_copy_substring(PCRE_SPTR16 \fIsubject\fP, int *\fIovector\fP,
.ti +5n
-.B int \fIstringcount\fP, int \fIstringnumber\fP, PCRE_SCHAR16 *\fIbuffer\fP,
+.B int \fIstringcount\fP, int \fIstringnumber\fP, PCRE_UCHAR16 *\fIbuffer\fP,
.ti +5n
.B int \fIbuffersize\fP);
.PP
@@ -75,7 +75,7 @@
.PP
.B int pcre16_get_stringtable_entries(const pcre16 *\fIcode\fP,
.ti +5n
-.B PCRE_SPTR16 \fIname\fP, PCRE_SCHAR16 **\fIfirst\fP, PCRE_SCHAR16 **\fIlast\fP);
+.B PCRE_SPTR16 \fIname\fP, PCRE_UCHAR16 **\fIfirst\fP, PCRE_UCHAR16 **\fIlast\fP);
.PP
.B int pcre16_get_substring(PCRE_SPTR16 \fIsubject\fP, int *\fIovector\fP,
.ti +5n
@@ -137,7 +137,7 @@
.SH "PCRE 16-BIT API 16-BIT-ONLY FUNCTION"
.rs
.sp
-.B int pcre16_utf16_to_host_byte_order(PCRE_SCHAR16 *\fIoutput\fP,
+.B int pcre16_utf16_to_host_byte_order(PCRE_UCHAR16 *\fIoutput\fP,
.ti +5n
.B PCRE_SPTR16 \fIinput\fP, int \fIlength\fP, int *\fIbyte_order\fP,
.ti +5n
@@ -187,10 +187,10 @@
.sp
In the 8-bit library, strings are passed to PCRE library functions as vectors
of bytes with the C type "char *". In the 16-bit library, strings are passed as
-vectors of unsigned 16-bit quantities. The macro PCRE_SCHAR16 specifies an
-appropriate data type, and PCRE_SPTR16 is defined as "const PCRE_SCHAR16 *". In
+vectors of unsigned 16-bit quantities. The macro PCRE_UCHAR16 specifies an
+appropriate data type, and PCRE_SPTR16 is defined as "const PCRE_UCHAR16 *". In
very many environments, "short int" is a 16-bit data type. When PCRE is built,
-it defines PCRE_SCHAR16 as "short int", but checks that it really is a 16-bit
+it defines PCRE_UCHAR16 as "short int", but checks that it really is a 16-bit
data type. If it is not, the build fails with an error message telling the
maintainer to modify the definition appropriately.
.
@@ -224,7 +224,7 @@
conversion in place is supported. The output buffer must be at least as long as
the input.
.P
-The \fIlength\fP argument of specifies the number of 16-bit data units in the
+The \fIlength\fP argument specifies the number of 16-bit data units in the
input string; a negative value specifies a zero-terminated string.
.P
If \fIbyte_order\fP is NULL, it is assumed that the string starts off in host
Modified: code/trunk/doc/pcreapi.3
===================================================================
--- code/trunk/doc/pcreapi.3 2012-01-08 17:55:38 UTC (rev 858)
+++ code/trunk/doc/pcreapi.3 2012-01-09 17:43:54 UTC (rev 859)
@@ -1453,7 +1453,7 @@
unsigned char **\fImark\fP;
.sp
In the 16-bit version of this structure, the \fImark\fP field has type
-"PCRE_SCHAR16 **".
+"PCRE_UCHAR16 **".
.P
The \fIflags\fP field is a bitmap that specifies which of the other fields
are set. The flag bits are:
Modified: code/trunk/doc/pcrecallout.3
===================================================================
--- code/trunk/doc/pcrecallout.3 2012-01-08 17:55:38 UTC (rev 858)
+++ code/trunk/doc/pcrecallout.3 2012-01-09 17:43:54 UTC (rev 859)
@@ -95,7 +95,7 @@
int \fIpattern_position\fP;
int \fInext_item_length\fP;
const unsigned char *\fImark\fP; (8-bit version)
- const PCRE_SCHAR16 *\fImark\fP; (16-bit version)
+ const PCRE_UCHAR16 *\fImark\fP; (16-bit version)
.sp
The \fIversion\fP field is an integer containing the version number of the
block format. The initial version was 0; the current version is 2. The version
Modified: code/trunk/doc/pcrejit.3
===================================================================
--- code/trunk/doc/pcrejit.3 2012-01-08 17:55:38 UTC (rev 858)
+++ code/trunk/doc/pcrejit.3 2012-01-09 17:43:54 UTC (rev 859)
@@ -38,7 +38,7 @@
ARM v5, v7, and Thumb2
Intel x86 32-bit and 64-bit
MIPS 32-bit
- Power PC 32-bit and 64-bit (experimental)
+ Power PC 32-bit and 64-bit
.sp
The Power PC support is designated as experimental because it has not been
fully tested. If --enable-jit is set on an unsupported platform, compilation
Modified: code/trunk/doc/pcrepattern.3
===================================================================
--- code/trunk/doc/pcrepattern.3 2012-01-08 17:55:38 UTC (rev 858)
+++ code/trunk/doc/pcrepattern.3 2012-01-09 17:43:54 UTC (rev 859)
@@ -21,24 +21,27 @@
description of PCRE's regular expressions is intended as reference material.
.P
The original operation of PCRE was on strings of one-byte characters. However,
-there is now also support for UTF-8 character strings. To use this,
-PCRE must be built to include UTF-8 support, and you must call
-\fBpcre_compile()\fP or \fBpcre_compile2()\fP with the PCRE_UTF8 option. There
-is also a special sequence that can be given at the start of a pattern:
+there is now also support for UTF-8 strings in the original library, and a
+second library that supports 16-bit and UTF-16 character strings. To use these
+features, PCRE must be built to include appropriate support. When using UTF
+strings you must either call the compiling function with the PCRE_UTF8 or
+PCRE_UTF16 option, or the pattern must start with one of these special
+sequences:
.sp
(*UTF8)
+ (*UTF16)
.sp
-Starting a pattern with this sequence is equivalent to setting the PCRE_UTF8
-option. This feature is not Perl-compatible. How setting UTF-8 mode affects
+Starting a pattern with such a sequence is equivalent to setting the relevant
+option. This feature is not Perl-compatible. How setting a UTF mode affects
pattern matching is mentioned in several places below. There is also a summary
-of UTF-8 features in the
+of features in the
.\" HREF
\fBpcreunicode\fP
.\"
page.
.P
Another special sequence that may appear at the start of a pattern or in
-combination with (*UTF8) is:
+combination with (*UTF8) or (*UTF16) is:
.sp
(*UCP)
.sp
@@ -53,13 +56,13 @@
of newlines; they are described below.
.P
The remainder of this document discusses the patterns that are supported by
-PCRE when its main matching function, \fBpcre_exec()\fP, is used.
-From release 6.0, PCRE offers a second matching function,
-\fBpcre_dfa_exec()\fP, which matches using a different algorithm that is not
-Perl-compatible. Some of the features discussed below are not available when
-\fBpcre_dfa_exec()\fP is used. The advantages and disadvantages of the
-alternative function, and how it differs from the normal function, are
-discussed in the
+PCRE when one its main matching functions, \fBpcre_exec()\fP (8-bit) or
+\fBpcre16_exec()\fP (16-bit), is used. PCRE also has alternative matching
+functions, \fBpcre_dfa_exec()\fP and \fBpcre16_dfa_exec()\fP, which match using
+a different algorithm that is not Perl-compatible. Some of the features
+discussed below are not available when DFA matching is used. The advantages and
+disadvantages of the alternative functions, and how they differ from the normal
+functions, are discussed in the
.\" HREF
\fBpcrematching\fP
.\"
@@ -94,9 +97,8 @@
(*ANYCRLF) any of the three above
(*ANY) all Unicode newline sequences
.sp
-These override the default and the options given to \fBpcre_compile()\fP or
-\fBpcre_compile2()\fP. For example, on a Unix system where LF is the default
-newline sequence, the pattern
+These override the default and the options given to the compiling function. For
+example, on a Unix system where LF is the default newline sequence, the pattern
.sp
(*CR)a.b
.sp
@@ -130,13 +132,13 @@
.sp
matches a portion of a subject string that is identical to itself. When
caseless matching is specified (the PCRE_CASELESS option), letters are matched
-independently of case. In UTF-8 mode, PCRE always understands the concept of
+independently of case. In a UTF mode, PCRE always understands the concept of
case for characters whose values are less than 128, so caseless matching is
always possible. For characters with higher values, the concept of case is
supported if PCRE is compiled with Unicode property support, but not otherwise.
If you want to use caseless matching for characters 128 and above, you must
ensure that PCRE is compiled with Unicode property support as well as with
-UTF-8 support.
+UTF support.
.P
The power of regular expressions comes from the ability to include alternatives
and repetitions in the pattern. These are encoded in the pattern by the use of
@@ -192,7 +194,7 @@
non-alphanumeric with backslash to specify that it stands for itself. In
particular, if you want to match a backslash, you write \e\e.
.P
-In UTF-8 mode, only ASCII numbers and letters have any special meaning after a
+In a UTF mode, only ASCII numbers and letters have any special meaning after a
backslash. All other characters (in particular, those whose codepoints are
greater than 127) are treated as literals.
.P
@@ -249,16 +251,21 @@
Thus \ecz becomes hex 1A (z is 7A), but \ec{ becomes hex 3B ({ is 7B), while
\ec; becomes hex 7B (; is 3B). If the byte following \ec has a value greater
than 127, a compile-time error occurs. This locks out non-ASCII characters in
-both byte mode and UTF-8 mode. (When PCRE is compiled in EBCDIC mode, all byte
-values are valid. A lower case letter is converted to upper case, and then the
-0xc0 bits are flipped.)
+all modes. (When PCRE is compiled in EBCDIC mode, all byte values are valid. A
+lower case letter is converted to upper case, and then the 0xc0 bits are
+flipped.)
.P
By default, after \ex, from zero to two hexadecimal digits are read (letters
can be in upper or lower case). Any number of hexadecimal digits may appear
-between \ex{ and }, but the value of the character code must be less than 256
-in non-UTF-8 mode, and less than 2**31 in UTF-8 mode. That is, the maximum
-value in hexadecimal is 7FFFFFFF. Note that this is bigger than the largest
-Unicode code point, which is 10FFFF.
+between \ex{ and }, but the character code is constrained as follows:
+.sp
+ 8-bit non-UTF mode less than 0x100
+ 8-bit UTF-8 mode less than 0x10ffff and a valid codepoint
+ 16-bit non-UTF mode less than 0x10000
+ 16-bit UTF-16 mode less than 0x10ffff and a valid codepoint
+.sp
+Invalid Unicode codepoints are the range 0xd800 to 0xdfff (the so-called
+"surrogate" codepoints).
.P
If characters other than hexadecimal digits appear between \ex{ and }, or if
there is no terminating }, this form of escape is not recognized. Instead, the
@@ -300,9 +307,9 @@
Inside a character class, or if the decimal number is greater than 9 and there
have not been that many capturing subpatterns, PCRE re-reads up to three octal
digits following the backslash, and uses them to generate a data character. Any
-subsequent digits stand for themselves. In non-UTF-8 mode, the value of a
-character specified in octal must be less than \e400. In UTF-8 mode, values up
-to \e777 are permitted. For example:
+subsequent digits stand for themselves. The value of the character is
+constrained in the same way as characters specified in hexadecimal.
+For example:
.sp
\e040 is another way of writing a space
.\" JOIN
@@ -319,7 +326,7 @@
character with octal code 113
.\" JOIN
\e377 might be a back reference, otherwise
- the byte consisting entirely of 1 bits
+ the value 255 (decimal)
.\" JOIN
\e81 is either a back reference, or a binary zero
followed by the two characters "8" and "1"
@@ -440,9 +447,9 @@
accented letters, and these are then matched by \ew. The use of locales with
Unicode is discouraged.
.P
-By default, in UTF-8 mode, characters with values greater than 128 never match
+By default, in a UTF mode, characters with values greater than 128 never match
\ed, \es, or \ew, and always match \eD, \eS, and \eW. These sequences retain
-their original meanings from before UTF-8 support was available, mainly for
+their original meanings from before UTF support was available, mainly for
efficiency reasons. However, if PCRE is compiled with Unicode property support,
and the PCRE_UCP option is set, the behaviour is changed so that Unicode
properties are used to determine character types, as follows:
@@ -459,9 +466,8 @@
.P
The sequences \eh, \eH, \ev, and \eV are features that were added to Perl at
release 5.10. In contrast to the other sequences, which match only ASCII
-characters by default, these always match certain high-valued codepoints in
-UTF-8 mode, whether or not PCRE_UCP is set. The horizontal space characters
-are:
+characters by default, these always match certain high-valued codepoints,
+whether or not PCRE_UCP is set. The horizontal space characters are:
.sp
U+0009 Horizontal tab
U+0020 Space
@@ -492,6 +498,9 @@
U+0085 Next line
U+2028 Line separator
U+2029 Paragraph separator
+.sp
+In 8-bit, non-UTF-8 mode, only the characters with codepoints less than 256 are
+relevant.
.
.
.\" HTML <a name="newlineseq"></a>
@@ -499,7 +508,8 @@
.rs
.sp
Outside a character class, by default, the escape sequence \eR matches any
-Unicode newline sequence. In non-UTF-8 mode \eR is equivalent to the following:
+Unicode newline sequence. In 8-bit non-UTF-8 mode \eR is equivalent to the
+following:
.sp
(?>\er\en|\en|\ex0b|\ef|\er|\ex85)
.sp
@@ -514,7 +524,7 @@
line, U+0085). The two-character sequence is treated as a single unit that
cannot be split.
.P
-In UTF-8 mode, two additional characters whose codepoints are greater than 255
+In other modes, two additional characters whose codepoints are greater than 255
are added: LS (line separator, U+2028) and PS (paragraph separator, U+2029).
Unicode character property support is not needed for these characters to be
recognized.
@@ -530,19 +540,19 @@
(*BSR_ANYCRLF) CR, LF, or CRLF only
(*BSR_UNICODE) any Unicode newline sequence
.sp
-These override the default and the options given to \fBpcre_compile()\fP or
-\fBpcre_compile2()\fP, but they can be overridden by options given to
-\fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP. Note that these special settings,
-which are not Perl-compatible, are recognized only at the very start of a
-pattern, and that they must be in upper case. If more than one of them is
-present, the last one is used. They can be combined with a change of newline
-convention; for example, a pattern can start with:
+These override the default and the options given to the compiling function, but
+they can themselves be overridden by options given to a matching function. Note
+that these special settings, which are not Perl-compatible, are recognized only
+at the very start of a pattern, and that they must be in upper case. If more
+than one of them is present, the last one is used. They can be combined with a
+change of newline convention; for example, a pattern can start with:
.sp
(*ANY)(*BSR_ANYCRLF)
.sp
-They can also be combined with the (*UTF8) or (*UCP) special sequences. Inside
-a character class, \eR is treated as an unrecognized escape sequence, and so
-matches the letter "R" by default, but causes an error if PCRE_EXTRA is set.
+They can also be combined with the (*UTF8), (*UTF16), or (*UCP) special
+sequences. Inside a character class, \eR is treated as an unrecognized escape
+sequence, and so matches the letter "R" by default, but causes an error if
+PCRE_EXTRA is set.
.
.
.\" HTML <a name="uniextseq"></a>
@@ -551,7 +561,7 @@
.sp
When PCRE is built with Unicode character property support, three additional
escape sequences that match characters with specific properties are available.
-When not in UTF-8 mode, these sequences are of course limited to testing
+When in 8-bit non-UTF-8 mode, these sequences are of course limited to testing
characters whose codepoints are less than 256, but they do work in this mode.
The extra escape sequences are:
.sp
@@ -738,9 +748,9 @@
a modifier or "other".
.P
The Cs (Surrogate) property applies only to characters in the range U+D800 to
-U+DFFF. Such characters are not valid in UTF-8 strings (see RFC 3629) and so
-cannot be tested by PCRE, unless UTF-8 validity checking has been turned off
-(see the discussion of PCRE_NO_UTF8_CHECK in the
+U+DFFF. Such characters are not valid in Unicode strings and so
+cannot be tested by PCRE, unless UTF validity checking has been turned off
+(see the discussion of PCRE_NO_UTF8_CHECK and PCRE_NO_UTF16_CHECK in the
.\" HREF
\fBpcreapi\fP
.\"
@@ -771,7 +781,7 @@
.\"
Characters with the "mark" property are typically accents that affect the
preceding character. None of them have codepoints less than 256, so in
-non-UTF-8 mode \eX matches any one character.
+8-bit non-UTF-8 mode \eX matches any one character.
.P
Note that recent versions of Perl have changed \eX to match what Unicode calls
an "extended grapheme cluster", which has a more complicated definition.
@@ -780,8 +790,7 @@
a structure that contains data for over fifteen thousand characters. That is
why the traditional escape sequences such as \ed and \ew do not use Unicode
properties in PCRE by default, though you can make them do so by setting the
-PCRE_UCP option for \fBpcre_compile()\fP or by starting the pattern with
-(*UCP).
+PCRE_UCP option or by starting the pattern with (*UCP).
.
.
.\" HTML <a name="extraprops"></a>
@@ -870,7 +879,7 @@
A word boundary is a position in the subject string where the current character
and the previous character do not both match \ew or \eW (i.e. one matches
\ew and the other matches \eW), or the start or end of the string if the
-first or last character matches \ew, respectively. In UTF-8 mode, the meanings
+first or last character matches \ew, respectively. In a UTF mode, the meanings
of \ew and \eW can be changed by setting the PCRE_UCP option. When this is
done, it also affects \eb and \eB. Neither PCRE nor Perl has a separate "start
of word" or "end of word" metasequence. However, whatever follows \eb normally
@@ -965,7 +974,7 @@
.sp
Outside a character class, a dot in the pattern matches any one character in
the subject string except (by default) a character that signifies the end of a
-line. In UTF-8 mode, the matched character may be more than one byte long.
+line.
.P
When a line ending is defined as a single character, dot never matches that
character; when the two-character sequence CRLF is used, dot does not match CR
@@ -989,31 +998,32 @@
name; PCRE does not support this.
.
.
-.SH "MATCHING A SINGLE BYTE"
+.SH "MATCHING A SINGLE DATA UNIT"
.rs
.sp
-Outside a character class, the escape sequence \eC matches any one byte, both
-in and out of UTF-8 mode. Unlike a dot, it always matches line-ending
-characters. The feature is provided in Perl in order to match individual bytes
-in UTF-8 mode, but it is unclear how it can usefully be used. Because \eC
-breaks up characters into individual bytes, matching one byte with \eC in UTF-8
-mode means that the rest of the string may start with a malformed UTF-8
-character. This has undefined results, because PCRE assumes that it is dealing
-with valid UTF-8 strings (and by default it checks this at the start of
-processing unless the PCRE_NO_UTF8_CHECK option is used).
+Outside a character class, the escape sequence \eC matches any one data unit,
+whether or not a UTF mode is set. In the 8-bit library, one data unit is one
+byte; in the 16-bit library it is a 16-bit unit. Unlike a dot, \eC always
+matches line-ending characters. The feature is provided in Perl in order to
+match individual bytes in UTF-8 mode, but it is unclear how it can usefully be
+used. Because \eC breaks up characters into individual data units, matching one
+unit with \eC in a UTF mode means that the rest of the string may start with a
+malformed UTF character. This has undefined results, because PCRE assumes that
+it is dealing with valid UTF strings (and by default it checks this at the
+start of processing unless the PCRE_NO_UTF8_CHECK option is used).
.P
PCRE does not allow \eC to appear in lookbehind assertions
.\" HTML <a href="#lookbehind">
.\" </a>
(described below)
.\"
-in UTF-8 mode, because this would make it impossible to calculate the length of
+in a UTF mode, because this would make it impossible to calculate the length of
the lookbehind.
.P
-In general, the \eC escape sequence is best avoided in UTF-8 mode. However, one
-way of using it that avoids the problem of malformed UTF-8 characters is to
-use a lookahead to check the length of the next character, as in this pattern
-(ignore white space and line breaks):
+In general, the \eC escape sequence is best avoided. However, one
+way of using it that avoids the problem of malformed UTF characters is to use a
+lookahead to check the length of the next character, as in this pattern, which
+could be used with a UTF-8 string (ignore white space and line breaks):
.sp
(?| (?=[\ex00-\ex7f])(\eC) |
(?=[\ex80-\ex{7ff}])(\eC)(\eC) |
@@ -1043,12 +1053,12 @@
a member of the class, it should be the first data character in the class
(after an initial circumflex, if present) or escaped with a backslash.
.P
-A character class matches a single character in the subject. In UTF-8 mode, the
-character may be more than one byte long. A matched character must be in the
-set of characters defined by the class, unless the first character in the class
-definition is a circumflex, in which case the subject character must not be in
-the set defined by the class. If a circumflex is actually required as a member
-of the class, ensure it is not the first character, or escape it with a
+A character class matches a single character in the subject. In a UTF mode, the
+character may be more than one data unit long. A matched character must be in
+the set of characters defined by the class, unless the first character in the
+class definition is a circumflex, in which case the subject character must not
+be in the set defined by the class. If a circumflex is actually required as a
+member of the class, ensure it is not the first character, or escape it with a
backslash.
.P
For example, the character class [aeiou] matches any lower case vowel, while
@@ -1059,19 +1069,20 @@
string, and therefore it fails if the current pointer is at the end of the
string.
.P
-In UTF-8 mode, characters with values greater than 255 can be included in a
-class as a literal string of bytes, or by using the \ex{ escaping mechanism.
+In UTF-8 (UTF-16) mode, characters with values greater than 255 (0xffff) can be
+included in a class as a literal string of data units, or by using the \ex{
+escaping mechanism.
.P
When caseless matching is set, any letters in a class represent both their
upper case and lower case versions, so for example, a caseless [aeiou] matches
"A" as well as "a", and a caseless [^aeiou] does not match "A", whereas a
-caseful version would. In UTF-8 mode, PCRE always understands the concept of
+caseful version would. In a UTF mode, PCRE always understands the concept of
case for characters whose values are less than 128, so caseless matching is
always possible. For characters with higher values, the concept of case is
supported if PCRE is compiled with Unicode property support, but not otherwise.
-If you want to use caseless matching in UTF8-mode for characters 128 and above,
-you must ensure that PCRE is compiled with Unicode property support as well as
-with UTF-8 support.
+If you want to use caseless matching in a UTF mode for characters 128 and
+above, you must ensure that PCRE is compiled with Unicode property support as
+well as with UTF support.
.P
Characters that might indicate line breaks are never treated in any special way
when matching character classes, whatever line-ending sequence is in use, and
@@ -1093,22 +1104,21 @@
"]" can also be used to end a range.
.P
Ranges operate in the collating sequence of character values. They can also be
-used for characters specified numerically, for example [\e000-\e037]. In UTF-8
-mode, ranges can include characters whose values are greater than 255, for
-example [\ex{100}-\ex{2ff}].
+used for characters specified numerically, for example [\e000-\e037]. Ranges
+can include any characters that are valid for the current mode.
.P
If a range that includes letters is used when caseless matching is set, it
matches the letters in either case. For example, [W-c] is equivalent to
-[][\e\e^_`wxyzabc], matched caselessly, and in non-UTF-8 mode, if character
+[][\e\e^_`wxyzabc], matched caselessly, and in a non-UTF mode, if character
tables for a French locale are in use, [\exc8-\excb] matches accented E
-characters in both cases. In UTF-8 mode, PCRE supports the concept of case for
+characters in both cases. In UTF modes, PCRE supports the concept of case for
characters with values greater than 128 only when it is compiled with Unicode
property support.
.P
The character escape sequences \ed, \eD, \eh, \eH, \ep, \eP, \es, \eS, \ev,
\eV, \ew, and \eW may appear in a character class, and add the characters that
they match to the class. For example, [\edABCDEF] matches any hexadecimal
-digit. In UTF-8 mode, the PCRE_UCP option affects the meanings of \ed, \es, \ew
+digit. In UTF modes, the PCRE_UCP option affects the meanings of \ed, \es, \ew
and their upper case partners, just as it does when they appear outside a
character class, as described in the section entitled
.\" HTML <a href="#genericchartypes">
@@ -1178,7 +1188,7 @@
syntax [.ch.] and [=ch=] where "ch" is a "collating element", but these are not
supported, and an error is given if they are encountered.
.P
-By default, in UTF-8 mode, characters with values greater than 128 do not match
+By default, in UTF modes, characters with values greater than 128 do not match
any of the POSIX character classes. However, if the PCRE_UCP option is passed
to \fBpcre_compile()\fP, some of the classes are changed so that Unicode
character properties are used. This is achieved by replacing the POSIX classes
@@ -1266,17 +1276,17 @@
behaviour otherwise.
.P
\fBNote:\fP There are other PCRE-specific options that can be set by the
-application when the compile or match functions are called. In some cases the
-pattern can contain special leading sequences such as (*CRLF) to override what
-the application has set or what has been defaulted. Details are given in the
-section entitled
+application when the compiling or matching functions are called. In some cases
+the pattern can contain special leading sequences such as (*CRLF) to override
+what the application has set or what has been defaulted. Details are given in
+the section entitled
.\" HTML <a href="#newlineseq">
.\" </a>
"Newline sequences"
.\"
-above. There are also the (*UTF8) and (*UCP) leading sequences that can be used
-to set UTF-8 and Unicode property modes; they are equivalent to setting the
-PCRE_UTF8 and the PCRE_UCP options, respectively.
+above. There are also the (*UTF8), (*UTF16), and (*UCP) leading sequences that
+can be used to set UTF and Unicode property modes; they are equivalent to
+setting the PCRE_UTF8, PCRE_UTF16, and the PCRE_UCP options, respectively.
.
.
.\" HTML <a name="subpattern"></a>
@@ -1295,10 +1305,13 @@
.sp
2. It sets up the subpattern as a capturing subpattern. This means that, when
the whole pattern matches, that portion of the subject string that matched the
-subpattern is passed back to the caller via the \fIovector\fP argument of
-\fBpcre_exec()\fP. Opening parentheses are counted from left to right (starting
-from 1) to obtain numbers for the capturing subpatterns. For example, if the
-string "the red king" is matched against the pattern
+subpattern is passed back to the caller via the \fIovector\fP argument of the
+matching function. (This applies only to the traditional matching functions;
+the DFA matching functions do not support capturing.)
+.P
+Opening parentheses are counted from left to right (starting from 1) to obtain
+numbers for the capturing subpatterns. For example, if the string "the red
+king" is matched against the pattern
.sp
the ((red|white) (king|queen))
.sp
@@ -1474,7 +1487,7 @@
a literal data character
the dot metacharacter
the \eC escape sequence
- the \eX escape sequence (in UTF-8 mode with Unicode properties)
+ the \eX escape sequence
the \eR escape sequence
an escape such as \ed or \epL that matches a single character
a character class
@@ -1505,11 +1518,11 @@
quantifier, is taken as a literal character. For example, {,6} is not a
quantifier, but a literal string of four characters.
.P
-In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to individual
-bytes. Thus, for example, \ex{100}{2} matches two UTF-8 characters, each of
-which is represented by a two-byte sequence. Similarly, when Unicode property
-support is available, \eX{3} matches three Unicode extended sequences, each of
-which may be several bytes long (and they may be of different lengths).
+In UTF modes, quantifiers apply to characters rather than to individual data
+units. Thus, for example, \ex{100}{2} matches two characters, each of
+which is represented by a two-byte sequence in a UTF-8 string. Similarly,
+\eX{3} matches three Unicode extended sequences, each of which may be several
+data units long (and they may be of different lengths).
.P
The quantifier {0} is permitted, causing the expression to behave as if the
previous item and the quantifier were not present. This may be useful for
@@ -1972,10 +1985,11 @@
match. If there are insufficient characters before the current position, the
assertion fails.
.P
-In UTF-8 mode, PCRE does not allow the \eC escape (which matches a single byte,
-even in UTF-8 mode) to appear in lookbehind assertions, because it makes it
-impossible to calculate the length of the lookbehind. The \eX and \eR escapes,
-which can match different numbers of bytes, are also not permitted.
+In a UTF mode, PCRE does not allow the \eC escape (which matches a single data
+unit even in a UTF mode) to appear in lookbehind assertions, because it makes
+it impossible to calculate the length of the lookbehind. The \eX and \eR
+escapes, which can match different numbers of data units, are also not
+permitted.
.P
.\" HTML <a href="#subpatternsassubroutines">
.\" </a>
@@ -2222,7 +2236,7 @@
option is set, an unescaped # character also introduces a comment, which in
this case continues to immediately after the next newline character or
character sequence in the pattern. Which characters are interpreted as newlines
-is controlled by the options passed to \fBpcre_compile()\fP or by a special
+is controlled by the options passed to a compiling function or by a special
sequence at the start of the pattern, as described in the section entitled
.\" HTML <a href="#newlines">
.\" </a>
@@ -2524,8 +2538,9 @@
.P
PCRE provides a similar feature, but of course it cannot obey arbitrary Perl
code. The feature is called "callout". The caller of PCRE provides an external
-function by putting its entry point in the global variable \fIpcre_callout\fP.
-By default, this variable contains NULL, which disables all calling out.
+function by putting its entry point in the global variable \fIpcre_callout\fP
+(8-bit library) or \fIpcre16_callout\fP (16-bit library). By default, this
+variable contains NULL, which disables all calling out.
.P
Within a regular expression, (?C) indicates the points at which the external
function is to be called. If you want to identify different callout points, you
@@ -2534,16 +2549,16 @@
.sp
(?C1)abc(?C2)def
.sp
-If the PCRE_AUTO_CALLOUT flag is passed to \fBpcre_compile()\fP, callouts are
+If the PCRE_AUTO_CALLOUT flag is passed to a compiling function, callouts are
automatically installed before each item in the pattern. They are all numbered
255.
.P
-During matching, when PCRE reaches a callout point (and \fIpcre_callout\fP is
-set), the external function is called. It is provided with the number of the
-callout, the position in the pattern, and, optionally, one item of data
-originally supplied by the caller of \fBpcre_exec()\fP. The callout function
-may cause matching to proceed, to backtrack, or to fail altogether. A complete
-description of the interface to the callout function is given in the
+During matching, when PCRE reaches a callout point, the external function is
+called. It is provided with the number of the callout, the position in the
+pattern, and, optionally, one item of data originally supplied by the caller of
+the matching function. The callout function may cause matching to proceed, to
+backtrack, or to fail altogether. A complete description of the interface to
+the callout function is given in the
.\" HREF
\fBpcrecallout\fP
.\"
@@ -2561,10 +2576,10 @@
remarks apply to the PCRE features described in this section.
.P
Since these verbs are specifically related to backtracking, most of them can be
-used only when the pattern is to be matched using \fBpcre_exec()\fP, which uses
-a backtracking algorithm. With the exception of (*FAIL), which behaves like a
-failing negative assertion, they cause an error if encountered by
-\fBpcre_dfa_exec()\fP.
+used only when the pattern is to be matched using one of the traditional
+matching functions, which use a backtracking algorithm. With the exception of
+(*FAIL), which behaves like a failing negative assertion, they cause an error
+if encountered by a DFA matching function.
.P
If any of these verbs are used in an assertion or in a subpattern that is
called as a subroutine (whether or not recursively), their effect is confined
@@ -2643,11 +2658,10 @@
(*MARK) as you like in a pattern, and their names do not have to be unique.
.P
When a match succeeds, the name of the last-encountered (*MARK) on the matching
-path is passed back to the caller via the \fIpcre_extra\fP data structure, as
-described in the
+path is passed back to the caller as described in the section entitled
.\" HTML <a href="pcreapi.html#extradata">
.\" </a>
-section on \fIpcre_extra\fP
+"Extra data for \fBpcre_exec()\fP"
.\"
in the
.\" HREF
@@ -2845,7 +2859,7 @@
.rs
.sp
\fBpcreapi\fP(3), \fBpcrecallout\fP(3), \fBpcrematching\fP(3),
-\fBpcresyntax\fP(3), \fBpcre\fP(3).
+\fBpcresyntax\fP(3), \fBpcre\fP(3), \fBpcre16(3)\fP.
.
.
.SH AUTHOR
@@ -2862,6 +2876,6 @@
.rs
.sp
.nf
-Last updated: 29 November 2011
-Copyright (c) 1997-2011 University of Cambridge.
+Last updated: 09 January 2012
+Copyright (c) 1997-2012 University of Cambridge.
.fi
Modified: code/trunk/doc/pcreperform.3
===================================================================
--- code/trunk/doc/pcreperform.3 2012-01-08 17:55:38 UTC (rev 858)
+++ code/trunk/doc/pcreperform.3 2012-01-09 17:43:54 UTC (rev 859)
@@ -11,9 +11,9 @@
.SH "COMPILED PATTERN MEMORY USAGE"
.rs
.sp
-Patterns are compiled by PCRE into a reasonably efficient byte code, so that
-most simple patterns do not use much memory. However, there is one case where
-the memory usage of a compiled pattern can be unexpectedly large. If a
+Patterns are compiled by PCRE into a reasonably efficient interpretive code, so
+that most simple patterns do not use much memory. However, there is one case
+where the memory usage of a compiled pattern can be unexpectedly large. If a
parenthesized subpattern has a quantifier with a minimum greater than 1 and/or
a limited maximum, the whole subpattern is repeated in the compiled code. For
example, the pattern
@@ -34,12 +34,12 @@
.sp
((ab){1,1000}c){1,3}
.sp
-uses 51K bytes when compiled. When PCRE is compiled with its default internal
-pointer size of two bytes, the size limit on a compiled pattern is 64K, and
-this is reached with the above pattern if the outer repetition is increased
-from 3 to 4. PCRE can be compiled to use larger internal pointers and thus
-handle larger compiled patterns, but it is better to try to rewrite your
-pattern to use less memory if you can.
+uses 51K bytes when compiled using the 8-bit library. When PCRE is compiled
+with its default internal pointer size of two bytes, the size limit on a
+compiled pattern is 64K data units, and this is reached with the above pattern
+if the outer repetition is increased from 3 to 4. PCRE can be compiled to use
+larger internal pointers and thus handle larger compiled patterns, but it is
+better to try to rewrite your pattern to use less memory if you can.
.P
One way of reducing the memory usage for such patterns is to make use of PCRE's
.\" HTML <a href="pcrepattern.html#subpatternsassubroutines">
@@ -68,11 +68,11 @@
.SH "STACK USAGE AT RUN TIME"
.rs
.sp
-When \fBpcre_exec()\fP is used for matching, certain kinds of pattern can cause
-it to use large amounts of the process stack. In some environments the default
-process stack is quite small, and if it runs out the result is often SIGSEGV.
-This issue is probably the most frequently raised problem with PCRE. Rewriting
-your pattern can often help. The
+When \fBpcre_exec()\fP or \fBpcre16_exec()\fP is used for matching, certain
+kinds of pattern can cause it to use large amounts of the process stack. In
+some environments the default process stack is quite small, and if it runs out
+the result is often SIGSEGV. This issue is probably the most frequently raised
+problem with PCRE. Rewriting your pattern can often help. The
.\" HREF
\fBpcrestack\fP
.\"
@@ -101,8 +101,9 @@
backwards compatibility, and partly for performance reasons. However, you can
set PCRE_UCP if you want Unicode character properties to be used. This can
double the matching time for items such as \ed, when matched with
-\fBpcre_exec()\fP; the performance loss is less with \fBpcre_dfa_exec()\fP, and
-in both cases there is not much difference for \eb.
+a traditional matching function; the performance loss is less with
+a DFA matching function, and in both cases there is not much difference for
+\eb.
.P
When a pattern begins with .* not in parentheses, or in parentheses that are
not the subject of a backreference, and the PCRE_DOTALL option is set, the
@@ -172,6 +173,6 @@
.rs
.sp
.nf
-Last updated: 16 May 2010
-Copyright (c) 1997-2010 University of Cambridge.
+Last updated: 09 January 2012
+Copyright (c) 1997-2012 University of Cambridge.
.fi
Modified: code/trunk/doc/pcreposix.3
===================================================================
--- code/trunk/doc/pcreposix.3 2012-01-08 17:55:38 UTC (rev 858)
+++ code/trunk/doc/pcreposix.3 2012-01-09 17:43:54 UTC (rev 859)
@@ -24,13 +24,14 @@
.SH DESCRIPTION
.rs
.sp
-This set of functions provides a POSIX-style API to the PCRE regular expression
-package. See the
+This set of functions provides a POSIX-style API for the PCRE regular
+expression 8-bit library. See the
.\" HREF
\fBpcreapi\fP
.\"
documentation for a description of PCRE's native API, which contains much
-additional functionality.
+additional functionality. There is no POSIX-style wrapper for PCRE's 16-bit
+library.
.P
The functions described here are just wrapper functions that ultimately call
the PCRE native API. Their prototypes are defined in the \fBpcreposix.h\fP
@@ -264,6 +265,6 @@
.rs
.sp
.nf
-Last updated: 16 May 2010
-Copyright (c) 1997-2010 University of Cambridge.
+Last updated: 09 January 2012
+Copyright (c) 1997-2012 University of Cambridge.
.fi
