Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
PCRE2PARTIAL(3)		   Library Functions Manual	       PCRE2PARTIAL(3)

       PCRE2 - Perl-compatible regular expressions


       In  normal use of PCRE2,	if there is a match up to the end of a subject
       string, but more	characters are needed to  match	 the  entire  pattern,
       PCRE2_ERROR_NOMATCH  is	returned,  just	 like any other	failing	match.
       There are circumstances where it	might be helpful to  distinguish  this
       "partial	match" case.

       One  example  is	 an application	where the subject string is very long,
       and not all available at	once. The requirement here is to be able to do
       the  matching  segment  by segment, but special action is needed	when a
       matched substring spans the boundary between two	segments.

       Another example is checking a user input	string as it is	typed, to  en-
       sure  that  it conforms to a required format. Invalid characters	can be
       immediately diagnosed and rejected, giving instant feedback.

       Partial matching	is a PCRE2-specific feature; it	is  not	 Perl-compati-
       ble.  It	 is  requested	by  setting  one  of the PCRE2_PARTIAL_HARD or
       PCRE2_PARTIAL_SOFT options when calling a matching function.  The  dif-
       ference	between	 the  two options is whether or	not a partial match is
       preferred to an alternative complete match, though the  details	differ
       between	the  two  types	of matching function. If both options are set,
       PCRE2_PARTIAL_HARD takes	precedence.

       If you want to use partial matching with	just-in-time  optimized	 code,
       as  well	 as  setting a partial match option for	the matching function,
       you must	also call pcre2_jit_compile() with one or both	of  these  op-


       PCRE2_JIT_COMPLETE  should also be set if you are going to run non-par-
       tial matches on the same	pattern. Separate code is  compiled  for  each
       mode.  If  the appropriate JIT mode has not been	compiled, interpretive
       matching	code is	used.

       Setting a partial matching option disables two of PCRE2's standard  op-
       timization  hints. PCRE2	remembers the last literal code	unit in	a pat-
       tern, and abandons matching immediately if it is	 not  present  in  the
       subject	string.	 This optimization cannot be used for a	subject	string
       that might match	only partially.	PCRE2 also remembers a minimum	length
       of  a matching string, and does not bother to run the matching function
       on shorter strings. This	optimization  is  also	disabled  for  partial


       A  possible  partial  match  occurs during matching when	the end	of the
       subject string is reached successfully, but either more characters  are
       needed  to complete the match, or the addition of more characters might
       change what is matched.

       Example 1: if the pattern is /abc/ and the subject is "ab", more	 char-
       acters  are  definitely	needed	to complete a match. In	this case both
       hard and	soft matching options yield a partial match.

       Example 2: if the pattern is /ab+/ and the subject is "ab", a  complete
       match  can  be  found, but the addition of more characters might	change
       what is matched.	In this	case, only PCRE2_PARTIAL_HARD returns  a  par-
       tial match; PCRE2_PARTIAL_SOFT returns the complete match.

       On  reaching the	end of the subject, when PCRE2_PARTIAL_HARD is set, if
       the next	pattern	item is	\z, \Z,	\b, \B,	or $ there is always a partial
       match.	Otherwise, for both options, the next pattern item must	be one
       that inspects a character, and at least one of the  following  must  be

       (1)  At	least  one  character has already been inspected. An inspected
       character need not form part of the final  matched  string;  lookbehind
       assertions  and the \K escape sequence provide ways of inspecting char-
       acters before the start of a matched string.

       (2) The pattern contains	one or more lookbehind assertions. This	condi-
       tion  exists in case there is a lookbehind that inspects	characters be-
       fore the	start of the match.

       (3) There is a special case when	the whole pattern can match  an	 empty
       string.	 When  the  starting  point  is	at the end of the subject, the
       empty string match is a possibility, and	if PCRE2_PARTIAL_SOFT  is  set
       and  neither  of	the above conditions is	true, it is returned. However,
       because adding more characters  might  result  in  a  non-empty	match,
       PCRE2_PARTIAL_HARD  returns  a  partial match, which in this case means
       "there is going to be a match at	this point, but	until some more	 char-
       acters are added, we do not know	if it will be an empty string or some-
       thing longer".


       When  a	partial	 matching  option  is  set,  the  result  of   calling
       pcre2_match() can be one	of the following:

       A successful match
	 A complete match has been found, starting and ending within this sub-

	 No match can start anywhere in	this subject.

	 Adding	more characters	may result in a	complete match that  uses  one
	 or more characters from the end of this subject.

       When a partial match is returned, the first two elements	in the ovector
       point to	the portion of the subject that	was matched, but the values in
       the rest	of the ovector are undefined. The appearance of	\K in the pat-
       tern has	no effect for a	partial	match. Consider	this pattern:


       If it is	matched	against	"456abc123xyz" the result is a complete	match,
       and  the	ovector	defines	the matched string as "123", because \K	resets
       the "start of match" point. However, if a partial  match	 is  requested
       and  the	subject	string is "456abc12", a	partial	match is found for the
       string "abc12", because all these characters are	needed	for  a	subse-
       quent re-match with additional characters.

       If  there  is more than one partial match, the first one	that was found
       provides	the data that is returned. Consider this pattern:


       If this is matched against the subject string "abc123dog", both	alter-
       natives	fail  to  match,  but the end of the subject is	reached	during
       matching, so PCRE2_ERROR_PARTIAL	is returned. The offsets are set to  3
       and  9, identifying "123dog" as the first partial match.	(In this exam-
       ple, there are two partial matches, because "dog" on its	own  partially
       matches the second alternative.)

   How a partial match is processed by pcre2_match()

       What happens when a partial match is identified depends on which	of the
       two partial matching options is set.

       If PCRE2_PARTIAL_HARD is	set, PCRE2_ERROR_PARTIAL is returned  as  soon
       as  a partial match is found, without continuing	to search for possible
       complete	matches. This option is	"hard" because it prefers  an  earlier
       partial match over a later complete match. For this reason, the assump-
       tion is made that the end of the	supplied subject  string  is  not  the
       true  end of the	available data,	which is why \z, \Z, \b, \B, and $ al-
       ways give a partial match.

       If PCRE2_PARTIAL_SOFT is	set, the  partial  match  is  remembered,  but
       matching	continues as normal, and other alternatives in the pattern are
       tried. If no complete match can be found,  PCRE2_ERROR_PARTIAL  is  re-
       turned instead of PCRE2_ERROR_NOMATCH. This option is "soft" because it
       prefers a complete match	over a partial match. All the various matching
       items  in a pattern behave as if	the subject string is potentially com-
       plete; \z, \Z, and $ match at the end of	the subject,  as  normal,  and
       for \b and \B the end of	the subject is treated as a non-alphanumeric.

       The  difference	between	the two	partial	matching options can be	illus-
       trated by a pattern such	as:


       This matches either "dog" or "dogsbody",	greedily (that is, it  prefers
       the  longer  string  if	possible). If it is matched against the	string
       "dog" with PCRE2_PARTIAL_SOFT, it yields	a complete  match  for	"dog".
       However,	 if  PCRE2_PARTIAL_HARD	is set,	the result is PCRE2_ERROR_PAR-
       TIAL. On	the other hand,	if the pattern is made ungreedy	the result  is


       In  this	 case  the  result  is always a	complete match because that is
       found first, and	matching never	continues  after  finding  a  complete
       match. It might be easier to follow this	explanation by thinking	of the
       two patterns like this:

	 /dog(sbody)?/	  is the same as  /dogsbody|dog/
	 /dog(sbody)??/	  is the same as  /dog|dogsbody/

       The second pattern will never match "dogsbody", because it will	always
       find the	shorter	match first.

   Example of partial matching using pcre2test

       The  pcre2test data modifiers partial_hard (or ph) and partial_soft (or
       ps) set PCRE2_PARTIAL_HARD and PCRE2_PARTIAL_SOFT,  respectively,  when
       calling	pcre2_match(). Here is a run of	pcre2test using	a pattern that
       matches the whole subject in the	form of	a date:

	   re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
	 data> 25dec3\=ph
	 Partial match:	23dec3
	 data> 3ju\=ph
	 Partial match:	3ju
	 data> 3juj\=ph
	 No match

       This example gives the same results for	both  hard  and	 soft  partial
       matching	options. Here is an example where there	is a difference:

	   re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
	 data> 25jun04\=ps
	  0: 25jun04
	  1: jun
	 data> 25jun04\=ph
	 Partial match:	25jun04

       With   PCRE2_PARTIAL_SOFT,  the	subject	 is  matched  completely.  For
       PCRE2_PARTIAL_HARD, however, the	subject	is assumed not to be complete,
       so there	is only	a partial match.


       PCRE  was  not originally designed with multi-segment matching in mind.
       However,	over time, features (including	partial	 matching)  that  make
       multi-segment matching possible have been added.	A very long string can
       be searched segment by segment  by  calling  pcre2_match()  repeatedly,
       with the	aim of achieving the same results that would happen if the en-
       tire string was available for searching all  the	 time.	Normally,  the
       strings	that  are  being  sought are much shorter than each individual
       segment,	and are	in the middle of very long strings, so the pattern  is
       normally	not anchored.

       Special	logic  must  be	implemented to handle a	matched	substring that
       spans a segment boundary. PCRE2_PARTIAL_HARD should be used, because it
       returns	a  partial match at the	end of a segment whenever there	is the
       possibility of changing	the  match  by	adding	more  characters.  The
       PCRE2_NOTBOL option should also be set for all but the first segment.

       When a partial match occurs, the	next segment must be added to the cur-
       rent subject and	the match re-run, using	the  startoffset  argument  of
       pcre2_match()  to  begin	 at the	point where the	partial	match started.
       For example:

	   re> /\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d/
	 data> ...the date is 23ja\=ph
	 Partial match:	23ja
	 data> ...the date is 23jan19 and on that day...\=offset=15
	  0: 23jan19
	  1: jan

       Note the	use of the offset modifier to start the	new  match  where  the
       partial match was found.	In this	example, the next segment was added to
       the one in which	 the  partial  match  was  found.  This	 is  the  most
       straightforward approach, typically using a memory buffer that is twice
       the size	of each	segment. After a partial match,	the first half of  the
       buffer  is discarded, the second	half is	moved to the start of the buf-
       fer, and	a new segment is added before repeating	the match  as  in  the
       example above. After a no match,	the entire buffer can be discarded.

       If there	are memory constraints,	you may	want to	discard	text that pre-
       cedes a partial match before adding the	next  segment.	Unfortunately,
       this  is	 not  at  present straightforward. In cases such as the	above,
       where the pattern does not contain any lookbehinds, it is sufficient to
       retain  only  the  partially matched substring. However,	if the pattern
       contains	a lookbehind assertion,	characters that	precede	the  start  of
       the  partial match may have been	inspected during the matching process.
       When pcre2test displays a partial match,	it indicates these  characters
       with '<'	if the allusedtext modifier is set:

	   re> "(?<=123)abc"
	 data> xx123ab\=ph,allusedtext
	 Partial match:	123ab

       However,	 the  allusedtext  modifier is not available for JIT matching,
       because JIT matching does not record  the  first	 (or  last)  consulted
       characters.  For	this reason, this information is not available via the
       API. It is therefore not	possible in general to obtain the exact	number
       of characters that must be retained in order to get the right match re-
       sult. If	you cannot retain the  entire  segment,	 you  must  find  some
       heuristic way of	choosing.

       If  you know the	approximate length of the matching substrings, you can
       use that	to decide how much text	to retain. The only lookbehind	infor-
       mation  that  is	 currently  available via the API is the length	of the
       longest individual lookbehind in	a pattern, but this can	be  misleading
       if  there  are  nested  lookbehinds.  The  value	 returned  by  calling
       pcre2_pattern_info() with the PCRE2_INFO_MAXLOOKBEHIND  option  is  the
       maximum number of characters (not code units) that any individual look-
       behind  moves  back  when  it  is  processed.   A   pattern   such   as
       "(?<=(?<!b)a)"  has a maximum lookbehind	value of one, but inspects two
       characters before its starting point.

       In a non-UTF or a 32-bit	case, moving back is just a  subtraction,  but
       in  UTF-8  or  UTF-16  you  have	 to count characters while moving back
       through the code	units.

PARTIAL	MATCHING USING pcre2_dfa_match()

       The DFA function	moves along the	subject	string character by character,
       without	backtracking,  searching  for  all possible matches simultane-
       ously. If the end of the	subject	is reached before the end of the  pat-
       tern, there is the possibility of a partial match.

       When PCRE2_PARTIAL_SOFT is set, PCRE2_ERROR_PARTIAL is returned only if
       there have been no complete matches. Otherwise,	the  complete  matches
       are  returned.	If  PCRE2_PARTIAL_HARD	is  set, a partial match takes
       precedence over any complete matches. The portion of  the  string  that
       was  matched  when  the	longest	 partial match was found is set	as the
       first matching string.

       Because the DFA function	always searches	for all	possible matches,  and
       there  is no difference between greedy and ungreedy repetition, its be-
       haviour is different from the pcre2_match(). Consider the string	 "dog"
       matched against this ungreedy pattern:


       Whereas	the  standard  function	stops as soon as it finds the complete
       match for "dog",	the DFA	function also  finds  the  partial  match  for
       "dogsbody", and so returns that when PCRE2_PARTIAL_HARD is set.


       When a partial match has	been found using the DFA matching function, it
       is possible to continue the match by providing additional subject  data
       and  calling  the function again	with the same compiled regular expres-
       sion, this time setting the PCRE2_DFA_RESTART option. You must pass the
       same working space as before, because this is where details of the pre-
       vious partial match are stored. You can set the	PCRE2_PARTIAL_SOFT  or
       PCRE2_PARTIAL_HARD  options  with PCRE2_DFA_RESTART to continue partial
       matching	over multiple segments.	Here is	an example using pcre2test:

	   re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
	 data> 23ja\=dfa,ps
	 Partial match:	23ja
	 data> n05\=dfa,dfa_restart
	  0: n05

       The first call has "23ja" as the	subject, and requests  partial	match-
       ing;  the  second  call	has  "n05"  as	the  subject for the continued
       (restarted) match.  Notice that when the	match is  complete,  only  the
       last  part  is  shown;  PCRE2 does not retain the previously partially-
       matched string. It is up	to the calling program to do that if it	 needs
       to.  This  means	 that, for an unanchored pattern, if a continued match
       fails, it is not	possible to try	again at a  new	 starting  point.  All
       this facility is	capable	of doing is continuing with the	previous match
       attempt.	For example, consider this pattern:


       If the first part of the	subject	is "ABC123", a partial	match  of  the
       first  alternative  is found at offset 3. There is no partial match for
       the second alternative, because such a match does not start at the same
       point  in  the  subject	string.	Attempting to continue with the	string
       "7890" does not yield a match  because  only  those  alternatives  that
       match  at one point in the subject are remembered. Depending on the ap-
       plication, this may or may not be what you want.

       If you do want to allow for starting again at the next  character,  one
       way  of	doing it is to retain some or all of the segment and try a new
       complete	match, as described for	pcre2_match() above. Another possibil-
       ity  is to work with two	buffers. If a partial match at offset n	in the
       first buffer is followed	by "no match" when PCRE2_DFA_RESTART  is  used
       on  the	second buffer, you can then try	a new match starting at	offset
       n+1 in the first	buffer.


       Philip Hazel
       University Computing Service
       Cambridge, England.


       Last updated: 04	September 2019
       Copyright (c) 1997-2019 University of Cambridge.

PCRE2 10.34		       04 September 2019	       PCRE2PARTIAL(3)


Want to link to this manual page? Use this URL:

home | help