Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
PPIx::Regexp::TokenizeUser Contributed Perl DocumentPPIx::Regexp::Tokenizer(3)

       PPIx::Regexp::Tokenizer - Tokenize a regular expression

	use PPIx::Regexp::Dumper;
	PPIx::Regexp::Dumper->new( 'qr{foo}smx'	)

       "PPIx::Regexp::Tokenizer" is a PPIx::Regexp::Support.

       "PPIx::Regexp::Tokenizer" has no	descendants.

       This class provides tokenization	of the regular expression.

       This class provides the following public	methods. Methods not
       documented here (or documented below under "EXTERNAL TOKENIZERS") are
       private,	and unsupported	in the sense that the author reserves the
       right to	change or remove them without notice.

	my $tokenizer =	PPIx::Regexp::Tokenizer->new( 'xyzzy' );

       This static method instantiates the tokenizer. You must pass it the
       regular expression to be	parsed,	either as a string or as a
       PPI::Element of some sort. You can also pass optional name/value	pairs
       of arguments. The option	names are specified without a leading dash.
       Supported options are:

       default_modifiers array_reference
	   This	argument specifies default statement modifiers.	It is
	   optional, but if specified must be an array reference. See the
	   PPIx::Regexp	new() documentation for	the details.

       encoding	name
	   This	option specifies the encoding of the string to be tokenized.
	   If specified, an "Encode::decode" is	done on	the string (or the
	   "content" of	the PPI	class) before it is tokenized.

	   This	Boolean	option specifies that the locations of the generated
	   tokens are to be computed.

       postderef boolean
	   PPIx::Regexp	for the	details.

	   This	option specifies whether the tokenizer recognizes postfix
	   dereferencing. See the PPIx::Regexp new() documentation for the

	   $PPIx::Regexp::Tokenizer::DEFAULT_POSTDEREF is not exported.

       strict boolean
	   This	option specifies whether tokenization should assume "use re
	   'strict';" is in effect.

	   The 'strict'	pragma was introduced in Perl 5.22, and	its
	   documentation says that it is experimental, and that	there is no
	   commitment to backward compatibility. The same applies to the
	   tokenization	produced when this option is asserted.

       trace number
	   Specifying a	positive value for this	option causes a	trace of the
	   tokenization. This option is	unsupported in the sense that the
	   author reserves the right to	alter it without notice.

	   If this option is unspecified, the value comes from environment
	   VARIABLES").	If this	environment variable does not exist, the
	   default is 0.

       Undocumented options are	unsupported.

       The returned value is the instantiated tokenizer, or "undef" if
       instantiation failed. In	the latter case	a call to "errstr" will	return
       the reason.

	print $tokenizer->content();

       This method returns the string being tokenized. This will be the	result
       of the PPI::Element->content() method if	the object was instantiated
       with a PPI::Element.

	print join ', ', @{ $tokenizer->default_modifiers() };

       This method returns a reference to a copy of the	array passed to	the
       "default_modifiers" argument to new(). If this argument was not used to
       instantiate the object, the return is a reference to an empty array.

       This method returns the encoding	of the data being parsed, if one was
       set when	the class was instantiated; otherwise it simply	returns	undef.

	my $tokenizer =	PPIx::Regexp::Tokenizer->new( 'xyzzy' )
	    or die PPIx::Regexp::Tokenizer->errstr();

       This static method returns an error description if tokenizer
       instantiation failed.

	print $tokenizer->failures(), "	tokenization failures\n";

       This method returns the number of tokenization failures encountered. A
       tokenization failure is represented in the output token stream by a

	$tokenizer->modifier( 'x' )
	    and	print "Tokenizing an extended regular expression\n";

       This method returns true	if the given modifier character	was found on
       the end of the regular expression, and false otherwise.

       Starting	with version 0.036_01, if the argument is a single-character
       modifier	followed by an asterisk	(intended as a wild card character),
       the return is the number	of times that modifier appears.	In this	case
       an exception will be thrown if you specify a multi-character modifier
       (e.g.  'ee*'), or if you	specify	one of the match semantics modifiers
       (e.g.  'a*').

       If called by an external	tokenizer, this	method returns true if if the
       given modifier was true at the current point in the tokenization.

	my $token = $tokenizer->next_token();

       This method returns the next token in the token stream, or nothing if
       there are no more tokens.

       This method exists simply for the convenience of	PPIx::Regexp::Dumper.
       It always returns true.

	my @tokens = $tokenizer->tokens();

       This method returns all remaining tokens	in the token stream.

       This class does very little of its own tokenization. Instead the	token
       classes contain external	tokenization routines, whose name is
       '__PPIX_TOKENIZER__' concatenated with the current mode of the
       tokenizer ('regexp' for regular expressions, 'repl' for the replacement

       These external tokenizers are called as static methods, and passed the
       "PPIx::Regexp::Tokenizer" object	and the	current	character in the
       character stream.

       If the external tokenizer wants to make one or more tokens, it returns
       an array	containing either length in characters for tokens of the
       tokenizer's own class, or the results of	one or more "make_token" calls
       for tokens of an	arbitrary class.

       If the external tokenizer is not	interested in the characters starting
       at the current position it simply returns.

       The following methods are for the use of	external tokenizers, and are
       not part	of the public interface	to this	class.

	if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
	    foreach ( $tokenizer->capture() ) {
		print "$_\n";

       This method returns all the contents of any capture buffers from	the
       previous	call to	"find_regexp". The first element of the	array (i.e.
       element 0) corresponds to $1, and so on.

       The captures are	cleared	by "make_token", as well as by another call to

	$tokenizer->cookie( foo	=> sub { 1 } );
	my $cookie = $tokenizer->cookie( 'foo' );
	my $old_hint = $tokenizer->cookie( foo => undef	);

       This method either creates, deletes, or accesses	a cookie.

       A cookie	is a code reference which is called whenever the tokenizer
       makes a token. If it returns a false value, it is deleted. Explicitly
       setting the cookie to "undef" also deletes it.

       When you	call "$tokenizer->cookie( 'foo'	)", the	current	cookie is
       returned. If you	pass a new value of "undef" to delete the token, the
       deleted cookie (if any) is returned.

       When the	"make_token" method calls a cookie, it passes it the tokenizer
       and the token just made.	If a token calls a cookie, it is recommended
       that it merely pass the tokenizer, though of course the token can do
       whatever	it wants.

       The cookie mechanism seems to be	a bit of a crock, but it appeared to
       be more work to fix things up in	the lexer after	the tokenizer got
       something wrong.

       The recommended way to write a cookie is	to use a closure to store any
       necessary data, and have	a call to the cookie return the	data;
       otherwise the ultimate consumer of the cookie has no way	to access the
       data. Of	course,	it may be that the presence of the cookie at a certain
       point in	the parse is all that is required.

	$tokenizer->expect( 'PPIx::Regexp::Token::Code'	);

       This method inserts a given class at the	head of	the token scan,	for
       the next	iteration only.	More than one class can	be specified. Class
       names can be abbreviated	by removing the	leading	'PPIx::Regexp::'.

       If no class is specified, this method does nothing.

       The expectation lasts from the next time	"get_token" is called until
       the next	time "make_token" makes	a significant token, or	until the next
       "expect"	call if	that is	done sooner.

	my $end	= $tokenizer->find_regexp( qr{ \A \w+ }smx );
	my ( $begin, $end ) = $tokenizer->find_regexp(
	    qr{	\A \w+ }smx );

       This method finds the given regular expression in the content, starting
       at the current position.	If called in scalar context, the offset	from
       the current position to the end of the matched string is	returned. If
       called in list context, the offsets to both the beginning and the end
       of the matched string are returned.

	my $offset = $tokenizer->find_matching_delimiter();

       This method is used by tokenizers to find the delimiter matching	the
       character at the	current	position in the	content	string.	If the
       delimiter is an opening bracket of some sort, bracket nesting will be
       taken into account.

       When searching for the matching delimiter, the back slash character is
       considered to escape the	following character, so	back-slashed
       delimiters will be ignored. No other quoting mechanisms are recognized,
       though, so delimiters inside quotes still count.	This is	actually the
       way Perl	works, as

	$ perl -e 'qr<(?{ print	"}" })>'


       This method returns the offset from the current position	in the content
       string to the matching delimiter	(which will always be positive), or
       undef if	no match can be	found.

       This method returns the name of the current mode	of the tokenizer.

	my $start_delimiter = $tokenizer->get_start_delimiter();

       This method is used by tokenizers to access the start delimiter for the
       regular expression.

	my $token = $tokenizer->make_token( 3 );
	my @tokens = $tokenizer->get_token();

       This method returns the next token that can be made from	the input
       stream. It is not part of the external interface, but is	intended for
       the use of an external tokenizer	which calls it after making and
       retaining its own token to look at the next token ( if any ) in the
       input stream.

       If any external tokenizer calls get_token without first calling
       make_token, a fatal error occurs; this is better	than the infinite
       recursion which would occur if the condition were not trapped.

       An external tokenizer must return anything returned by get_token;
       otherwise tokens	get lost.

       This method returns true	if the top-level structure being tokenized
       interpolates; that is, if the delimiter is not a	single quote.

	return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );

       This method is used by this class (and possibly by individual
       tokenizers) to manufacture a token. Its arguments are the number	of
       characters to include in	the token, and optionally the class of the
       token. If no class name is given, the caller's class is used. Class
       names may be shortened by removing the initial 'PPIx::Regexp::',	which
       will be restored	by this	method.

       The token will be manufactured from the given number of characters
       starting	at the current cursor position,	which will be adjusted.

       If the given length would include characters past the end of the	string
       being tokenized,	the length is reduced appropriately. If	this means a
       token with no characters, nothing is returned.

	if ( $tokenizer->find_regexp( qr{ \A \w+ }smx )	) {
	    print $tokenizer->match(), "\n";

       This method returns the string matched by the previous call to

       The match is set	to "undef" by "make_token", as well as by another call
       to "find_regexp".


       This method duplicates the modifiers on the top of the modifier stack,
       with the	intent of creating a locally-scoped copy of the	modifiers.
       This should only	be called by an	external tokenizer that	is actually
       creating	a modifier scope. In other words, only when creating a
       PPIx::Regexp::Token::Structure token whose content is '('.

	$tokenizer->modifier_modify( name => $value ...	);

       This method sets	new values for the modifiers in	the local scope. Only
       the modifiers whose names are actually passed have their	values

       This method is intended to be called after manufacturing	a
       PPIx::Regexp::Token::Modifier token, and	passed the results of its
       "modifiers" method.


       This method removes the modifiers on the	top of the modifier stack.
       This should only	be called by an	external tokenizer that	is ending a
       modifier	scope. In other	words, only when creating a
       PPIx::Regexp::Token::Structure token whose content is ')'.

       Note that this method will never	pop the	last modifier item off the
       stack, to guard against unmatched right parentheses.

	$tokenizer->modifier_seen( 'i' )
	    and	print "/i was seen at some point.\n";

       Unlike modifier(), this method returns a	true value if the given
       modifier	has been seen in any scope visible from	the current location
       in the parse. There is no magic for group match semantics ( /a, /aa,
       /d, /l, /u) or modifiers	that can be repeated, like /x and /xx, or /e
       and /ee.

	my $character =	$tokenizer->peek();
	my $next_char =	$tokenizer->peek( 1 );

       This method returns the character at the	given non-negative offset from
       the current position. If	no offset is given, an offset of 0 is used.

       If you ask for a	negative offset	or an offset off the end of the	sting,
       "undef" is returned.

       This method makes a PPI document	out of the remainder of	the string,
       and returns it.

	$tokenizer->prior_significant_token( 'can_be_quantified' )
	   and print "The prior	token can be quantified.\n";

       This method calls the named method on the most-recently-instantiated
       significant token, and returns the result. Any arguments	subsequent to
       the method name will be passed to the method.

       Because this method is designed to be used within the tokenizing
       system, it will die horribly if the named method	does not exist.

       If called with no arguments at all the most-recently-instantiated
       significant token is returned.

	say 'Parse is ', $tokenizer->strict() ?	'strict' : 'lenient';

       This method simply returns true or false, depending on whether the
       'strict'	option to "new()" was true or false.

       A tokenizer trace can be	requested by setting environment variable
       PPIX_REGEXP_TOKENIZER_TRACE to a	numeric	value other than 0. Use	of
       this environment	variable is unsupported	in the same sense that the
       "trace" option of "new" is unsupported. Explicitly specifying the
       "trace" option to "new" overrides the environment variable.

       The real	reason this is documented is to	give the user a	way to
       troubleshoot funny output from the tokenizer.

       Support is by the author. Please	file bug reports at
       <>, or in electronic mail to the author.

       Thomas R. Wyant,	III wyant at cpan dot org

       Copyright (C) 2009-2020 by Thomas R. Wyant, III

       This program is free software; you can redistribute it and/or modify it
       under the same terms as Perl 5.10.0. For	more details, see the full
       text of the licenses in the directory LICENSES.

       This program is distributed in the hope that it will be useful, but
       without any warranty; without even the implied warranty of
       merchantability or fitness for a	particular purpose.

perl v5.32.1			  2020-10-08	    PPIx::Regexp::Tokenizer(3)


Want to link to this manual page? Use this URL:

home | help